Reinforced Lifelong Editing for Language Models
Authors: Zherui Li
Houcheng Jiang
Hao Chen
Baolong Bi
Zhenhong Zhou
Fei Sun
Junfeng Fang
Xiang Wang
Venue: ICML 2025 poster (2025)
Keywords: Model Editing
Meta-learning
Large Language Model
Primary area: deep_learning->large_language_models
TL;DR: We propose RLEdit, a hypernetwork-based lifelong model editing method that achieves both effectiveness and efficiency.

Abstract: Large language models (LLMs) acquire information from pre-training corpora, but their stored knowledge can become inaccurate or outdated over time. Model editing addresses this challenge by modifying model parameters without retraining, and prevalent approaches leverage hypernetworks to generate these parameter updates. However, they face significant challenges in lifelong editing due to their incompatibility with LLM parameters that dynamically change during the editing process. To address this, we observed that hypernetwork-based lifelong editing aligns with reinforcement learning modeling and proposed **RLEdit**, an RL-based editing method. By treating editing losses as rewards and optimizing hypernetwork parameters at the full knowledge sequence level, we enable it to precisely capture LLM changes and generate appropriate parameter updates. Our extensive empirical evaluation across several LLMs demonstrates that RLEdit outperforms existing methods in lifelong editing with superior effectiveness and efficiency, achieving a **59.24%** improvement while requiring only **2.11%** of the time compared to most approaches.

Reinforced Lifelong Editing for Language Models
Zherui Li
Houcheng Jiang" 22 Hao Chen'
Baolong Bi Zhenhong Zhou' Fei Sun
Junfeng Fang"
Xiang Wang?
Abstract
ing to ongoing demand for continuous updates to model parameters (Lazaridou et al., 2021; De Cao et al., 2021). A straightforward solution is lifelong model editing, which enables sequential knowledge updates in LLMs without compromising their core performance (Hartvigsen et al., 2023; Wang et al., 2024; Fang et al., 2024). For each knowl- edge K, conventional lifelong editing methods individually calculate an optimal update, denoted as A, to adjust the LLM parameters W through matrix operation (Meng et al., 2022; 2023), as illustrated in Figure 1 (a). Since the com- plexity and approximations involved in repetitive matrix operation make conventional methods computationally ex- pensive and error-prone, recently, hypernetwork ods (Mitchell al., 2022a; Tan al., 2024) have emerged as more efficient and elegant solution. By modeling the K. - A process with a hypernetorkk, as shown in Figure (b), the hypernetwork can efficiently map knowledge K. to the required update A once trained on these K-A pairs. This approach eliminates the need for expensive and repet- itive matrix operations, thus enabling a more concise and practical model editing process.
Large language models (LLMs) acquire informa- tion from pre-training corpora, but their stored knowledge can become inaccurate or outdated over time. Model editing addresses this chal- lenge by modifying model parameters without retraining, and prevalent approaches leverage hy- pemetworks to generate these parameter updates. However, they face significant challenges in life- long editing due to their incompatibility with LLM parameters that dynamically change dur- ing the editing process. To address this, we observed that hypernetwork-bbiee based lifelong edit- ing aligns with reinforcement learning model- ing and proposed RLEdit, RL-based editing method. By treating editing losses as rewards and optimizing hypernetwork parameters at the full knowledge sequence level, we enable it to precisely capture LLM changes and generate ap- propriate parameter updates. Our extensive em- pirical evaluation across several LLMs demon- strates that RLEdit outperforms existing meth- ods in lifelong editing with superior effectiveness and efficiency, achieving a 59.24% improvement while requiring only 2.11% % of the time compared to most approaches. Our code is available at: https://gtthbb.oomm//iii44.....
meth-
However, current hypemnetwor-- based methods struggle to handle long-term lifelong editing tasks (e.g., tasks involv- ing more than 100 edits) (Yang et al., 2024a;b). To ensure convergence during training, all K-A pairs used to train the hypernetwork must be collected from the same LLM with identical parameters. Consequently, a single hyper- network can only model the K. + A process for a fixed LLM,, limiting its applicability across evolving models in lifelong editing scenarios. As illustrated in Figure (b), while locate-then 2023) and Alphalddit (Fang et al., 2024) can effectively support up to 5,000 edits, hypernetwork-bon based approaches tend to fail after only around 100 edits. This limitation severely restricts the broader use and further advancement of syspernetworr-bbaeed editing approaches.
1. Introduction
methods like MEMIT (Meng et al.,
Although large language models (LLMs) have achieved sig- nificant success in various downstream tasks (Brown et al., 2020; Zhao al., 2024), their performance is hindered by the outdated or erroneous knowledge they may store, lead-
"gqual contribution 'Beijing University of Posts and Telecom- munications UUniversity of Science and Technology of China 2 "Institute of Computing Technology, Chinese Academy of Sci- ences "National University of Singapore. Correspondence to: Junfeng Fang <fangjf1997G gmail.com>, Xiang Wang <xiang- wang 123 @ gmail.com>.
This leads to natural question: "How to keep the hyper- network effective for lifelong editing?" To address this, the hyperneworr must: 1) capture dynamic changes in the LLM, and 2) adaptively provide
based on the current LLM. This inspires us to turn to Reinforcement Learning (RL) (Arulkumaran et al.. 2017), which aims to: 1) capture dynamic changes in the environment, and 2) adaptively pro-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reinforced Lifelong Editing for Language Models
(b) Hypernet -hssed (a) Conventional Editing
Knowledge
Editing Process
Time & Accuracy
the RL-based training process, and it reduces interference between successive updates.
Ace 8.0% per edit
W W A
To further validate RLEdit's capabilities, we conducted ex- tensive experiments on several LLMs. When compared to existing parameter-modifying methods (e.g., RECT (Gu et al., 2024), DAFNet (Zhang et al., 2024b), AlphaEdit (Fang et al., 2024)), RLEdit consistently outperforms these approaches across most evaluation metrics, requiring only 2% of the computation time (including the hypernetwork training overhead) on average. As shown in Figure (c), RLEdit maintains satisfactory performance even after more than 20,000 edits, with each edit taking only 0.17s. Addi- tionally, as the first approach to model lifelong editing as an RL. problem, our ablation studies underscore the critical role of the RL framework in RLEdit's performance, paving the way for broader applications of lifelong editing and further advancements in hypernetwork-based methods.
W+ A
W+
Namber of Edits
0.50
5000
K2,
w W- W-WWss
0.42s per edit
Truin
Train
Ace 9.95
HyperNetwork
Apply
Apply
Number of Edite
K.
W W+A X W.A.* As
8:50
(c) RLEdi (Ours)
K + K2
W A +
0.77% per edit
Train vin RL
Ace 0.95
HyperNelwork
Apply
Apple
Mumber Edits
K.
KC
w
W+ W+A.+A
0.50
20000
2. Preliminary
Figure 1: Comparison of different lifelong editing paradigms. Here, K., and Ks, represent two consecutive new knowledge samples, while A₂ and Ag are their corre- sponding parameter updates (K. and A with other subscripts paired data used for training the hypernetwork). (a) re- quires independent calculations for each knowledge sample, resulting in low efficiency.
2.1. Lifelong Model Editing
Lifelong model editing requires continuous and sequen- tial model editing on the same LLM,, with the total num- ber of edits potentially reaching thousands or even tens of thousands. It requires the post-edited LLM to remember the knowledge from recent edits, retain previously edited knowledge, and maintain comprehensive performance. Let Swo X + Y be the initially pre-trained LLM with parameters Wo. which maps the input set X to the out- put set Y. In the lifelong editing task, we have an edit- ing dataset D, = {(X, (z)1 (In, yn)], where X = (Z),..., I.,)) is the input stream of knowledge to be edited, and Y (yI,... Vn) is the target output. For the entire input stream X, the pre-trained model outputs [w,(X) Y', where Y" = (u'....,33) I/s) represents the ground truth. Through sequential editing, we aim to modify the model parameters to achieve fvv (X) = Y. Therefore, we introduce the model editor (ME). During model editing, the LLM sequentially reads input-output pairs (x,, y) from dataset D. When editing the t-th knowledge, ME modifies the LLM parameters according to the following formula:
(b) is efficient but the hyper- network cannot be generalized to the post-edited LLM. (c) ensures both efficiency and effectiveness, even after more than 20,000 edits. The time shows both initial setup time (including covariance matrix computation or hypernetwork training) and editing time. Best viewed in color.
vide actions based the current state. Additionally, RL is typically applied in the context of the Markov Decision Pro- cess (MDP) (Bellman, 1957). Lifelong editing naturally fits into this framework, as each edit depends solely on the new knowledge and the current LLM state, while the existing loss function and hypernetwork parameters can be modeled as reward and policy respectively.
Building on these observations, we propose RLEdit, which applies RL to construct and train the hypernetwork. Specifi- cally, RLEdit treats the hypernetwork as the agent, defines A as the action, and quantifies the performance of L.M as the reward function. We employ an offline policy update approach, enabling RLEdit to recognize the current state of LLMs and adaptively adjust its output A, while retaining the efficiency of current hypemetwork-baat based methods. To meet the requirement that post-edited L.MSs must retain previ- ously edited knowledge in lifelong editing, we optimize the reward function by incorporating the difference between the current update and the cumulative sum of previous updates. This modification serves two key purposes: it accelerates
fw. ME(Jw. Mt), t t= I....,,,
2.2. Hypernetwork-basee Editing Methods
One approach in model editing is hypernetwork-base edit- ing methods, which involve training an editing hypernet- work generate LLM parameter updates. Hypernetwork- based methods recognize that LLM fine-tuning gradients contain rich information and use the loss L of the edited LLM on both edited and unrelated knowledge as the start- ing point for hypernetwork training. It aims to train a hypernetwork to map fine-tuning gradients to LLM pa-
2

---

Reinforced Lifelong Editing for Language Models
rameter updates. To address the computational burden of d x d weight matrices, Mitchell et al. (2022a) decomposes each layer's gradient matrix into rank-1 product form V Vw = 6 where 61+1 is the gradient of the loss with respect to pre-activations of layer 1 + 1, and wy is the input to layer l. Through low-rank decomposition, the hy- pernetwork H can learn a d + d mapping:
Table 1: The correspondence between MDP elements and
hypernetwork- beaseeetwor--based lifelong editing components.
Markov Decision Process
Lifelong Editing
Agent Environment
H fw
Policy æ
0
Action AA
Vw (W, y) or Vw
State S
H:: 6141 un + 8141 x aj;
(2)
Reward R
where 5/+1 and ün are pseudo-activation. and pseudo- increments respectively, and the final updates applied to LLM parameters is Vw, = 8
3.1.1. MDP FORMULATION
The training process of hypernetwork H. in lifelong editing can be formalized as the following procedure:
In this paper, Vw and A are equivalent, both representing the parameter updates required for editing. We use Vw consistently throughout the rest of this paper.
1. At the I-th edit (i.e., time step 1), new knowledge sam- ple (Iz,, is input to the LLM with parameters Wi-1 to collect fine-tuning gradient Vw..i
2.3. Reinforcement Learning
2. Vw.-; is fed into H to get parameter update Vw.
reinforcement learning task is usually formalized as a Markov Decision Process (MDP). An MDP can be defined as tuple (S,A, R, 7). At each time step t, the RL system is in state 8t E S, and the agent generates an action a E A based on its policy π and the current state 8t, ie., at ~ π (st). The MDP then transitions to state 81+1 and the agent receives a reward rt = R( Se: at), rt E R. After all time steps, the agent optimizes its policy to maximize the total trajectory return (T) Ener 7'r,, where Y represents discount factor and represents one episode trajectory of the agent in the MDP.
Vw, is used update W to produce Wi:
4. The loss of (m,, yt) on the LLM with parameters Wi is computed to update the hypernetwork.
This process iterates until all knowledge in the training set is traversed. We can observe that the above process only depends on Wi-1 and (I2,, ye) at time step f, Le., it is independent of states from time step 0 to t - 2. Therefore, it naturally satisfies the Markov property. Formally:
Pr
=(VW..))VWw....
3. Method
(3)
In this section, we elaborate RLEdit, method that achieves efficient and effective lifelong editing through training hypernetwork adapted to knowledge sequences. We begin by introducing how to establish a reinforcement learning (RI.) paradigm for hypernetworks training in lifelong editing (Section 3.1), which includes the key component of RLEdit: reward function design. We then describe RLEdit's training strategy Section 3.2. Finally, in Section 3.3, we detail the practical implementation and considerations of RLEdit.
= Pr -Pr[H(V fw H(Ww, 2) - .
To further extend the above Markov process into an MDP, we introduce several concepts: state 8 E S, action E A, reward r E R. and policy オ, which are formalized in Table 1. Here, donates the parameters of H. and L represents the loss on LLM for (r,y), Given n new knowledge samples (I,, y1 y in the training set, the interactions between 8, o, and r over n time steps constitute a trajectory of the MDP:
{No. (F1, Vwn fi....,
3.1. Modeling Lifelong Editing as an RL Task
Building an RL paradigm for hypernetworks training in life- long editing aims to capture dynamic changes in LLM and adaptively generate updates that fit the current parameter state, mentioned in Section 1. Two key aspects enable this: (1) modeling the hypernetwork training process in lifelong editing as a Markov Decision Process (MDP) (Bell- man, 1957; Puterman, 1994), which RL excels at solving (Sutton& & Barto, 1998), and (2) carefully designing a reward function tailored to the lifelong editing task.
(Wn-1, (Ini vn)) V.., -Cn)
(4)
+ {81, a1, ,,...., Sny any rn].
More specifically, in each time step, agent H. employs pa- rameters 0 as policy i to generate action Vw based on system state (W, (xx w)}. This action is then applied to environment fw. resulting in a state transition:
x (T())) = azs 8t + at -> 81+1-
(5)
3

---

Reinforced Lifelong Editing for Language Models
W
Presklent of USA? Ay. Donald Tramp
action
Prosectied LTM
fw
V Viica
fWN-2
D0 LLM fw
HyperNet
Agentpucty
RLEdit
Envinonmentt.co
H6.8
4m
Vw
HyperNetwork Har (Trained RLEdit)
reward
Pre sained LLM
fw
f
state
(x1,,))) u)) (2N (
Loss
XN
V w
Loass
Losss
Hg + 0' = arg max.
Return
Lossa
Loss
Post LLM
fw
aa) Training RLEdit
(b) Lifelong Editing via RLEdit
Figure 2: Overview of lifelong editing with RLEdit. (a) illustrates the training process RLEdit's hypernetwork, while (b)
demonstrates how the trained hypernetwork performs lifelong editing. Best viewed in color.
Meanwhile, the reward function R computes the reward r for this interaction:
knowledge, both derived from the previously mentioned in- put knowledge sample (x; y): Wo denotes the pre-trained pa- rameters of the LLM. Following prior hypernetwork-bae editing methods (Mitchell et al., 2022a: Tan et al., 2024; Zhang et al., 2024b), we introduce a coefficient Aioc to bal- ance the trade-off between these two terms:
R(s a.) Tb.
(6)
This process is illustrated in Figure 2(a). Through this modeling approach, we have successfully formulated the training process of hypernetworks in lifelong editing as an MDP. For more detailed proof, please refer to Appendix D.
Lbase = Le + Aloc Liee-
(8)
3.1.2. DESIGN OF THE REWARD FUNCTION
Memory Backtracking Component. lifelong editing, subsequent edits may interfere with previous ones, poten- tially degrading the effectiveness of earlier modifications. To address this issue and avoid focusing solely on current knowledge while ignoring previously edited knowledge, propose memory backtracking component L Specifi- cally, at time step t, we not only compute the basic loss of current knowledge (Iz, 3t) on the current LLM fw. ac- cording to Equation 8, but also calculate the loss of previous k knowledge samples (ar;, yi yi) on fw. as the back- tracking term using the same equation. In essence, each time step, simultaneously consider the basic losses of both current knowledge and the previous i pieces of knowl- edge on the LLM. Considering that previous knowledge has already been edited and only needs review rather than re-editing, we introduce a decay factor μl, which assigns weights to the losses from previous knowledge through ex- ponential decay based on temporal distance:
Next, we focus on the design of the reward function R mentioned above, which is crucial for RLEdit to achieve both efficient and effective editing. First, let's recall the fundamental objective of RL: to determine optimal policy for the agent, where π specifies which action a ~ (()) the agent should execute state s (Sutton & Barto, 1998). The reward function serves as a quantitative measure of policy performance. Specifically, the optimization objective of RL system maximize the total reward J over the entire trajectory. In this work, to align the RL optimization objective with the lifelong editing objective, we design multi-component reward function for RLEdit as follows.
Basic Component. We first incorporate two fundamental objectives of model editing as the basic components of the reward function: target knowledge updating and unrelated knowledge preservation. Formally:
Le log Pw (Ve xe):
1-1
Lioe = KL (+(xte)))) (-
(7)
Ltack,
Σ u (Frg.Wi1 + (...))o......)) b=--
where C. and Cioe measure the effectiveness of target knowl- edge updating and unrelated knowledge preservation, re- spectively; (Ie. ye) represents the equivalence neighbor- hood of target knowledge, and Toee represents unrelated
where Le, and Lio,,,._, denotes the loss calculated for the i-th batch of knowledge, evaluated using the LLM parameters Wi-1 from time step 1.

---

Reinforced Lifelong Editing for Language Models
Ablation studies in Section 4 demonstrate that the general- ization improvement from the memory backtracking compo- nent is one of the key factors enabling RLEdit to maintain effectiveness through more than 10,000 edits.
Algorithm RLEdit Hypernetwork Training
Input: Pre-trained LLM .NN. hypernetwork H with ini- tial parameter 0 hyperparameters k. 7, Aloc and n
Output: Optimized hypernetoork parameter 6' repeat
Regularization Component. Finally, to constrain the mag- nitude of updates Vw generated by the hypernetwork at each time step, we introduce the lg norm V|l'l? 2 of Ww as a regularization term. The ablation study in Section 4 shows two benefits of this regularization: First, limiting the magnitude of Vw minimizes the disruption to the orig- inal parameter distribution, thereby preserving the LLM's general capabilities; Second, it enhances training stability, ensuring the convergence of the hypernetwrk.
Randomly sample (Ii. Mi. Fe, Bes I Tlocs Di=1 for f 1==11o 1o do
L. -- -og log pw;-1 (yt (2g))
prok-propagate La and cache Vw,-
Vw, - W(Ww,
W; + Wi-1 + Vw: for i = t - k: to do Log C.. x- - log pwv (Ye, (Te,)
Cloc t KL (pwo (: (Tloe: )llww (: ( ))
Based on the above three components, the reward r, time step & formulated as:
end for
L. Ei Di=t-k (Co, + Aionsinos)
- (Laase, + Lback, + 1) VW: 113),
(10)
Tt +- - (C, + nll Vw.113)
end for - Er= 7'r
where 7) serves as the regularization coefficient. With this reward function, the hypernetwork- -base lifelong editing task has been completely formulated as an RI. problem.
Back-propagate and update 0 until hypernetwork convergence
return 0'
3.2. Training Process of RLEdit
The most straightforward training approach would be to opti- mize the policy (i.e., hypernetoork parameters 0) using gra- dient ascent after obtaining rewards via Equation 10 each time step. This online reinforcement learning algorithm, however, becomes computationally expensive when applied at each time step. given that training sets typically contain substantial knowledge to ensure generalization. Moreover, this approach may cause the hypernetwork to overfit specific knowledge samples, thereby reducing its effectiveness in generating general parameter updates. We therefore opti- mize collectively after traversing the dataset and collecting all rewards along the trajectory. More formally:
approach enables policy optimization from a higher-level sequential perspective. Ablation studies in Section 4 indi- cate that this update approach substantially enhances the hypernetwork's effectiveness in lifelong editing.
In summary, the training of RLEdit hypernetwork follows this process. At each time step t: (1) We collect gradi- ents Vw,,, from new knowledge sample (are yt) on LLM fw through one step of parameter- frozen fine-tuning; (2) Through hypernetwork H. we map Vw,,, ; to LLM parame- ter updates Vw, adding it to Wt-1 to obtain Wi: (3) Cal- culate reward rr on fw. using Equation 10; (4) Repeat steps (1)-(3) until traversing the entire training set, obtaining total reward J via Equation 12; (5) Optimize hypernetwork pa- rameters 0 using J through stochastic gradient descent; (6) Repeat steps (1)-(5) until hypernetwork convergence. This training process yields the final hypernttork parameters 6' that demonstrate strong adaptability to lifelong editing tasks. The pseudo-code is provided in Algorithm 1.
0' = arg max J,
(11)
n
J= -
(Coaee, + Cbock, + VW, I1²), (12)
where J denotes the total reward over the entire trajectory, 0' represents the optimized hypemnetwor parameters, and y represents the discount factor. Here, we set y = 1 to ensure that the importance of all knowledge entries across the entire sequence is uniformly weighted during the training phase.
3.3. Applying RLEdit for Lifelong Editing
After completing the training process in Section 3.2, given knowledge (ar,) to be edited, RLEdit only needs to per- form parameter-frozeen fine-tuning on LLM fw collect gradients Vw, feed them into the hypernetwork to generate Vw. and add it to W. as shown in Figure 2(b).
This training approach significantly accelerates hypernet- work convergence. We experimentally demonstrate that RLEdit achieves a 100-fold efficiency improvement over current lifelong editing methods primarily due to this strat- egy. After updates of the hypernetwork using J in Equation 12, it gradually learns how to edit LLMs with varying pa- rameter states. Morcover, after capturing the intrinsic rela- tionships between knowledge samples, our offline update
In this process, the hypemeework can typically be con- structed with a simple 4-layer MLP, yet effectively handles lifelong editing tasks with varying numbers of knowledge samples. We attribute this excellent performance to the
5

---

Reinforeed Lifelong Editing for Language Models
Table 2: Comparison of RlEdiit and baseline methods on lifelong editing tasks. The upper section represents fine-tuning and thitte-then-edit methods, while the lower represents bypernetwork-basee methods. MEND* and MALMEN* represent methods where the hypernetworks of MEND and MALMEN are retrained for each new knowledge editing.
FEVER
Methods
LLaMA-3-8B
Gemma-2-9B
Time|
Eff.
Gen.
Spe.
Etf.
Gen.
Spe.
Etf.
Gen.
Spe.
FT
1.80+0:10
6.39=a.17
26.33+o.15
34,23=031
29.22+0.30
34.23+0051
17.08+024
21.57=0.30
37.47=0.31
0.3358s
ROME
38.56=001
44.53=0029
9.29=017
30.07=006
13
10.79
0.00=000
0000
0.000m.0
6.0243s
MEMIT
0.18+
0.01=0001
0,00=a00
11.12=006
10.09+025
8.04=0021
0,00=a00
0.00=0.00
0.00+o.m
6.3423s
PRUNE
26.64=024
0.85=a0s
13.34=002
11.17
9,43+a017
5.27=0.34
3.11 16
5.89=023
6.3356s
RECT
60.95=027
52.40-0.26
1.75=0007
59.81=0025
54.89
0.05#0011
0,55
0.05=0001
0.00
6.0486s
AlphaEdit
94.22=005
94.14=0.18
25.57++.14
94.37
88.34
31.21 #0.54
32.74+042
30.03+00.9
8.44
6.2307s
MEND
0.00+0:00
0,00=0000
0.00=0000
50.93=0.3
50.76=0.77
0.36
0.00=a00
0.00=00000
0.00
0.9175s
MEND
10,37=001
9.34-+0.22
4.88=024
50.87=024
52.01
0.39
0,00=a00
0.00=0.00
0. 0.00+
6.1280s
MALMEN
0.01
0.01
0.14
94,56.00.9
91.94..0.16
68. 68.65 10.53
16.67..2.1
16.61
12.16.0.16
1.9858s
MALMEN
5.74+0.00
5.29=0013
1,26=a118
88.32=033
84.29
69.70
17.73+024
13.30#027
13.85
9.3358s
DAFNet
31.271047
28.82_o.4
66.5510.1
20.78
19.9910.35
13.1010..6
4.86
1.11
41.7110.45
8,2553s
RLEdit
95334=034
93
70.36
95.44=0211
92.83
68.76
88.99
88.25
73.64:0.11
0.2238s
ZsRE
Methods
LLaMA-3-8B
Gemma-2-9B
-B--
Timel.
Efr.
Gen.
Spe.
Eff.
Gen.
Spe.
Eff.
Gen.
Spe.
FT
17.10=022
16.73=022
8,27=013
12.90=020
13.09+0.0
0.07=a02
32.84+0 30
33.78+0
42. 12.39
0.3366s
ROME
0.54:10.4
0.57=2004
0.40=0002
3.45=2032
1.
2,
0.00=0000
00
0.00:20.00
6.0012s
MEMIT
0.00+0000
0.00=0000
0,13=ao2
5.23=a021
3,11=a012
5.98=012
0,00=a00
0.00=0.00
0. 0.13+oo.2
6.0677s
PRUNE
12
12.01 20:33
9,88=0229
10.21
8.88=
11. 11.95
0.00=
0.00=zo.
0.00=z.00
6. 15888
RECT
11.05=0.1
8.12##035
28.12
12.45=045
10.11
26.09.0044
8.18
8.04=2040
11.32+u.49
6.6558s
AlphaEdit
86,83=002
81.48=028
29.09+0.22
81.18-0.3
43.24+0.4
30.34
0.00=a00
0.00=g000
0.00
6.1831s
MEND
0.00+0:00
0,00=
0.00=0000
0.00=0000
0.00=aa00
0.00
0,00=0000
0.00=
0.00
0.9686s
MEND
0,00+0000
0.00==0.0
0,00=a00
8, 2.84
8,45+aa21
10.21
0,00=000
0.00=0.00
0.00+o.m
5.9265s
MALMEN
9.87.10.1
9.00
2.11
46.60=0.3
42.5010.32
19.664..85
0.00
0.00
0.00
2.2779s
MALMEN
12.23=aa1
11.08=o22
2.43=009
=6.97=024
44.28+022
15.02++0.1
0.01=000
0.02=0001
1.25
9.4277s
DAFNet
21.991047
11.17_0.4
32.21
5,941021
1.38
16.2910045
1,25
2.12.0012
25.2710.54
8,2383s
RLEdit
89.42:034
87.32:20.3
44.7810.50
84.37=0222
79.82:9.36
37.15:0.41
71.121431
67.42:0.27
27.43:0.44
0.2224s
enhanced generalization capability brought by RL.
4.1. Experimental Settings
Boosting Current Methods with RLEdit. Since RLEdit's superior efficiency and effectiveness primarily stem from its RL paradigm, it can serve as a plug-and-play module to in- tegrate with most existing hypemettork-bbat approaches. Specifically, hypernetwork-basee methods consist of three fundamental elements: (1) hypernetwork architecture, (2) selection of hypernetwork inputs, and (3) loss function for training the hypernetoork. For the first two elements, we can directly adopt their corresponding components from RLEdit. For the third element, we can use it to replace the basic term in RLEdit's ' reward function (Equation 10). Our experimental results in Section 4.6 demonstrate these optimizations, further validating the benefits of formulating hypernetwork training in lifelong editing as an RL paradigm.
We begin with a brief overview of the LLMs, datasets, evalu- ation metrics, and baseline methods used in our experiments. More detailed information can be found in Appendix A.
Base LLMs. We conduct experiments on three 8B-scale autoregressive LLMs: Llama-3-8B Gemma-2-98 (Team et al., 2024), and Mistral-7B-v0.3 (Jiang et al., 2023).
Datasets & Evaluation Metrics. We evaluate RLEdit on three widely-used datasets: ZsRE (Levy et al., 2017). FEVER (Thorne et al., 2018), and CounterFact (Meng et al., 2022). Following previous evaluation standards (Mitchell et al., 2022a; Meng et al., 2022; 2023), we use three metrics to assess editing success for each dataset: Efficacy, General- ization, and Specificity, as described in Appendix A.3.
Baseline Methods. We compare RLEdit against multiple editing methods, including FT (Zhu et al., 2021), ROME (Meng et al., 2022), MEND (Mitchell al., 2022a), MAL- MEN (Tan et al., 2024), DAFNet (Zhang et al., 2024b), MEMIT (Meng et al., 2023), PRUNE (Ma et al., 2024), RECT (Gu et al., 2024), and AlphaEdit (Fang et al., 2024).
4. Experiments
We conduct extensive experiments to evaluate both the ef- fectiveness and efficiency of our approach. Additionally, we perform ablation studies to analyze the contribution of each component in RLEdit, which can be found in Appendix B.
'htps:/llama.meta.cm http://llamammeea.ccmmaalaa
6

---

Reinforeed Lifelong Editing for Language Models
(a) 20x100
(b) 50x 100
Gereniloaioo
Convrilionize
MEND
Initial Setup Time
6924
Specifiitt
Specifiity
MALMEN
Editing Time
6029
DAFNE
6.004
Effecayy
Effracy
ROME
120
2.22
MEMIT
820
PRUNE
451
General Ciuiliiy
Geecra Capbility
RECT
2202
Elfxicsey
Efficssy
AlphaEdia
7800
(c) 100x100
(d) 150x100
Gerentiation
Gonoriliaatoon
RLEdi
Specifirry
Specifiity
3
Time per Edit (seconds)
Per-sample editing time comparison across meth- ods. Best viewed in color.
Figure 4:
Elficacy
Elfiaey
General Caplilit
Genera Capacilit
ically, RLEdit demonstrates average improvements of 66% in Efficacy, 65% in Generalization, and 40% in Specificity compared to baseline methods, while requiring only 4% of the time compared to most of them.
Eifiierry
RtLEdx
AlphaEaltt
RECT
Figure 3: Performance comparison of editing methods across multiple metrics: Efficacy, Generalization, Speci- ficity, General Capability, and Efficiency. (a)-(d) show re- sults for 20x 100, 50x 100, 100x 100, and 150x 100 config- urations. For General Capability, we measure that using the average of six GLUE metrics in Section 4.5; for Efficiency, we calculate the ratio between the total editing time of each method and that of the fastest method. Best viewed in color.
4.3. Editing with Varying Numbers of Knowledge
To verify RLEdit's versatility across different lifelong edit- ing scenarios, we investigated its performance under multi- ple configurations. We conducted extensive experiments on Llama-3-8B using 20x 100, 50x 100, 100x 100, and 150x 100 configurations on ZsRE dataset. The results are presented in Figure 3. We also performed experiments to investigate the impact of varying knowledge batch sizes on performance. Detailed results can be found in Appendix B.
4.2. Performance on Knowledge Update and Preservation
To evaluate the performance on both updating of target knowledge and preservation of unrelated information in lifelong editing tasks, we conducted sequential editing ex- periments comparing RLEdit against baseline methods on three LLMs, measuring performance across all metrics. We randomly sampled 8,000 knowledge samples from ZsRE and FEVER respectively. performing edits over 400 batches with 20 knowledge samples per batch (denoted as a 400 20 configuration throughout this paper). Upon completion of all batch edits, we evaluate all knowledge metries on the post-edited LLM. Results are presented in Table 2. The "Time" column in Table 2 indicates the average editing time per knowledge sample, including the time spent on covari- ance matrices computing or hypernetorr training.
Figure 3 shows that as the number of edits increases, base- line methods show progressive performance degradation, with catastrophic forgetting occurring beyond certain thresh- olds. RLEdit, however, consistently demonstrates superior performance across all configurations, outperforming base- line methods on most evaluation metrics. RLEdit represents the first method to successfully scale lifelong editing beyond 10,000 knowledge samples. Unlike existing approaches, RLEdit maintains stable performance as knowledge quantity increases, suggesting potential applicability to even longer sequences. These results demonstrate RLEdit's effective- ness for long-term lifelong editing tasks.
4.4. Editing Efficiency
In Table 2, conventional single-step methods (e.g., ROME, MEND, MEMIT) demonstrate poor performance, with Ef- ficacy and Generalization metrics falling below 50%, re- vealing their inability handle knowledge conflicts and forgetting issues in lifelong editing scenarios. Other sequen- tial editing methods (e.g., PRUNE, RECT, DAFNet) show varying limitations across datasets, failing to maintain satis- factory performance across all metrics. In contrast, RLEdit achieves superior results across all tested LLMs and datasets, maintaining strong performance in all three metrics. Specif-
To evaluate RLEdit's efficiency, we compared the average editing time per knowledge sample across most methods, Tests were conducted on 3 LLMs using ZsRE dataset un- der 20x 100 configuration while keeping all other variables constant (such the number of LLM editing layers) to en- sure fair comparison. For locate-then-edi methods, the time included covariance matrix computation, parameter update calculation, and edit execution. For hypernetwork based methods, we measured hypernetwork training, parameter update calculation, and edit execution time. Since covari-
7

---

Reinforeed Lifelong Editing for Language Models
KL
Alphalidtr
RECT
PRUNE
MEND" (c) MRPC
(a) MEND and Variants
(b) MALMEN and Varlants
1.00
(a) SST
(b) MIMLU
1.0
MUND
MALMEN
0.75
0.7
MEND
MALMEN-
Editing Rate
MEND-XLES
MALMEN -XLBR
0.50
0.4
0.24
0.00
(d) CoLAA
(e) KIE
(1) NLI
1.00
0.75
0.50
0.00
200
20m
1k
"Llana3
Misial Liana
Mistral
Num Edits
Figure 6: Performance comparison of hypernetwork- -based methods (MEND and MALMEN) with and without RI framework integration, including variants with hypernet- work retraining before each edit. Results presented in the figure show the weighted average of metrics across Coun- terFact and ZsRE, where Efficacy and Generalization assigned weights of 1, and Specificity is assigned a weight of 0.5. Best viewed in color.
Figure 5: LLM capability assessment using Fl scores on six GLUE tasks. (a)-(f) shows results for SST, MMLU, MRPC, CoLA, RTE, and NLI respectively. Best viewed in color.
ance matrices and trained hypemeeworks can be reused, we also analyzed editing time excluding these initial setup costs. Results are presented in Figure 4.
Hypernetwork-baat methods demonstrate superior speed compared locate-then-edif approaches, primarily due to better generalization capabilities. Locate-then-edit methods require complex matrix operations for each edit, resulting in slower execution. RLEdit requires only 0.145 seconds sample for training, outperforming other hypernetwork- based methods like MEND, MALMEN, and DAFNet. This efficiency derives from RL.Edit's sequence-basee approach for training, enabling better generalization with reduced training data. RLEdit achieves superior editing results while requiring only 1.14% of the time needed by locate-then- edit methods, demonstrating its exceptional efficiency and promising potential in long-term lifelong editing scenarios.
4.6. Improving MEND/MALMEN
While existing hypernetwork boaee editing methods focus on single-edit scenarios, RLEdit's RL framework and train- ing methodology specifically address lifelong editing chal- lenges, which could integrate with other hypernetwork- based methods a plug-and-play module. To demon- strate this portability, we conducted 20 100 editing ex- periments across 3 LLMS and 3 datasets, comparing MEND, MEND* MEND+RL MALMEN, MALMEN", and MALMEN+RLEdit Results are shown in Figure 6.
The results demonstrate RLEdit's portability across several methods. After incorporating RLEdit's 's training methodol- ogy, MEND and MALMEN showed average improvements of 22.89% and 51.86% respectively across all metrics, de- spite their initial suboptimal performance in lifelong editing tasks. This confirms RLEdit's effectiveness as plug-and- play module for enabling lifelong editing capabilities in both existing and future hypernetwork- based methods.
4.5. General Capability Tests
To evaluate how lifelong editing affects LLMs' general ca- pabilities, we assessed downstream performance using 6 tests from GLUE (Wang et al., 2019): SST (Socher et al., 2013), MMLU (Hendrycks et al., 2021), MRPC (Dolan & Brockett, 2005), CoLA (Warstadt et al., 2019), RTE (Ben- tivogli et al., 2009), and NLI (Williams et al., 2018). We measured F1 Scores Llama-3-8B using 7sRE dataset under 30x 100 configuration, and the results shown in Figure 5. Complete tests are available in Appendix B.
5. Related Works
Parameter-modiffin Editing. Parameter-modifyyee meth- ods modify LLM parameters through various approaches to achieve knowledge editing. One approach is locate-then-ede method. Knowledge-neurge (Dai et al., 2022) pioneered the use of causal analysis to locate knowledge in LLMs. ROME (Meng al., 2022) adopted this idea, using causal trace to identify factual associations and edit knowledge in MLP layers. MEMIT (Meng et al., 2023) extended this approach to mass-editing. PMET (Li et al., 2024) specifically investi- gated the role of MHSA in model editing. ECE (Zhang et al., 2025c) incorporates LLM explainability into the editing pro- cess and clusters similar knowledge based on explanation results. Another approach learnig..., KE (De Cao
Baseline methods show progressive degradation of general capabilities as edited knowledge samples increase, with some methods leading to complete performance collapse. This degradation likely stems from cumulative editing errors, caused by causal trace instabilities or imprecise hypernet- work training. In contrast, RLEdit maintains consistent per- formance across all tests, with LLMss edited for 3,000 knowl- edge samples performing comparable to their pre-trained versions. This stability, achieved through regularization, demonstrates RLEdit's s ability to both effectively edit target knowledge and preserve the model's general capabilities.
8

---

Reinforeed Lifelong Editing for Language Models
et al., 2021) proposed using hypernetwork for knowledge editing. MEND (Mitchell al., 2022a) applied low-rank de- composition to LLM fine-tuning gradients, enabling faster and more effective training in hypernetworks. MALMEN (Tan et al., 2024) aggregated parameter shifts using normal equations, extending hypernetworks to mass-editing.
Limitations
While RLEdit demonstrates promising results in lifelong editing tasks, several limitations should be acknowledged. Our evaluation methodology follows conventional datasets from existing model editing literature, primarily focusing on factual knowledge modifications without exploring other data domains. Furthermore, the capability of post-edited LLMs in processing multi-hop information remains unex- plored in our current study. Although achieving robust lifelong editing capabilities continues to pose significant challenges, our future work will extend these experiments, potentially providing valuable insights for advancing re- search in lifelong editing.
Parameter-preseevingg Editing. Parameter- Parameter-preserving methods guide LLM outputs by incorporating external mod- ules additional parameters. SERAC (Mitchell et al., 2022b) stores edits in additional memory and uses classi- fiers for edit matching. T-Patcher (Huang et al., 2023) edits knowledge by using additional neurons. IKE (Zheng et al., 2023) improves LLM's s in-context learning capabilities us- ing memory retrieval-based examples. OneEdit (Zhang al., 2024a) proposed a neural-symbolic prototype system for collaborative knowledge editing using natural language.
Impact Statement
RLEdit significantly enhances the capabilities of lifelong model editing, making it invaluable for updating and main- taining knowledge in real-world applications. Given that the ability to directly modify model parameters introduces potential risks, such as the injection of false or harmful in- formation, we strongly urge researchers to implement strict validation and oversight to ensure the ethical use of these techniques. The original goal of our work is positive, aiming to facilitate efficient knowledge updates in large language models. We encourage researchers leverage this technol- ogy responsibly while maintaining appropriate safeguards for its deployment.
Lifelong Model Editing. To meet the requirements of lifelong model editing, researchers have studied sequential editing methods. T-Patcher (Huang et al., 2023) achieves se- quential editing by continuously introducing additional cor- rection neurons. GRACE (Hartvigsen et al., 2023) enables sequential knowledge editing through dynamic updates of external codebook modules. WISE (Wang et al., 2024) uses a knowledge-shardee mechanism to optimize the balance of various metrics in sequential editing. RECT (Gu et al., 2024) studies the impact of regularization on sequential editing, using smaller parameter shifts to maintain stability. PRUNE (Ma et al., 2024) reduces disturbance to original knowledge by limiting singular values of the edit update matrix. O- Edit (Cai& & Cao, 2024) orthogonalizes knowledge update directions to reduce mutual interference. AlphaEdit (Fang et al., 2024) maintains knowledge stability by projecting existing knowledge into null space. NSE (Jiang et al., 2024) employs neuron-level sorting to selectively identify and up- date influential neurons, while utilizing weights rewinding to prevent model failures during sequential editing.
A Acknowledgement
This research is supported by the National Science and Technology Major Project (2023ZD0121 the Beijing Natural Science Foundation (4252023), and the Innovation Funding of ICT, CAS (E361 120).
References
6. Conclusion
Arulkumaran, K., Deisenroth, M. P., Brundage, M., and
Bharath, A. A. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
In this work, we represent RLEdit, a hypernetwork--aade editing method designed for lifelong editing. RLEdit for- mulates lifelong editing as an RL task, employing an offline update approach to enhance the model's retention of entire knowledge sequences. Additionally, RLEdit proposes the use of memory backtracking to review previously edited knowledge and applies regularization to mitigate knowledge forgetting over long sequences. Through extensive testing on several LLMs across multiple datasets, our experimental results demonstrate that RLEdit significantly outperforms existing baseline methods in lifelong editing tasks, show- ing superior performance in editing effectiveness, editing efficiency, and general capability preservation.
Bellman, R. A markovian decision process. Indiana Univ. Math. J., 6:679-684, 1957. ISSN 0022-2518.
Bentivogli, L., Magnini, B., Dagan, I., Dang, H. T., and Giampiccolo, D. The fifth PASCAL recognizing textual entailment challenge. In TAC. NIST, 2009.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C.,, Hesse, C., Chen, M., Sigler, E., Litwin, M.,
9

---

Reinforced Lifelong Editing for Language Models
Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Curran Associates, Inc., 2020.
Gu, J.-C., Xu, H.-X., Ma, J.-Y., Lu, P., Ling, Z.-H., Chang, K.-W., and Peng, N. Model editing harms general abili- ties of large language models: Regularization to the res- cue, 2024. URL https: /arxiv. org/ http//bb//// 2401. 04700.
Hartvigsen. T., Sankaranarayanan, S., Palangi, H.,, Kim, Y., and Ghassemi, M. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Oh, A., Nau- mann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 47934 47959. Curran Associates, Inc., 2023.
Cai, Y. and Cao, D. O-edit: Orthogonal subspace editing for language model sequential editing, 2024. URL https: //aa//aia /aarxiv. org/ abs/ 2410 11469.
Chen, C., Huang, B., Li, Z.. Chen, Z., Lai, S., Xu, X., Gu, J.-C., Gu, J., Yao. H., Xiao, C., Yan, X., Wang, W. Y., Torr, P., Song, D., and Shu, K. Can editing lims inject harm?, 2024a. URL https: /ttp//a.vvv.o/////////.a org/ abs/ 20224.
He, J., Zhu, K., Guo, H., Fang, J., Hua, Z., Jia, Y., Tang, M., Chua, T.-.., and Wang, J. Cracking the code of hallucination in Ivims with vision-aware head divergence. arXiv preprint arXiv:2412. 13949, 2024.
Chen, S., Cai, Y., Fang, H., Huang, X., and Sun, M. Differ- entiable neuro-symbolic reasoning on large-scale knowl- edge graphs. In NeurIPS, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https: openreview net forumorrummmm 3m
Chen, S., Zhang, Q., Dong, J., Hua, W., Li, Q., and Huang, X. Entity alignment with noisy annotations from large language models. In NeurlPS, 2024b.
Huang, B., Chen, C., Xu, X., Payani. A., and Shu, Can knowledge editing really correct hallucinations?? 2025. URL https: / www.i..ooro....... abs wtt.ab///41.20 22102241.11.1
Dai. D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/1/22022. acl-long.581. URL pttttp:://ccaaatttee 2022.acl-1cs 581//
Huang, Z., Shen, Y., Zhang, X., Zhou, J., Rong, W., and Xiong, Z. Transformer-patcher. One mistake worth one neuron. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openreview.coo -oot://roouuunoooo0G00o---a--
org/
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //arxiv. org aaa//b//222 06825.
De Cao, N., Aziz, W., and Titov, I. Editing factual knowl- edge in language models. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. pp. 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/1/2021. emnip-main.522. URL org/ wwnleeeemmmmmeeeeee cannp-mminn.. 522n.522/.
Jiang, H., Fang. J., Zhang, T., Zhang, A., Wang, R., Liang, T., and Wang. X. Neuron-level sequential editing for large language models, 2024. URL https: /tatps://arxiv... org/ abs///////// / /21124100 00.100000 04045.
Dolan, W. B. and Brockett, C. Automatically construct- ing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https:: attps://aclantholoogy org/ 1055-0055
Jiang, H.. Fang. J., Zhang, N., Ma, G., Wan, M.. Wang, X., He, X., and Chua, T.s. Anyedit: Edit any knowl- edge encoded in language models. arXiv preprint vrXh:2252.006222 2025.
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., de Masson d' d'Autume, C., Kocisky, T., Ruder, S., Yogatama, D., Cao, K., Young, S., and Blunsom, P. Mind the gap: Assessing temporal generalization in neural language models. In Ranzato, M.,
Fang, J. Jiang, H., Wang, K., Ma, Y., Wang, X., He, X., and Chua, T. Alphaedit: Null-space constrained knowledge editing for language models. CoRR, abs/2410.02355, 2024.
10

---

Reinforced Lifelong Editing for Language Models
Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Process- ing Systems, volume 34, pp. 29348-29363. Curran Asso- ciates, Inc., 2021.
Puterman, M. L.. Markov Decision Processes: Discrete
Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994, ISBN 0471619779.
Socher, R., Perelygin, A., Wu, J., Chuang. J., Manning. C. D., Ng, A., and Potts, C. Recursive deep models for Semantic compositionality over a sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., and Bethard, S. (eds.), Proceedings of the 2013 Confer- ence on Empirical Methods in Natural Language Process- ing. pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguusties URL wtt.:////rr///1101100000
Levy, O., Seo, M., Choi, E., and Zettlemoyer, L. Zero- shot relation extraction via reading comprehension. In Levy, R. and Specia, L. (eds.), Proceedings of the 21st Conference on Computational Natural Language Learn- ing (CoNLL 2017), pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguis- URL https://
tics. doi: 10.18653/11KK171100 oww.anttoooy..ogg/11111-1---1
Sutton, R. and Barto, A. Reinforcement learning: An intro- duction. /EEE Transactions on Neural Networks, 9(5): 1054,005,, 1998. doi: 10.1109/TNN.199 MAAN.1998.711 712192.
Li, X., Li, S., Song, S., Yang. J., Ma. J., and Yu, J. Precise model editing in a transformer. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, (8(17):18564-1855 baai.v3811..29818 URL http::: //ojs aaai.org/ index. php/ AAAI/ hat.A//artileevvieeeee
Pmee::
Mar. 2024. doi: 10.1609/
Tan, C., Zhang, G., and Fu, J. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representations, 2024. https:/ id id=16LICJQ2PE.
/ /tpp///veeeeee.ee////eeee neteeeewwwweeeeeeeeeeeee.e.eeeee forrm??rmm
Liu, Y., He, X., Xiong, M., Fu, J., Deng. S., and Hooi, B. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832, 2024.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., and et al., C. H. Gemma 2: Improving open language models at a practical size, 2024. URL https: ttttts://arxivee /ttp//xxxvv.og//////./././..
Ma, J.-Y., Wang, H., Xu, H.-X., Ling, Z.-H., and Gu, J.-C. Perturbation--restrainddd sequential model editing, 2024. URL /tactts://arxiie org/ 2405 1 16821.
abs/ 2408. 00118.
Thorne, J., Vlachos, A., Christodoulopoulls, C., and Mit- tal, A. FEVER: a large-scale dataset for fact extrac- tion and VERification. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long Papers), pp. 809-819, New Or- leans, Louisiana, June 2018. Association for Computa- tional Linguistics. doi: 10.1 N0.18653/v1/N188 1074. URL https://aclattai http://aaaaattolaaooeoooo org/ N18 1074//
Meng, K., Bau, D., Andonian, A., and Belinkov, Locat- ing and editing factual associations in gpt. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Pro- cessing Systems, volume 35, pp. 17359-17372. Curran Associates, Inc., 2022.
Meng, K., Sharma, A. S., Andonian, A. J., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Represen- fations, 2023. URL https:/ /oppereeevviieeeeeeeeeeeeeeeeeee ew. net. Dorum?id-MKbcAHTYS
Wang, A., Singh, A., Michael, J., Hill, F,, Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In International Conference on Learning Representations, 2019. URL https:: haap:::///ppereeeee www.reeeeee.nneeeeeeeeee nwt.:foruttf forum? id=rJ4km2R5t7.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, D. Fast model editing at scale. In International Confer- ence on Learning Representations, 2022a. URL https: //openreview net httpa//reeeee.eeeeooooommooeeoed
Wang. P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., and Chen, H. Wise: Rethinking the knowledge memory for lifelong model editing of large language models, 2024. URL /ttps://./xx/v.//////// abs 0tt./2405..4475
Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. Memory-based model editing at scale. In Chaudhuri, K., Jegelka, S., Song. L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.), Proceed- ings of the 39th International Conference chine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pp. 15817-15833 PMLR, 17- 23 Jul 2022b. URL https: http:////oocedddee mml.. press/vi wmmmmmmmmmme1222/ html.
Ma-
Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 7:625-641, 2019, doi: 10. 1162/tacl URL https: / aclanthooogy. org/q19-104C/:

---

Reinforeed Lifelong Editing for Language Models
Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10. 10.18653/v1/N18- 101. URL https: //aclanthology org.e/ooooooooggoooo. /18 1101/.
92. URL https: /ttps://acllaaaa F Findings-ace .2/
Zhang. T., Fang. J., Jiang, H., Bi, B., Wang. X., and He, X. Explainable and efficient editing for large lan- guage models. In THE WEB CONFERENCE 2025, 2025c. URL https:ttt/////mmmm peaett:::/opeeeeviiie wttpeviwwwwweeeeeeeeeeeeeeeeee forum? id id-iAn7r1IEgo
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang. J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X.,
Yang. W. Sun, F., Ma, X., Liu, X., Yin, D., and Cheng, X. The butterfly effect of model editing: Few edits can trigger large language models collapse. In Ku, L.WW.,K Mar- tins, A., and Srikumar, V. ), Findings of the Associa- tion for Computational Linguistics: ACL 2024, pp. 5419- 5437, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 18653881120244 findings-acl.322. URL https: aclaathologg...... 20g/
Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models, 2024. URL https: / /arxiv. 8223.
org/ abaaaba220000
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., and Chang, B. Can edit factual knowledge by in-context learning? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: / opeenreview. net / forum? Suuuumidhhhs jOHAMBMV.
Yang. W., Sun, F. Tan, J., Ma. X. Su. D.. Yin, D., and Shen, H. The fall of ROME: Understand- ing the collapse of LLMs in model editing. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Find- ings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 4079-4087, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 10..8655/1120044..dddnn----m 236. URL https:://// / org.0/24....//222.....// -
Zhou, Z., Yu, H., Zhang, X., Xu, R., Huang, F., Wang, K., Liu, Y., Fang, J., and Li, Y. On the role of attention heads in large language model safety, arXiv preprint arXiv:2410.13700 2024.
Zhu, C.,, Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., and Kumar, S. Modifying memories in transformer models, 2021. URL https: btttps:/opeereve// hppp//eeeeeeeeeeeeeeeeee/ee / forum? forum?id=Kuise d=kumiidd=uubbaadd5...
Zhang, N., Xi, Z., Luo, Y., Wang, P., Tian, B., Yao, Y., Zhang, J., Deng, S., Sun, M., Liang, L.., Zhang, Z., Zhu, X., Zhou, J., and Chen, H. Oneedit: A neural-symbolic collaboratively knowledge editing system, 2024a. URL https: //arx/aaavvvva/// org/ abs 2409 07497.
Zhang, Q. Chen, H., Dong, J., Chen, S., Huang, F., and Huang. X. cauutture-gudee large language models for text-to-SQL generation. In Forty-second International Conference on Machine Learning, 2025a.
Zhang, Q., Chen, S., Bei, Y., Yuan, 7., Zhou, H., Hong, Z.,, Dong, J., Chen, H., Chang, Y., and Huang, X. A survey of graph retrieval-aus augmenntd generation for customized large language models. arXiv preprint arXiv:2501. 13958, 2025b.
Zhang, T., Chen, Q. Li, D., Wang, C., He, X., Huang, L., Xue', H., and Huang, J. DAFNet: Dynamic aux- iliary fusion for sequential model editing in large lan- guage models. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Findings of the Association for Computa- tional Linguistics: ACL 2024, pp. 1588-1602 Bangkok, Thailand, August 2024b. Association for Computa- tional Linguistics. doi: 20.18653/v1/2222 ffnc...niings-ac...
12

---

Reinforced Lifelong Editing for Language Models
A. Detailed Experimental Setup
In this section, we elaborate on our experimental setup, which consists of five parts: baseline methods, datasets, evaluation metrics, GLUE benchmarks, and hyperparameter configuration. Most experiments were conducted on a single NVIDIA A100 (80GB) GPU. Editing time for all methods was measured using LLMs in half-precision mode. To better simulate real-world applications, we used the instruction tuning versions of LLMs.
A.I. Baseline Methods
We utilized the code from AlphaEdit and MALMEN to evaluate the performance of baseline methods. The baseline methods used in this paper are as follows:
FT-L (Zhu al., 2021) is a knowledge editing approach that focuses on fine-tuning specific layers of the LLM through autoregressive loss. We implemented this baseline method using the hyperparameter settings from the original paper.
MEND (Mitchell et al., 2022a) is an efficient editing method based on hypernetorrs. trains a hypernetwork to learn patterns in knowledge editing by mapping low-rank decomposed fine-tuning gradients to LLM parameter updates. This approach enables efficient and localized knowledge editing. We implemented this baseline method using the hyperparameter settings from the original paper, completing training over the entire training set. Additionally, we introduce MEND* as a baseline. To address the mismatch between the initial hypernetwork and post-edited LLM in lifelong editing scenarios, we periodically retrain the hypernetwork using post-edited parameters. We adopted a strategy of retraining the hypernetwork every three editing batches.
ROME (Meng et al., 2022) is a method for updating specific factual associations in LLM parameters. It identifies key neuron activations in MLP layers through perturbation-based knowledge localization, then modifies MLP layer weights by computing Lagrange remainders to edit knowledge. Since ROME doesn't 't support massive editing, we followed the original paper's configuration and evaluated it through multiple batches of single editing.
MEMIT (Meng et al., 2023) is a method supporting large-batch knowledge updates. Building upon ROME's modeling approach, MEMIT extends it by using least squares approximation to directly manipulate parameters at specific layers, enabling multi-layer updates. This allows MEMIT to simultaneously update hundreds or thousands of knowledge facts. We evaluated MEMIT's performance in lifelong editing using the original configuration from its paper.
MALMEN (Tan et al., 2024) is a hypemeetwor--bure -assed method designed for massive editing. To aggregate parameter shifts across large batches of knowledge, MALMEN employs a least squares approach, deriving optimal parameter shifts by solving normal equations. This algorithm effectively addresses knowledge conflicts in massive editing scenarios. We implemented this baseline method using the original paper's hyperparameter configuration and completed training across the entire training set. Similar to MEND, we also introduce MALMEN* as a baseline.
DAFNet (Zhang et al., 2024b) is a model editing method specifically designed for sequential editing. It features dynamic auxiliary fusion network that enhances semantic interactions between knowledge triples in the sequence, enabling continuous mistake rectification. Through this auxiliary network, DAFNet improves the performance of hypernetwork approaches in sequential editing tasks. We implemented this baseline method using the original paper's hyperparameter configuration and completed training across ZsRE and CounterFact datasets.
PRUNE (Ma et al., 2024) is an editing method focused on sequential editing scenarios. By imposing conditional restraints on edited matrices, PRUNE limits the interference of new knowledge on previously stored model knowledge, thereby addressing the problem of model performance decline during multiple sequential edits. We implemented this baseline method using the perraarraetter configuration from their original paper.
RECT (Gu et al., 2024) is an editing method designed to minimize the impact of editing on LLM's general capabilities. It investigates the role of regularization in lifelong editing and prevents editing overfitting by regularizing weight updates during the editing process. This enables RECT to achieve high editing performance while maintaining the LLM's general capabilities. We implemented this baseline method using the hyperparameter configuration from their original paper.
AlphaEdit (Fang et al., 2024) is an editing method aimed at mitigating knowledge disruption in LLM lifelong editing. By introducing the concept of null space, AlphaEdit projects parameter updates onto
null space
13

---

Reinforced Lifelong Editing for Language Models
before applying them, thus reducing interference between different knowledge updates. AlphaEdit has been proven achieve SOTA performance across multiple metries while maintaining strong transferability. We implemented this baseline method using the hyperparameter configuration from their original paper.
A.2. Datasets
Next, we introduce the datasets used in this paper.
ZsRE (Zero-shot Relation Extraction) (Levy al., 2017) dataset serves benchmark dataset in the field of language model editing research. The dataset's structure incorporates three distinct components for each entry: primary question and its corresponding answer intended for editing purposes, multiple paraphrased variations of the original question created using fack-translation techniques, and locality questions that are semantically unrelated to the original query. This comprehensive structure enables researchers to evaluate model editing performance across three critical dimensions: accuracy in incorporating new information, robustness when faced with differently worded but semantically equivalent queries, and precision in maintaining unrelated knowledge without interference. For locate-then-edit methods, we the version from MEMIT; for hypernetwork-basee methods, we the version from MEND, where ZsRE is divided into training and test sets for hypernetwork training and editing performance evaluation respectively.
CounterFact (Meng et al., 2022) represents an advanced dataset specifically designed evaluate language models' ability to handle contradictory factual information. The dataset's distinetive feature lies in its of false statements that require correction, making it particularly challenging since models typically provide incorrect answers before editing. Each entry in the dataset contains three elements: an original false statement requiring editing, semantically equivalent rephrased versions of the statement, and unrelated statements for locality purposes. For locate-then-edit methods, we use the version from MEMIT; for hypemetwork- based methods, we also use the version from MEMIT and divide it into training and test sets, each containing approximately 10,000 knowledge instances.
FEVER (Fact Extraction and VERification) (Thorne et al., 2018) dataset is comprehensive dataset for fact-checking tasks, constructed through systematic modification of Wikipedia content. The dataset contains claims that were created by altering original Wikipedia sentences and then independently verified without reference to their source material. In its structure, FEVER implements a three-category classification system: claims can be marked as Supported, Refuted, or NotEnoughInfo. For claims classified as either Supported Refuted, the dataset includes supporting evidence sentences that justify the classification decision. When used in editing tasks, the dataset is often simplified binary classification problem, where the editing targets are equally distributed between two possible labels (1 and O), representing the veracity of the claims. For locate-then-edii methods, we extract the subject from queries to adapt to (s, o) modeling; for hypernetwork- based methods, we use the version from MEND, where FEVER is divided into training and test sets.
A.3. Metrics
1.3.1. ZSRE METRICS
Following previous research (Meng et al., 2022; Mitchell et al., 2022a), we evaluate various model editing methods using standard metrics on the ZsRE dataset. Specifically, given an LLM fw. an editing knowledge pair (x, y). equivalent knowledge Ze, and unrelated knowledge pairs (Ntees Mooe), we examine the following three metries:
Efficacy. This metric measures the success rate of editing knowledge in fw- It compares the top-1 logits output y' (z) with the target output y when inputting x into fw:
{ arg max PP / (s' [=)]
(13)
Generalization. This metric measures the success rate of editing equivalent knowledge xe in fw- It evaluates whether the LLM has truly learned the intrinsic relationships of the knowledge and can extend to other equivalent knowledge. We compare the top-1 logits output y' (wee with the target output y when inputting xg into fwi
E = arg max PSw (3/ (2/1 }
(14)
y'
14

---

Reinforced Lifelong Editing for Language Models
Specificity. This metric measures the retention rate of unrelated knowledge Tioc after editing, examining whether the knowledge editing maintains locality and only modifies the target knowledge. We compare the top-1 logits output y' fww(2to) with the original output Vloc when inputting Tiee into fw:
IE {
Vloc arg max fw (v/ (zioc) }
(15)
A.3.2. COUNTERFACT METRICS
Similarly, following previous research (Meng et al., 2022; 2023), evaluate various model editing methods using standard metrics on the CounterFact dataset. We follow ROME'S original setup, comparing the probabilities of different answers in the logits for measurement. Specifically, given an LLM fw, an editing knowledge pair (z, y). original knowledge pair (zr 30 equivalent knowledge Te, and unrelated knowledge pairs (Nloes Moc), we examine the following three metrics, calculated following ROME and MEMIT:
Efficacy. This metric measures the success rate of editing knowledge æ in fw: We compare whether the probability of the target output y is higher than the probability of the original answer yo in the logits when inputting x into Jw:
E Jw ly | > P/w [yo |x]]-
(16)
Generalization. This metric measures the success rate of editing equivalent knowledge ze in fw. We compare whether the probability of the target output y is higher than the probability of the original answer in the logits when inputting Te into fw:
E PP fw ly [ze]]
> Pyw [vo []e
(17)
Specificity. This metric measures the retention rate of unrelated knowledge Tioc after editing. We compare whether the probability of the original answer Yooc is higher than the probability of the edited output in the logits when inputting Floe into Jw:
E [P fw [Niio [2tee > P.w [y) [2o0e 11.
(18)
A.3.3. FEVER METRICS
We also evaluate various model editing methods using standard metrics on the FEVER dataset. Specifically, given LLM fw. an editing knowledge pair (ar, v), equivalent knowledge T.. and unrelated knowledge pairs (Nloe; Mloc), we examine the following three metrics:
Efficacy. This metric measures the success rate of editing knowledge x in fw. It compares whether the top-1 logits output y' [ww(z) matches the target output y when inputting x into sw:
arg maxmaxP Iw (2' (s [t)] E y
(19)
V'
Generalization. This metric measures the success rate of editing equivalent knowledge x. fw. Since we need to examine whether a knowledge edit is truly successful, must verify if the LLM has genuinely learned the intrinsic relationships of the knowledge and can extend to other equivalent knowledge. We compare whether the top-1 logits output y/ [w((tte matches the target output when inputting Ie into fw:
E y arg max P Fw (3/ |mc)) (1/l
(20)
Specificity. This metric measures the retention rate of unrelated knowledge Toee after editing, examining whether the knowledge editing maintains locality and only modifies the target knowledge. We compare whether the top-1 logits output y' [w((toc) matches the original output Vloe when inputting Fioe into fw:
Mloe = arg max fw (v/ (210c) }.
E
(21)
3/
15

---

Reinforced Lifelong Editing for Language Models
A.4. GLUE Benchmark
GLUE (General Language Understanding Evaluation) (Wang al., 2019) benchmark is collection of resources for training. evaluating, and analyzing natural language understanding systems. We selected 6 metrics from this benchmark to evaluate how well different methods maintain general language capabilities.
Stanford Sentiment Treebank (SST) (Socher et al., 2013) is dataset consisting movie review sentences with their associated sentiment labels. This binary classification task requires models categorize the sentiment expressed in each individual sentence.
Massive Multi-task Language Understanding (MMLU) (Hendrycks et al., 2021) a comprehensive benchmark designed to assess language models' capabilities across multiple domains. It specifically evaluates model performance in zero-shot and few-shot learning scenarios.
Microsoft Research Paraphrase Corpus (MRPC) (Dolan & Brockett, 2005) serves as a benchmark for evaluating semantic similarity. The task challenges models to identify whether two given sentences convey the same meaning.
Recognizing Textual Entailment (RTE) (Bentivogli et al., 2009) examines logical relationships between sentences. The task requires determining whether a given premise sentence logically implies hypothesis sentence.
Corpus of Linguistic Acceptability (CoLA) (Warstadt al., 2019) focuses on grammatical judgment. This single- sentence classification task uses sentences extracted from linguistics publications, requiring models to distinguish between grammatically acceptable and unacceptable sentences.
Natural Language Inference (NLI) (Williams et al., 2018) evaluates natural language understanding capabilities. The benchmark requires models analyze sentence pairs and determine their logical relationships.
A.5. Hyperparameter Configuration
Now we describe the hyperparameter configurations in our experiments.
For the hyperparameters in RLEdit training and editing, we set the memory backtracking decay factor μ to 0.95, the backtracking depth &: to 10, the regularization coefficient 17 to le-4 and the discount factor 7 in the total reward formula. Additionally, the initial learning rate was set to 1e-6, while the meta-learning rate set to le-5. The specific hyperparameter configurations for different models and datasets are shown in Table 3, where rank refers to the rank of linear transformation in hypernetwork, loc_coef refers to the weight coefficient Atoe of the locality loss function Lise- Table 3 represent settings that were empirically selected based on their strong experimental performance. Our analysis revealed that the performance of RLEdit notably sensitive to the choice the "Layer" parameter, while exhibiting limited sensitivity to variations in "Rank" and "oo__coee".
Table 3: The specific hyperparameter configurations for different models and datasets.
Datasets
Models
Layer
Rank
loc_coef Atoe
Llama-3-8B
gate 1-5], up[18-24
1024
0.6
ZsRE
Gemma-2-9B
gate[32-40), up[32-40]
512
0.6
Mistral-7B
down[17, 18]
1024
0.8
Llama-3-8B
gate[22-30], up[22-30]
512
0.6
CounterFact
Gemma-2-9B
gate[32-40), up[32-40]
1024
0.6
Mistral-7B
177, 18, 19]
1024
0.8
Llama-3-8P
gate[22-30), up[22-30]
1024
0.6
FEVER
Gemma-2-98 gate[32-40), up[32-40]
1024
0.6
Mistral-71
down[17, 18]
1024
0.6
16

---

Reinforced Lifelong Editing for Language Models
B. More Experimental Results
In this section, we present additional experimental results.
B.1. Ablation Study
To assess the contribution of each component in RLFdit, we conducted ablation studies on models and 3 datasets by removing the RL training framework, memory backtracking, and regularization respectively. Table 4 shows the contribution of each component to RLEdit under the 20 100 task.
Table Ablation Study Results for RLEdit.
CounterFact
ZaRE
FEVER
Model
Dataset
Method
Eff.1
Gen.t
Spe.1
EfT.1
Gen.1
Spe.t
Eff.
Gen.7
Spe.t
RLEdit
91.75
62.40
52.38
88.65
83,91
47.61
94.46
91.56
69:01
Llamn-3
wlo RL training framework
50,30, 41.45
49.55, 1233
49.631
0.62; 58.00
0.53
2.36 45.25
21.28, 73.18
21.15; 70411
13.63,
w/o memory backtracking
191
60.42; 195
20.23
88, 88.62) 0.03
81.23,
44.43,
93.02;
87,29,422
68.87.014
w/o rogularization
92.12-037 0.3
60.99 L41
02.45
88,63 0.02
83.12; 0.79
48.43r 0.2
94.011
91.58; 0.02
68.68....3
Gemma-2
Riltt
90.11
61.34
48.57
89.22
79.85
35.57
95.13
91,70
71.83
wlo RL truining framework
18.13,
20.73,
80.49h 31.92
12.98,
10.49; 30
9.12;
0.00,
0.00; 91.70
0.00, 71:0
w/o memory backtracking
077
60.93 6.41
87.13 2.09
78.29; 1.56
35.27
35,10; ous
90.98, (.72
69.28.2.55
w/o regularization
89.87,024 024
6
48.74 0.17
89.21 0.01
77.25; 2.60
33.60) 1.97
94.95; 0.19
92.33+ 0x3
73.98 2:15
RLEddit
84,24
63,93
60.79
84.60
78.00
50.18
97.78
96.34
83,71
Mistral
w/o RL truining framework
50.95, 33.29
1400
51.1119.88
6.66, 77.94
21.42
2.10, 48.00
46.38,
45.26, $1.00
30.16,
w/o memory backtracking
2.11
60.51,
58.99, 1.30
76.89, 1.11
48.47, 1:11
97.10; o.sa
94,29. 2:05
81.
w/o regularization
84.31 0.07
63.89, 0.04
59.881
85.17
77.73,027
48.211
97.79h a.m.
96.28. 0.06
82.92. 0.79
As observed, the RL training framework in RLEdit is crucial for adapting hypernetworks to lifelong editing. After ablating the RL training framework, RLEdit's performance metrics significantly decrease, essentially losing its lifelong editing capability. Memory backtracking effectively enhances long-sequence editing performance, as evidenced by the decline in Efficacy and Generalization metrics when it is ablated. Regularization primarily serves to maintain L.LM's general capabilities after sequential editing, as ablating regularization results in decrease in the post-edited LLM's general capabilities.
B.2. More Results of Editing with Varying Numbers of Knowledge
To evaluate the generalization capabilities of RLEdit across various editing tasks and sequence lengths, we conducted additional experiments. Specifically, examined the editing performance tasks of dimensions 20x 100, 50x 100, 100x 100, and 150x 100 using Llama-3-8B on three datasets², as shown in Table 5.
Comparisons with baseline methods demonstrate that RLEdit maintains relatively stable editing performance across knowledge sequences of any length, while most baselines lose their effectiveness beyond 5,000 samples.
B.3. General Capability Test on CounterFact
To comprehensively analyze RLEdit's effectiveness in maintaining LLM's general capabilities, to complement the down- stream task evaluations on ZsRE presented in Section 4.5, we conducted GLUE task testing on CounterFac Fact for all methods. All experiments were performed on LLamm-3-8B, as illustrated in Figure 7.
The results demonstrate that on CounterFact, LLMs edited with RLEdit for 3,000 knowledge samples maintained general capabilities comparable to the pre-trained model, whereas most other methods showed significant degradation in performance.
B.4. Impact of Different Edit Frequencies and Batch Sizes on RLEdit
In lifelong editing, the frequency of edits and batch size are crucial parameters that significantly impact editing success rates. In this section, we investigate how these parameters affect RLEdit's performance. Using the ZsRE dataset, we edited 5,000 knowledge samples on Llama-3-8B with seven different configurations: 10x500, 220, 50x 100, 100x50, 200x 25,
Since RI.Edit and other hypernetwork-haaeee methods require least half of the dataset training data, and given the limited size of the CounterFact dataset, we could only test knowledge editing up to 10,000 samples in the table.
17

---

Reinforced Lifelong Editing for Language Models
Table 5: Lifelong editing results under different numbers of edited knowledge samples. denotes locate-then-edit methods while denotes hypernetwork-basee methods. The best results are highlighted in bold, while the second-best results underlined.
COUNTERFACT
ZsRE
FEVER
TASK
DATASET
METROD
EFF.
GRN.1
SPE."
EFF-1
GEN.1
SPE.1
EFF.1
GEN.T
SPE.1
FT
13
67. 27.79.0040
46.6320.39
30.54.0m
20
15.49.0018
a.35
6.00
24. 10
20 x 100 20000
MEND
49. 49.20.00.3
10.0.3
50.0520.25
0.00
0.00
0.00..0.0
37. 47.75
37.7840.30
29. 10
ROME
64.45
61.42=0.44
49.46s0.40
2.00sa.11
1.68
0,68
MEMIT
65.0520.44
04.6
52.33
57.7220:17
52.48.0.11
25.78.20
PRUNE
68.25
64.75.0.01
49.82...24
24.77:10.7
23.871008
20.69::0:4
RECT
64.00=0.4s
61.20=0.43
60 88
86.02=9.24
81.81
32.04
ALPHAEDII
98.90
94.22:0011
67.88
94.47
91.13
32.55
RLEDIT
91.75=0:1
62.40:a0.s
$2.38
88.65
83.91
47.61
94.46:0.18
91.56::0.29
69.01
FT
82. 10
66. 06.08
40.35
21.05
20, 80
9.69
8.40
6.01 116.35
23.69 15.
MEND
49.71
49.63 29.63
50.23
0.00
0.00=0.00
0.00
a3.49
28.26+0.22
26.77=0.35
5000
ROME
48. 48.62a0.50
49.78.0.48
$1.65
1,25ga.11
1.30
1.65
MEMIT
64.21
60.07:0.43
46.64
0.07
0.07
31.67
PRUNE
56.07g0.M 14
54.58.0.1
50.32g0.3 as
g.09
0.08.0.0
1.35
RECT*
52. 04.00
50.02:a.45
53,59.0.42
0.07
0.07
1.34
50
ALPHAEDII*
94.66:0115
92.35=0.32
62.00
93.76=0.15
88.65=e21
31.71
RLEDIT
85.54
60.96.0.27
48.60
85.46
80.68:0.44
43.35
94.37
91.84:0.23
67.58
FT
79. 00.00
62.02=0.29
38. 38.86.0.36
14.80=0.22
14.50=0.22
5.22=0010
23 02.0.2
8.38
22. 16
100 10000
MEND
47.24
47.1 47.19
53.10
0.00
0.00
0.00
27.24
28. 28.93...32
27.56-0.34
ROME
46.43=0039
40.99=0.31
46.68
1.87a0112
1.8420.12
1.65=0.06
MEMIT
47.53
43.59..0.4
$1.3020.44
0.00
0.00:..00
0.00
PRUNE*
28.29.0.23
48.01
10.91
0.09
0.06:2002
1.26
RECT*
49.95
50.1 1
46.27
0.08
0.08
1.3620004
100
ALPHAEDD*
75.34:0.45
$2.78
89.60=0.17
86.75:e023
30.96:0.22
RLEDIT
80.69
62.31
45.1620.34
86.45
83.07
41.96:0.34
95.06:0.21
92.26:0.15
69.19.0.30
FT
89.79_0.43
70.05
36.64g0.22
13.95 n0
13.49
3.93 os
44 14.844
32. 19
33.54-044
150 XX 100 15000
MEND
0.00
0.00:00.0
0.00
21. 21.70:0.4
20.30:0.24
13.86.0.25
ROME
46.51=0.50
47.10=0.47
53.36:0.45
1.47a0.11
1.43:0.11
0.72=0.06
MEMIT
52.07.045
50.232004
48 20.47
0.00g0.00
0.00..000
0.00 t0
PRUNE
52. 12.19
$2.87=0.39
44.43
0. 0.00
0.00=0.00
0.00=0.00
RECT
54.49
53.24=0.31
46 26.22
0.02#0.01
0.03-2.01
1.02=0.21
ALPHAEDII
09.3
69.28:0.17
42.01:0.55
84.43+0025
78,
38.28:0030
28.48
RLEDIT
87.81
85.13
40.98
95.11
91.85
68.59_m.29
FT
86.23
69.77
33.94
13.21
11.11
2.24
55. 29.3
50.07=0.52
49.79=0.30
001 = 10000
MEND
0.00
0.00=0.00
0.00
11.08
8.01
15.38=0.51
ROME
44.43:0.1 39
46.21
50.77.0.33
1.5620a:
1.34
1.04::000
MEMIT
49.98=0.29
46.55
43.23
0.0020.00
0.00:0..0
0.00=00.0
PRUNE*
50.2 22:32
47. 23.333
0. 0.00sa..0
D. 0.00g0..0
0.00=00.0
RECT*
43.32=0.9
42.21
42.01s0.20
0.00
0.00=0..0
0.00
200
ALPHAEDI
74
63.94.0.4
39.12.0.4
77.74 42
70.01 1:
27.96.100
RLEDIT
91.38.2445
41.98 89.93
94.03
90.67.9.41
68.71 n.:41
18

---

Reinforced Lifelong Editing for Language Models
ReEdit
AlphaEdit
RECT
PRUNE
MEND
(a) SST
(b) MMLU
(c) MRPC
1.0
0.5
Fl Score
0.0
(d) CoLA
(e) RTE
(f) NLI
1.0
0.5
0.0
0
1000
2000
3000 0
1000
2000
3000 0
1000
2000
3000
Num Edits
Figure 7: General Capability Test on CounterFact.
500 10, and 1000x5. Figure shows how various metrics vary across these configurations.
From Figure we observe that RLEdit demonstrates excellent editing performance across all configurations. However, we find that under extreme conditions with very large batch sizes, the hypernetorkk training in RLEdit consumes substantial GPU memory, leading high resource utilization. Similarly, under extreme conditions with very high edit frequencies, the hypernetwork training becomes unstable and often fails to converge. These observations suggest that appropriate edit frequencies and batch sizes should be selected to achieve more effective and stable editing results.
C. Algorithms
In Sections 3.2 and Section 3.3, we introduced the hypernetwork training and editing procedures of RLEdit. The correspond- ing pseudocode for RLEdit's editing algorithms is presented in Algorithm 2.
Algorithm 2 RLEdit Editing
Input: Knowledge sequence (I; ui D, pre-trained LLM fwo, post-trained hypernetwork 76 with parameter 0', hyper-parameters k; 7, Atee and y]
Output: Post-edited LLM with parameters W,
for t n do
L. log Pw; (ye (
Back Prokappo L, and cache Vw--
Vw. -- H(Vw. .)
W: < W + Vw.
end for return W,,
19

---

Reinforced Lifelong Editing for Language Models
100
Efficacy
90
Generalization Specificity
80
Metrics (%))
70
60
50
40
10x500
25x200
50x100
100x50
200x25
500x10
1000x5
Configuration
Figure 8: RLEdit's final metric scores when editing 5,000 knowledge instances under different configurations.
D. Detailed Proof
In Section 3.1, we briefly introduced how hypernetwork-baseee lifelong editing can be modeled as an MDP. Here we provide detailed proof that hypernetwork-basee lifelong editing indeed an MDP.
In Section 1, we stated that hypernetwork- lifelong editing is Markov process (MP). An MP consists several elements: (1) well-defined state space, (2) stationary state transitions, and (3) satisfaction of probability axioms. For (1), we (W, (ar, y)) in hypernetwork-baseee lifelong editing, or specifically, the fine-tuning gradient Vw. This state defined in continuous parameter space that take tensors of fixed dimensions. Each state has concrete mathematical representation. For (2), we modify LLM parameters through hypernetwork eepereetwork-ggneraaeeddd parameter updates, corresponding to state transitions in MDP; when we add Gaussian noise from a fixed distribution to the hypernetwrr output, our transition function's probability distribution becomes stationary. For (3), Gaussian noise satisfies probability axioms (probabilities are non-nege non-negative and sum to 1), thus state transitions also satisfy these axioms. conclusion, hypernetoork baased lifelong editing perfectly aligns with MP modeling and its characteristics.
model the
We now prove that this MP an MDP. Compared MP, MDP introduces concepts of action space, state transition function, and reward function. Regarding action space, bypernetwork-basee lifelong editing's actions are continuous parameter change matrices with dimensions matching the parameter matrices, which are fixed; since actions are controlled through hypernetwork parameters, they satisfy controllability; parameter change matrices in finiteddimensional real space possess complete metric structure, thus are well-defined. For the state transition function, we have already proven it well-defined and satisfies probability axioms. The non-deterministic component in the state transition function is Gaussian noise, whose distribution remains unchanged over time, proving state transition stationarity. Regarding the reward function, Section 3.1 explicitly detailed its computation method. Since rewards derived from losses that have bounds, the reward function is bounded; furthermore, the computation method remains constant over time, demonstrating stability. Therefore, hypernetwork-basse lifelong editing process constitutes standard MDP.
E. More Discussion
RLEdit training achieves impressive results with relatively few trajectory samples. This suggests that the lifelong editing task may be simpler than classic RL tasks, while also demonstrating RLEdit's strong generalization capability. This insight inspired us explore simpler few-shot knowledge editing paradigms.
In our experiments, we found that different choices of LLM editing layers significantly impact editing effectiveness. While locate-then-edit methods (Meng et al., 2023) typically choose to edit the first few layers of LLM due to their optimal editing performance, we discovered that RLEdit performs best when editing the middle and final layers of LLM. We believe this insight motivates further research on lifelong editing from the perspective of sequential knowledge localization. Additionally, we will investigate the use of advanced approaches of LLM interpretability (He et al., 2024; Zhou al., 2024) identify and
20

---

Reinforeed Lifelong Editing for Language Models
extract crucial layers within the model, enabling targeted editing and refinement for improved performance and adaptability.
Furthermore, in the future, aim apply the reinforcement learning paradigm to broader range of model editing methods, such as locate-and-ede (Jiang et al., 2025) and memory-based (Wang et al., 2024) approaches. Additionally, to address the challenges of updating multi-hop and related knowledge. we plan to enhance RLEdit by integrating external knowledge bases or knowledge graphs (Chen et al., 2023; 2024b; Zhang et al., 2025b;a), which crucial for further advancements. We will continue to investigate the hallucination and security issues (Chen et al., 2024a; Yang et al., 2024a;b; Liu al., 2024; Huang et al., 2025) brought by model editing, striving enhance the reliability and safety of this technology achieve more responsible Al development.
21

---

Reinforced Lifelong Editing for Language Models
F. Dataset Visualization through Examples
To help readers better understand the lifelong editing task and our implementation approach, we provide examples from the ZsRE, FEVER, and CounterFact datasets, as illustrated in Figures 9, 10, and 11.
"subject": "Lahti Town Hall",
"sre"; "Which person is the architect of Lahti Town Hall?", "rephrase": "Who was the architect of Lahti Town Hall?", "alt": "Willem Marinus Dudok",
"loc": "nq question: who plays alec ramsay in the black stallion",
"loc ans": "Kelly Reno", "ans": "Elicl Saarinen"
"subject"; "Phantom Stranger", "sre"; "In which fictional universe Phantom Stranger a character?", "rephrase": "In what fictitious universe is Phantom Stranger part?", "alt": "Forgotten Realms",
"loc": "nq question: who was the successful commanding general of the northern forces in the civil war". "loc ans": "George McClellan",
"ans": "DC Universe"
"subject": "1963 BAC One-Eleven test crash", "sre"; "What is the full date that 1963 BAC One-Eleven test crash crashed on?".
"rephrase"; "What's the date on which 1963 BAC One-Eleven test crash happened?",
"alt": "I1 March 1963",
"loc": "nq question: what is the current rate of interest on ppf",
"loc ans": "7.6%,,
"ans": "22 October 1963"
Figure 9: A sample of ZsRE dataset.
"The Faroc Islands part of the Hereditary Kingdom of Norway between 1035 and 1814.", "oquiv_ prompt':
"prompt":
were part Kingdom Norway between and 1814."
"The Faroes part of the Kingdom of Norway between 1035 and 1814." "The Faroe Islands were part of the Kingdom of Norway from 1035 to 1814.",
"The
were part of Nurway between 1035 "The Faroe Islands part of the Hereditary Kingdom of Norway from 1035 1814."
J.
"ans": "SUPPORTS".
"alt": "SUPPORTS".
"unrel_ prorpppp "Prugue's historic center was first included in the UNESCO list of World Herilage Sites in the ninties of the twenlieth century.". "unrel ans": "SUPPORTS"
Figure 10: A sample of FEVER dataset.
22

---

Reinforced Lifelong Editing for Language Models
"case id": 7285,
"pararel_ idx": 18788,
"requested rewrite": {
"prompt": "The language of is",
"relation id": "P407".
"target new":
"str": "Norwegian",
"id": "Q9043"
"larget true":
stt"": "Dutch".
rid": "Q7411"
"subject"; "Goede Tijden, Slechte Tijden"
"prapphraec prompts":
"The story is adaptation of Swan Lake. Goede Tijden, Stechte Tijden written in", "Winnie YU Tsang (wu4fdeu7424,) b. Goede Tijden, Slechte Tijden is written in"
"neighborhood prompts":
"Counter-Strike: Global Offensive was written in".
"Marie Claire is written in",
"The language of Magdalena was". " Addbe Flash Player was written in",
"The language of Marie Claire is", "The language
"Betsy was written in".
"The language Ben is".
"The language of Treetty of Accession 2003 was", "The language of Treaty of Accession 2003 is"
"attribute prompts":
"The language Kristin Lavransdatter is",
"Kristin Lavransdatter written in".
"The language
'u00e5r med oss 'u2013 Vem lu00c9 d'uu00e9 du ha is", "Mikkel is written in".
"For all tid was written in", "The language of Klassekampen is",
"Ove written in".
"The language Ghosts
"The language Jakobsen is",
"The language Morgenbladet was"
"generation "generation_promple" prompts": [
"Goede Tijden, Slechte Tijden's author from".
"Goede Tijden, Slechte Tijden's author from".
"The language Goode Tijden, Slechte Tijden written is called",.
"Goede Tijden, Slechte Tijden's author from",
"Goede Tijden, Slechte Tijden's author from".
"The language Goode Tijden, Slechte Tijden was written in is called",
"The language Goede Tijden, Slechte Tijden was written in is called".
"The language Goede Tijden, Slechte Tijden was written in is called".
"Goede Tijden, Slechte Tijden's author from".
"The language Goede Tijden, Slechte Tijden written is called"
Figure 11: A sample of CounterFact dataset.
23


Reviews:
[{"review_id": "jRZRnaGYJW", "reviewer": "Reviewer_2E3w", "summary": "The paper proposes RLEdit, a novel Reinforcement Learning based hypernetwork approach to edit Large Language Models for knowledge update after model completes training. It primarily aims to solve the challenge that hypernetwork based editing methods are efficient yet struggle more on large amount of edits. By using the dynamic environmental feedback in RL process, the model can better capture and edit based on the dynamics in the LLM. The work introduces Memory Backtracking Component and Regularization to Reward function in addition to the basic component, knowledge update and preservation objective. The authors show that RLEdit can outperform previous methods both in speed and performance when applied to different LLMs and data, especially when edits increase.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2E3w:\n\n**Thank you for your positive feedback on our work!**\n\nIn brief, we propose a hypernetwork training method for lifelong editing tasks that can adaptively adjust LLM parameters based on knowledge update sequences, achieving efficient and low-interference lifelong editing. \n\nWe appreciate your recognition of our paper writing and language organization, and we are happy to further simplify the technical descriptions in the final version to make them more accessible.\n\n*****\n\n## For *Relation To Broader Scientific Literature*:\n\n> I think Retrieval-Augmented (RAG) is a popular way and drawing the most attention for updating LLM's knowledge.\n\nThank you for your insights.\n\nIndeed, Retrieval-Augmented Generation (RAG) is a popular technique that effectively enhances LLM's generation capabilities by retrieving relevant information from external vector databases as contextual input. The main value of RAG lies in expanding the model's knowledge scope and improving domain-specific performance, and it can also be used for knowledge editing, thanks to LLM's powerful contextual learning abilities.\n\nHowever, we believe RAG has several limitations in the field of knowledge editing:\n\n1. **Effectiveness Challenges**: When correcting outdated or incorrect knowledge stored in LLM parameters, RAG methods inevitably lead to knowledge conflicts. Specifically, contradictions between retrieved new knowledge and model's built-in knowledge significantly reduce editing effectiveness, particularly evident in factual update tasks.\n2. **Insufficient Persistence**: The core objective of model editing is to permanently modify LLM's internal knowledge representations. Unlike methods that modify model parameters, RAG can be characterized as a prompt tuning technique, where knowledge updates are only effective in specific contexts, making it difficult to achieve persistent model editing.\n\nIn Model Editing research, we categorize methods like RAG that add auxiliary modules without changing base model parameters as *parameter-preserving editing methods* (as stated in Appendix B, Line 836). If you're interested in such methods, we recommend referring to the related research cited in our Related Work section.\n\n## For *Other Comments Or Suggestions*:\n\n> typo: Section 3.1.1 \"he training process\" should be \"the\"\n\nThank you for your thorough review! The typo you identified in Section 3.1.1, where \"he training process\" should indeed be corrected to \"the training process\", is well noted.\n\nWe greatly appreciate your attention to this detail and will immediately correct this error in our next revision. \n\n## For *Questions For Authors*:\n\n> I wonder compared to RAG, what is the pros and cons using the editing method.\n\nAs mentioned earlier, RAG represents a parameter-preserving knowledge editing method that assists LLM generation through external knowledge retrieval rather than directly modifying the model's internal knowledge representations.\n\nParameter-preserving methods (*e.g.*, RAG, SERAC, GRACE)  and parameter-modifying methods (*e.g.*, MEND, MEMIT, our proposed RLEdit) each have their advantages and limitations for different application scenarios.\n\nFor **RAG**:\n\n* **Advantages**: RAG primarily targets knowledge-enhanced generation tasks, effectively supplementing information beyond LLM's knowledge base to produce more comprehensive and accurate outputs. It offers easy deployment, high computational efficiency, and enables dynamic knowledge updates.\n\n- **Limitations**: When target editing knowledge conflicts with existing knowledge stored in model parameters, RAG struggles to fundamentally alter LLM's internal cognition, potentially leading to inconsistent outputs or performance degradation. Additionally, RAG depends on external knowledge base quality and retrieval precision, increasing system complexity.\n\nFor **Editing Methods**:\n\n- **Advantages**: These methods (like our RLEdit) fundamentally reshape model knowledge representations through direct parameter modification. Such modifications offer persistence, scalability, and end-to-end characteristics. The modified model can reflect updated knowledge without requiring additional modules, making them particularly suitable for knowledge correction and update tasks.\n- **Limitations**: Parameter-modifying methods typically demand higher computational resources, especially for large-scale models. Furthermore, multiple consecutive edits may lead to knowledge  interference and error accumulation, which is precisely the core problem our RLEdit attempts to mitigates.\n\n******\n\nThank you again for your feedback! If you have any further questions or suggestions regarding our methodology, experimental design, or potential application scenarios, we very much welcome continued in-depth discussion.\n\nBest regards\n\nAuthors", "claims_and_evidence": "The claims of the paper are clear and well-supported.", "methods_and_evaluation_criteria": "Frankly, I am unfamiliar with the paper's topic on large language model editing. The proposed method is complicated and challenges my knowledge range, so I can only give educated guess.  The method is lifelong editing through training a hypernetwork adapted to knowledge sequences (Meta learning), which is trained using reinforcement learning (RL). I can see that the evaluation follows standard practice and is compared against previous methods.   The paper is well written and clearly organized.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments look comprehensive.", "relation_to_broader_scientific_literature": "I think Retrieval-Augmented (RAG) is a popular way and drawing the most attention for updating LLM's knowledge.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1. typo: Section 3.1.1 \"he training process\" should be \"the\"", "questions_for_authors": "I wonder compared to RAG, what is the pros and cons using the editing method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v83fZeIGEi", "reviewer": "Reviewer_TQMf", "summary": "The paper proposes a model editing method, named RLEdit, for lifelong editing that extends meta-learning methods to sequential editing tasks. RLEdit's hypernetwork is trained using reinforcement learning algorithms, effectively maintaining Effectiveness, Generalization, and Locality at the knowledge sequence level. Experiments demonstrate that RLEdit shows advantages in both editing success rate and efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer TQMf:\n\n**Thank you very much for your recognition and support of our work!**\n\nWe are very willing to engage in in-depth discussions with you regarding the research details.\n\n******\n\n## For *Experimental Designs Or Analyses*:\n\nThank you for your valuable suggestions!\n\nFollowing your suggestion, we have conducted additional experiments. Due to the word limit constraints of the rebuttal, detailed experimental results can be found in **our response to Reviewer kk6q**.\n\nThe results shows that even after completing 20k knowledge edits, the model maintained stable GLUE scores. We will include these experimental data in the revised paper to ensure evaluation completeness.\n\n## For *Essential References Not Discussed*:\n\nWe strongly agree with your suggestion.\n\nIn the revised version, we will add citations to key literature on RL theoretical foundations and supplement discussions of recent research developments in lifelong editing methods.\n\n## For *Weakness 1*:\n\nThank you very much for your suggestions!\n\nWe understand the importance of comparing RLEdit with parameter-preserving editing methods like GRACE and SERAC, although they operate on different mechanisms from RLEdit.\n\nFollowing your suggestion, we have conducted comparative experiments between RLEdit and GRACE, SERAC on 20$\\times$100 tasks. The experiments were performed on Llama-3-8B-Instruct using both ZsRE and CounterFact datasets. The results are shown in the following table:\n\n|        |              |     ZsRE     |              |              | CounterFact  |              |\n| :----: | :----------: | :----------: | :----------: | :----------: | :----------: | :----------: |\n|        |     Eff.     |     Gen.     |     Spe.     |     Eff.     |     Gen.     |     Spe.     |\n| GRACE  |  **93.58**   |     1.03     | <u>31.86</u> |  **96.72**   |    50.14     |  **72.23**   |\n| SERAC  |    67.75     | <u>33.96</u> |    22.17     |    71.21     | <u>61.05</u> | <u>66.90</u> |\n| RLEdit | <u>88.65</u> |  **83.91**   |  **47.61**   | <u>91.75</u> |  **62.40**   |    52.38     |\n\nWe observe that RLEdit shows comparable performance to parameter-preserving editing methods, demonstrating the effectiveness of the RLEdit approach.\n\n## For *Weakness 2*:\n\nThank you again for this key suggestion!\n\nAs stated above, we have added relevant experiments. Detailed results have been presented in **our response to Reviewer kk6q**.\n\n## For *Other Comments Or Suggestions*:\n\nThank you very much for your careful review of the paper!\n\nWe will carefully revise these errors in the revised manuscript to ensure that the final submitted paper meets high quality standards.\n\n## For *Question 1*:\n\nThank you for highlighting this important observation. We have conducted an in-depth analysis of this phenomenon and identified several main reasons:\n\n1. This high-density, long-sequence editing task significantly amplifies the  cumulative effect of editing errors, potentially triggering catastrophic forgetting or even severely damaging the LLM's basic functionalities.\n2. Although our baseline comparison methods perform excellently in  single or short-sequence editing scenarios, most were not specifically  optimized for long sequence continuous editing scenarios.\n\nThis finding was indeed one of the key motivations that drove us to develop RLEdit, aiming to achieve more precise and robust long-sequence lifelong editing capabilities.\n\n## For *Question 2*:\n\nThank you for raising this profound question.\n\nOur analysis suggests this phenomenon is primarily related to inherent limitations in hypernetwork training mechanisms.\n\nParameter changes and potential errors introduced by the previous batch of edits continue to accumulate and propagate to subsequent edits, and the model instability caused by this error accumulation effect significantly weakens the overall editing performance of these methods.\n\nWhile complete retraining might theoretically help, as you point out, this indeed poses serious computational efficiency challenges, especially in large-scale lifelong editing scenarios.\n\n## For *Question 3*:\n\nThank you for your attention to the results in Figure 8. We indeed systematically explored the performance of various baseline methods under different batch configurations.\n\nOur research found that, with a fixed total number of editing samples, the performance of most baseline methods is highly correlated with specific editing configurations. Particularly, when reducing batch size while increasing batch number, these methods generally show significant performance degradation. \n\nWe believe RLEdit's stability stems from its sequence-level perspective. RLEdit hypernetwork training focuses on the entire knowledge sequence rather than individual knowledge samples, enabling the hypernetwork to handle diverse editing configurations.\n\n******\n\nThank you again for your detailed and thorough feedback! If you have any other questions, we would be happy to continue the discussion.\n\nBest regards\n\nAuthors", "claims_and_evidence": "The authors' claims about RLEdit are substantiated through rigorous mathematical derivations and comprehensive experiments. The training of RLEdit's hypernetwork, based on reinforcement learning modeling and utilizing policy gradient concepts to optimize carefully designed reward functions, demonstrates both convincing theoretical foundation and practical implementability. The evaluation across various models and datasets further validates the credibility of their conclusions.", "methods_and_evaluation_criteria": "Yes. The authors designed a targeted solution for lifelong editing tasks, achieving long knowledge sequence lifelong editing through the application of the RLEdit hypernetwork for continuous editing. This establishes a new editing paradigm while maintaining both effectiveness and efficiency. The evaluation methods used in the paper are standard in the model editing field, such as examining Effectiveness,  Generalization, and Specificity on CounterFact, Zsre, and FEVER datasets, making their results convincing.", "theoretical_claims": "In this paper, the authors propose a novel idea of using reinforcement learning to model the training process of editing hypernetworks. The authors clearly demonstrate that meta-learning editing methods can be modeled as an MDP,  making them solvable through reinforcement learning. They then design reward functions targeting the key challenges of lifelong editing, with solid and well-founded theoretical derivations. I have no concerns regarding the theoretical aspects of this paper.", "experimental_designs_or_analyses": "The experimental design in the paper is comprehensive, covering 3 commonly used open-source LLMs (Llama3, Gemma2, Mistral) and 3 standard datasets in the model editing field (ZsRE, FEVER, CounterFact), demonstrating RLEdit's generalization ability and robustness across different scenarios. The authors also compare RLEdit with numerous baseline methods, proving RLEdit's superior performance and efficiency.\n\nHowever, one concern stands out: Figure 5 does not show the LLM's general capability metrics when editing more than 3k knowledge items. The authors should supplement related experiments to demonstrate RLEdit's robustness in editing 10k knowledge items.", "relation_to_broader_scientific_literature": "RLEdit proposed in this paper marks a significant advancement in extending meta-learning model editing methods to lifelong editing of long knowledge sequences. RLEdit optimizes existing meta-learning editing methods and can be generalized to MEND and MALMEN, endowing them with lifelong editing capabilities, thus providing novel and valuable insights for the field of model editing.", "essential_references_not_discussed": "The paper provides fairly comprehensive citations of landmark papers in the lifelong model editing field. However, I believe the authors could include more references to seminal works in reinforcement learning to strengthen the theoretical validation. Additionally, some recent lifelong editing methods (*e.g.*, GLAME, O-Edit) could also be discussed.", "other_strengths_and_weaknesses": "# Strengths\n\n1. The paper presents an innovative approach to long-sequence lifelong editing, addressing a practically significant problem. I agrees with authors' perspective that continuous knowledge editing requires modeling at the entire knowledge sequence level, which is an elegant and holistic idea.\n2. Based on this, the authors proposes an effective and efficient editing method. To my knowledge, it is the first to extend meta-learning editing to continuous editing tasks at 10k scale, which demonstrates pioneering work in hypernetwork-based large-scale knowledge editing.\n3. RLEdit shows strong transferability on MEND and MALMEN, revealing the potential of meta-learning methods in lifelong editing. This observation provides valuable insights for future research.\n4. The comprehensive experiments demonstrates RLEdit's versatility across multiple scenarios.\n\n# Weaknesses\n\n1. Comparing RLEdit with parameter-preserving editing methods (*e.g.*, GRACE, SERAC) would be more convincing, as they also target lifelong editing. This is only a suggestion and will not affect my rate.\n2. As mentioned in **Experimental Designs Or Analyses**, tests of LLM's general capabilities after editing more than 3k knowledge items should be conducted to demonstrate RLEdit's effectiveness at the 10k scale of knowledge editing.", "other_comments_or_suggestions": "There are some minor typos in the paper. For example, I believe the $\\nabla_W$ in line 117 should be $\\tilde{\\nabla}_\\mathcal{W}$, and \"he\" in line 159 should be \"The\".", "questions_for_authors": "1. In Table 2, many editing methods fail under specific LLMs, with metrics dropping to as low as 0.00%. Have you investigated the reasons for this?\n2. Why don't MEND\\* and MALMEN\\* show significant improvements over MEND  and MALMEN? I feel that retraining the hypernetwork on post-edited LLMs after each edit might be effective, although highly inefficient.\n3. In Figure 8, I'm surprised by RLEdit's consistently strong performance across different configurations. Have you explored how other baseline methods (*e.g.*, MEMIT, AlphaEdit) perform under different batch_size and batch_number ratios? How do you explain RLEdit's stability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXk6dafRaJ", "reviewer": "Reviewer_kk6q", "summary": "This paper introduces RLEdit, a method addressing long-term continuous LLM knowledge editing, known as lifelong editing. The authors apply reinforcement learning modeling to meta-learning knowledge editing approaches, proposing a hypernetwork training method specifically designed for lifelong editing tasks. They focus on addressing knowledge forgetting and conflicts in lifelong editing by  designing a reward function with memory backtracking to optimize hypernetwork performance on long sequence editing. Experiments demonstrate the method's effectiveness across various configurations while maintaining high efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kk6q:\n\n**Thank you for your positive feedback on our work!**\n\nWe are happy to discuss the details of our work and hope to address all your questions.\n\n******\n\n## For *Other Strengths And Weaknesses*: \n\n> Equation (9) requires more clarification. A detailed explanation would help readers better understand this notation.\n\nThank you for your correction! The terms $\\mathcal{L} _ {e _ i, \\mathcal{W} _ {t-1}}$ and $\\mathcal{L} _ {loc _ {i}, \\mathcal{W} _ {t-1}}$ in the equation represent the loss functions corresponding to the editing knowledge at time step $i$, computed on the model parameters $\\mathcal{W} _ {t-1}$ at time step $t-1$.\n\nWe agree that Equation (9) indeed needs clearer explanation, and we will add detailed clarification in the revised version to enhance readability.\n\n> Minor: Some typos. (*e.g.*, Line 97, right column: \"updated\" should be \"updates\"; Line 159, left column: \"he\" should be \"The\".)\n\nThank you for identifying these typos! We will correct these issues in the revised version, including changing \"updated\" to \"updates\" in the right column of line 97 and \"he\" to \"The\" in the left column of line 159.\n\n## For *Questions For Authors*:\n\n> Figure 5 only examines the LLM's general capacity after 3,000 edits  across different methods. Given that you've extended lifelong editing to over 10,000 edits, I believe the general capacity evaluation should be  expanded to cover more editing steps. Have you conducted related experiments?\n\nWe fully agree with your suggestion. To verify that RLEdit can maintain LLM's general capabilities after large-scale continuous editing, it is necessary to evaluate the model's performance after completing 10,000 edits.\n\nFollowing your suggestion, we have supplemented our experiments to assess the impact of large-scale model editing on the model's general capabilities. Consistent with Section 4.5 of our paper, we used a 200$\\times$100 configuration (*i.e.*, 200 batches with 100 edits per batch) to edit Llama-3-8B-Instruct model on the ZsRE dataset. We conducted GLUE tests after every 2,000 edits, recording F1 Scores for SST, MMLU, MRPC, CoLA, RTE, and NLI metrics. The experimental results are as follows:\n\n|       | SST    | MMLU   | MRPC   | CoLA   | RTE    | NLI    |\n| :---- | ------ | ------ | ------ | ------ | ------ | ------ |\n| 0     | 0.8311 | 0.5624 | 0.6576 | 0.7607 | 0.2839 | 0.6660 |\n| 2000  | 0.8791 | 0.4944 | 0.6862 | 0.7703 | 0.2809 | 0.5924 |\n| 4000  | 0.8797 | 0.5010 | 0.7093 | 0.7703 | 0.2870 | 0.5685 |\n| 6000  | 0.8797 | 0.4853 | 0.7190 | 0.7594 | 0.2809 | 0.5685 |\n| 8000  | 0.8779 | 0.5019 | 0.7097 | 0.7498 | 0.2870 | 0.5328 |\n| 10000 | 0.8654 | 0.4707 | 0.7196 | 0.7607 | 0.2870 | 0.5907 |\n| 12000 | 0.8654 | 0.4759 | 0.7278 | 0.7423 | 0.2930 | 0.6044 |\n| 14000 | 0.8381 | 0.4599 | 0.7300 | 0.7389 | 0.3056 | 0.5907 |\n| 16000 | 0.8443 | 0.4805 | 0.7298 | 0.7510 | 0.2967 | 0.5907 |\n| 18000 | 0.8456 | 0.4438 | 0.7383 | 0.7617 | 0.3030 | 0.5907 |\n| 20000 | 0.8395 | 0.4653 | 0.7391 | 0.7437 | 0.3030 | 0.5907 |\n\nFrom the experimental data, we can observe that as the number of knowledge samples continues to increase, our RLEdit method consistently maintains stable general capabilities of the LLM. Notably, even after 20,000 knowledge edits, the model maintains general capability metrics comparable to its initial state. This strongly demonstrates RLEdit's exceptional stability in large-scale continuous  knowledge editing scenarios.\n\n> I'm curious about how the choice of LLM layers affects model editing  performance. Have you explored varying the selection and number of  edited LLM layers and investigated their impact on the results?\n\nIn our experiments, we found that the choice of editing layers significantly impacts the editing effectiveness. Specifically, when selecting early layers for RLEdit, the final metrics might decrease by 5%-10% compared to selecting middle or later layers; certain layer choices might result in very high Efficacy but very low Generalization and Specificity.\n\nTherefore, the selection of LLM layers is a hyperparameter that requires careful tuning, which also demonstrates the complexity of LLM knowledge storage mechanisms.\n\nCurrently, several studies are focusing on investigating the impact of layer selection on editing effectiveness. We will include references to these works and add discussion of this issue in our subsequent revision.\n\n******\n\nThank you again for your positive feedback and valuable suggestions on our work! If you have any other questions, we would be happy to continue the discussion.\n\nBest regards\n\nAuthors", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.This paper presents the first attempt to model hypernetwork-based lifelong editing using reinforcement learning, demonstrating remarkable novelty.", "theoretical_claims": "Yes,all of them", "experimental_designs_or_analyses": "Extensive experiments validate RLEdit's high performance and efficiency, with the authors providing code and detailed hyperparameter configurations to ensure reproducibility.", "relation_to_broader_scientific_literature": "This paper presents the first attempt to model hypernetwork-based lifelong editing using reinforcement learning, demonstrating remarkable novelty，Previous articles usually start from the initial state.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Equation (9) requires more clarification. A detailed explanation would help readers better understand this notation.\n\nMinor: Some typos. (*e.g.*, Line 97, right column: \"updated\" should be \"updates\"; Line 159, left column: \"he\" should be \"The\".)", "other_comments_or_suggestions": "None", "questions_for_authors": "Figure 5 only examines the LLM's general capacity after 3,000 edits across different methods. Given that you've extended lifelong editing to over 10,000 edits, I believe the general capacity evaluation should be expanded to cover more editing steps. Have you conducted related experiments?\n\nI'm curious about how the choice of LLM layers affects model editing performance. Have you explored varying the selection and number of edited LLM layers and investigated their impact on the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces RLEdit a method that frames lifelong LLM editing as a reinforcement learning problem. RLEdit demonstrates significant advantages over existing approaches with a 59.24% improvement in effectiveness while requiring only 2.11% of computation time. 

Key strengths include: its theoretically sound framework, successful extension of meta-learning to long sequence editing (up to 20k edits), strong performance across multiple models and datasets, and stability in maintaining LLM capabilities after extensive editing. 

During the rebuttal phase, the authors addressed all reviewer concerns with additional experiments showing stable performance even after 20k edits, comparative analysis with parameter-preserving methods, and clear explanations of theoretical foundations, further strengthening the case for acceptance.
Decision: Accept (poster)