WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Authors: Peng Wang
Zexi Li
Ningyu Zhang
Ziwen Xu
Yunzhi Yao
Yong Jiang
Pengjun Xie
Fei Huang
Huajun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Lifelong Model Editing
Large Language Model
Knowledge Memory
Primary area: natural_language_processing
TL;DR: We propose an effective method of lifelong model editing for large language models.

Abstract: Large language models (LLMs) need knowledge updates to meet the ever-growing world facts and correct the hallucinated responses, facilitating the methods of lifelong model editing. Where the updated knowledge resides in memories is a fundamental question for model editing. In this paper, we find that editing either long-term memory (direct model parameters) or working memory (non-parametric knowledge of neural network activations/representations by retrieval) will result in an impossible triangle---reliability, generalization, and locality can not be realized together in the lifelong editing settings. For long-term memory, directly editing the parameters will cause conflicts with irrelevant pretrained knowledge or previous edits (poor reliability and locality). For working memory, retrieval-based activations can hardly make the model understand the edits and generalize (poor generalization). Therefore, we propose WISE to bridge the gap between memories. In WISE, we design a dual parametric memory scheme, which consists of the main memory for the pretrained knowledge and a side memory for the edited knowledge. We only edit the knowledge in the side memory and train a router to decide which memory to go through when given a query. For continual editing, we devise a knowledge-sharding mechanism where different sets of edits reside in distinct subspaces of parameters, and are subsequently merged into a shared memory without conflicts. Extensive experiments show that WISE can outperform previous model editing methods and overcome the impossible triangle under lifelong model editing of question answering, hallucination, and out-of-distribution settings across trending LLM architectures, e.g., GPT, LLaMA, and Mistral.

WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Peng Wang Zexi Li
Ningyu Zhang Ziwen Xu
Yunzhi Yao
Yong Jiang
Pengjun Xie? Fei Huang² Huajun Chen 133
Zhejiang University 2 Alibaba Group
3 Zhejiang Key Laboratory of Big Data Intelligent Computing {peng2001, zexi.. 1i zhangningyui edu.
Abstract
Large language models (LLMs) need knowledge updates to meet the ever-growing world facts and correct the hallucinated responses, facilitating the methods of lifelong model editing. Where the updated knowledge resides in memories is fundamental question for model editing. this paper, we find that editing either long-term term memory (direct model parameters) or working memory (non- parametric knowledge of neural network accivaatons/representttinos by retrieval) will result in an impossible triangle-relaaillii generalization, and locality can not realized together in the lifelong editing settings. For long-term memory, directly editing the parameters will cause conflicts with irrelevant pretrained knowledge previous edits (poor reliability and locality). For working memory, retrieval- based activations can hardly make the model understand the edits and generalize (poor generalization). Therefore, we propose WISE to bridge the gap between memories. In WISE, design a dual parametric memory scheme, which consists of the main memory for the pretrained knowledge and a side memory for the edited knowledge. We only edit the knowledge in the side memory and train router decide which memory to go through when given a query. For continual editing, we devise a knowledge-shardie distinct subspaces of parameters and are subsequently merged into a shared memory without conflicts. Extensive experiments show that WISE can outperform previous model editing methods and overcome the impossible triangle under lifelong model editing of question answering. hallucination, and -uttr--distribution settings across trending LLM architectures, c.g., GPT, LLaMA, and Mistral
mechanism where different
of edits reside
Introduction
Large language models (LLMs) show emergent intelligence when scaling the number of parameters and data [1-4], which reveals the sparks of artificial general intelligence [5]. However, when deployed, LLMs still make mistakes [6], generating responses with hallucinations [7], bias [8], and factual decays [9]. On the other hand, the world's knowledge ever-growing, so the up-to-date knowledge is usually different from the one during LLMs' pretraining [10]. Many such errors and emerging facts will arise sequentially in deployment, some of which have to be addressed timely and efficiently without waiting for retraining or finetuning [11, 12]. Also, retraining or finetuning is often too computationally expensive [13, 10], which not sustainable for lifelong growing knowledge. Therefore, lifelong model editing [10] was proposed to remedy the continual knowledge updates and injections for LLMs a cheap and timely manner.
Equal contribution. i Corresponding Author. Coode is available
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

An effective lifelong model editing approach should satisfy the following properties [14, 15, 11, 16, 17]: i) reliability, the model can remember both current and previous edits after sequential editing; ii) locality, model editing will not influence inherent pretrained knowledge which is irrelevant to the edited knowledge; iii) generalization, the model not just merely memorizing the query-target pairs; instead, it should understand and generalize when given other forms of queries with the same knowledge. We com- pare existing model editing and continual learning meth- ods on the three metrics in Figure and find that it seems to be an impossible triangle-reliabilie locality can not be realized at the same time in the contin- ual editing settings. We find that where the updated knowl- edge resides in memories affects editing performances, and previous methods can be generally divided into editing either long-term memory, e.g., ROME [18], MEMIT [19), and FT-EWC (Finetuning with Elastic Weight Consolida- tion [20], a continual learning method), or working mem- ory, e.g., GRACE [10]. Note that the categorization of long-term and working memories is derived from human recognition [21, 22] and neuroscience [23] which has re- cently been adopted in the study of LLMs [24-27]. Model editing of long-ten term memory refers to directly editing the model parameters, which contain generalizable parametric knowledge [28, 24]. However, editing long-term memory will cause conflicts with previous pretrained knowledge, resulting in poor locality (e.g., ROME and FT-EWC in Figure 1). Working memory refers to the non-parametrie knowledge of neural network activations/reeresenaatiinss by retrieval, and it does not change the network parameters [24]; instead, it replaces the representations by retrieval at working (inference) time, like GRACE. GRACE's working memory shows promising results in reliability and locality, but in our experiments, it shows poor generalization since retrieval -based representations can hardly make the model understand the edits and generalize to different queries. It reveals that long-term memory and working memory both have drawbacks for lifelong model editing. though there were some special memory designs for L.M architectures, like MemorryLLM [28], SPALM [27], and Memoria [25], they change the architectures and cannot be directly applied for different LLMs. Intuitively, there is a gap between editing working and long-term memories, thus, in this paper, we study:
Continusl Editing T- 100
- WISE 10ur Methodi
Reliabliity
FTEWC
DIFER
GRACE
generalization, and
ROME
Locality
Coneralization
Figure 1: Metric triangle among re- liability, generalization, and locality. ZsRE dataset, number of continual edits T 100, LLaMA-2-7B. Editing meth- ods based on long-term memory (ROME and FT-EWC) and working memory (DEFER and GRACE) show the impos- sible triangle in metrics, while our WISE is leading in all three metries.
What is the better memory mechanism for lifelong model editing break the impossible triangle?
Human brains contain the left and right hemispheres, which have different divisions as studied in recognition science [29, 30], e.g., the left brain is typically associated with logical tasks while the right brain is more involved in intuitive processes. This inspires us to design WISE, which makes model editor WISER in memories. WISE contains a dual parametric memory mechanism for LLMs' editing: the main memory for the pretrained knowledge and a side memory for the edited knowledge, realizing both long-term memory's generalization and retrieval-baa bassed working memory's 's reliability and locality. The side memory is a form of mid-term memory. We only edit the knowledge in the side memory and train a router to decide which memory to go through when given a query. For continual editing, we design a knowledge- mhowledge-hhardin mechanism where different sets of edits reside in distinct and orthogonal subspaces of parameters. These are then merged into a common side memory without conflicts. Our contributions are as follows:
We identify the pitfalls of current model editing methods in lifelong settings, that is, the impossible triangle among reliability, generalization, and locality. Behind the impossible triangle, we find there is a gap between editing long-term memory and working memory.
We propose WISE, with a side parametric memory as the mid-term memory, realizing the ad- vantages of both parametric long-term memory and baaeeval-baedd working memory. We design memory routing, sharding, and merging modules in WISE, making WISE lead in continual knowl- edge editing, reaching the three metries better simultaneously.
Extensive experiments on GPT, LLaMA, and Mistral across QA, Hallucination, and out-of- distribution datasets validate the effectiveness of WISE for lifelong model editing.
2

---

2 Methodology
2.1 Preliminaries: Lifelong Model Editing
We focus on lifelong model editing problem [10, 11], which can ensure hundreds or even thousands of sequential edits on LLMs to make the outputs of target queries align with human expectations while maintaining LLMs' previous knowledge and capability. Let fe X + W. parameterized by θ, denote a model function mapping an input x to the prediction fe (x). The initial model
before editing is Og. which is trained on a large corpus Diriin When the LLM makes mistakes or requires injections of new knowledge, it needs model editing with a time-evolving editing dataset as Deait {(X,, D')) ( yY), (T))..) At the time step T, a model editor (ME) takes the T-th edit and the LLM of the 1 time step fer as inputs and produce the revised LLM model for following the equation below:
fee (fe (x)
if E Xe, if & X.
fer ME(Jer YT), s.t. fex
Equation| describes that after model editing, the LLM should make the correct prediction on the current edit as fe, (xr) = yr, while also preserving knowledge from past editing instances (x<T, E Dedi as well as maintaining capability of fe, on the irrelevant data when æ & Xe. especially for general training corpus Drain-
2.2 Rethinking the Memory Design of Lifelong Model Editing
Table I: Comparison of current model editing methods. "V" refers to "yes" and "well-supported"
X refers to "no" or "badly-supported", and "O" refers to "less-supported". The three metrics of Reliability, Generalization, and Locality denote the performances on lifelong (continual) editing.
MeDhn'
Long temm Menusy
Wurking Monsry
Panmettic Knowladge
Retieeaa Knowlogge
Whether Lidlang Relibiliiy
Genendiation Locality
FLEWC
X
ROMEMEMIT
x
MEND
SERACTIFER
x
GRACE
/
WISE
In Table 1, we compare current model editing methods in terms of memory types and lifelong
editing abilities. FT-EWC [20], ROME [18], MEMIT [19], and MEND [31] edit the long-term memory stored in the LLMs' model parameters, but they either do not support continual editing or have negative effects on irrelevant knowledge (poor locality). GRACE [10] designed for lifelong editing via retrieval-basee irrelevant knowledge, but GRACE fails to generalize due to its codebook being a non non-parametric knowledge representation that solely memorizes queries without comprehension. It is worth noting that SERAC [32]/DFFER [10] uses working memory that is stored in additional small models: a scope classifier and counterfactual model, whose knowledge is parametric. However, the small counterfactual model cannot match the expressiveness and generalization capabilities of LLM itself, making it challenging for the edited knowledge generalize effectively.
working memory. The retrieval codebook
avoid the conflicts of
To enable effective lifelong model editing, the method should take advantage of both LLM parameters' long-term memory and retrieval-based working memory. Therefore, we propose WISE as follows.
2.3 WISE: Side Memory with Knowledge Sharding, Merging, and Routing
As illustrated in Figure 2. WISE comprises two key components: Side Memory Design: i) side memory: side memory is memory container that is initialized as copy of LLM's certain FFN layer, storing the stream of edits; ii) memory routing mechanism: similar to retrieval, a routing activation component is adopted to identify the scope of edits, routing the main (original) or side memories during inference; 2) Knowledge Sharding and Merging: i) knowledge in rundom memory subypuces: to make the edits in appropriate knowledge density and avoid forgetting, we shard the side memory into several random subspaces for editing; ii) knowledge merging: we leverage model merging techniques to merge different memory shards into one side memory without loss of knowledge.
3

---

Layero befro o5Ring
Ediling laper
Layors ahe oding AGg Lagurs LLatte-2-18
Intaaizze Wy
eig. Lagers LLaM/A22-70
M.g. Laper LLMMA2-77
Genersle nandon masko with steams
Attn
%2 %a by Tea 112
Main
FFN
W
menoory subspaces
Input Layers
Input Layers
Output Layers
Data
WE
Audivaatio
Routing
Merge sudrgaacee side memory via Tics Marg
T serect the max one
Editing
Side Wemoriaa
Layer
(+) Overviev Knowledge Bouting
(b) Koowledge Sharding Merging
Figure 2: Overview of WISE. Side memory (in blue) and main memory (in green) store edited and pretrained knowledge, respectively. Note: during inference, if will retrieve and select one side memory with maximal activation score.
the activation routing
2.3.1 Side Memory Design
Side memory in FFN's value matrix. Each layer in Transformer contains a multi-head self-attention (MHA) mechanism and a feed-forward network (FFN), where the FFN constitutes two-thirds of the model parameters [33]. The question of how Transformers retrieve and utilize stored knowledge remains unresolved [18, 34], yet past works [31, 33J have demonstrated that editing the weights of the FFN is consistently more effective for LLMs. The FFN typically consists of key-value linear matrices: W W o. i.c., two multi-layer perceptron (MLP) layers. For the output of attention network, omitting the bias terms, can be represented as:
feature f, the computation of the
FFN(f) = a W, = o(f WE) W..
(2)
where a is a nonlinear activation function (e.g. SwiGLU, GeLU), and represents the activation values of the first MLP layer. Following previous works [18. 33], we edit the value matrix W. of the chosen FFN layer.
However, directly editing the value matrix may cause forgetting and side effects in lifelong setting. Thus. we copy a value matrix as side memory and edit the side memory instead of the original matrix (main memory). Specifically, the side memory initialized with the copy of main memory as W. + W.. Given the side memory, the new output is expressed as FFN, (f) a W.. We will introduce how update the side memory Section 2.3.2.
Locating side memory's FEN layer. Transformer LLMs have been widely demonstrated encode "lower-level" information (c.g., parts of speech) in earlier layers while processing more advanced linguistic phenomena like anaphora and coreference in later layers [35-37]. Representations in later hidden layers propagate through residual connections without drastic changes [38, 18], enabling effective early exit in LLMs [39, 40]. Therefore, to minimize the side effects of editing and adjust advanced linguistic phenomena, we target mid-to-late layers (c.g. 27) for side memory. Further analysis of layer selection provided in Section 3.3.
Routing between side memories and main memory. Similar the retrieval-bast -based methods [10, 32], during inference, it is needed to decide whether the main memory or the side memory is used. If given query is within the scope of previous edits, the side memory is used; otherwise, the main memory. Inspired by [11], we introduce a routing activation indicator, given an input x, it is formulated:
(x) = ||A(x) (W,r W.)
where A(-) = a is the activation of the side memory's 's corresponding FFN layer in Equation 2. We want the activation indicators of editing queries to be larger than the ones of irrelevant queries by a large margin, which is:
min (xx))x Deait) > muax(Anale (x; )xx Dinh.
(4)
where DIn is the irrelevant dataset which includes Durain

---

To achieve the above objective, we design a margin- based loss function during editing training, similar to contrastive [41] or triplet loss [42]. The margin- based loss function for routing activation is:
La min m (x)( a) + max(0, 8 (x)( + max(0, Y (Xe) (x)(x)))) (5)
S.L. X2 E Dadii X, E Dir-
Equation 5 aims that for all queries of irrelevant examples x,, the activation indicators should be less than threshold a, and for the edit samples xx. the activations should be larger than threshold 3. with certain distance y between Ax, and (x,).
In the continual stream of incoming edits, the smallest activation indicator within the edits updated and saved: min Am (x.) (x.))xxx E }. We aim to recognize the local scope of edits in this form. During inference, if the activation indicator of a new input is greater than e, WISE will use the side memory W vr; otherwise, using the main memory W. Thus, given the query the output of the targeted FFN in Equation 2 is replaced by:
FFN (x) = [A(x) W A(xx) W.
][A (W Wo)l2>>
(6)
otherwise.
2.3.2 Knowledge Sharding and Merging
How to effectively and efficiently store continual knowledge in model parameters is important for lifelong editing. We introduce the notion of "knowledge density" (similar knowledge capacity [43]) that describes how many pieces of knowledge are stored per parameter on average. There is editing dilemma w.r.t. knowledge density: i) If only a few edits are made for full fine-tuning or editing the entire memory, the knowledge density is low, which may lead to overfitting. ii) If numerous edits are made within a common and limited parameter space, the knowledge density is high, resulting in conflicts within the edited knowledge and potentially causing catastrophic forgetting. To remedy this dilemma, we propose knowledge sharding and merging mechanism to divide the edits into several shards, store them in different parameter subspaces, and merge them into common side memory.
Knowledge in random memory subspaces. We edit the side memory W.,. We divide n edits into k shards, copy the side memory for k times, and generate it random gradient mask with mask ratio p for each copy of side memory. A random gradient mask M, E {0, /// e [k] is a binary mask whose proportion of is p [44]. For edit shard i, E [A). we edit the knowledge into the subspace M, as follows:
W W 8' (M, @ (W:)),
(7)
where W of is the i-th copy of the side memory, 17 the learning rate, (-()) is the gradient of the i-th shard of edits, and the gradient is the autoregressive loss plus the routing activation loss T. (Equation 5): Tredit = log Pw.,, (ye )e + I.g.
The random mask of gradients freezes the parameters intact when the elements are 0 and updates the weights when the elements are 1. is superior pruning because does not harm the network performance while regularizing optimization in subspace [44]. In addition, the p subspace will have higher knowledge density when k - < 1, resulting in higher generalization (e.g., Figure 5). Also, different shards of edits have different random masks, and due to the (sub)orthogonality of random masks, different shards will not conflict with each other. Therefore, we can non-destructively merge the k copies of side memory into one.
Knowledge merging. We merge the ie subspace pieces of side memory into one. Because we randomly generate the subspace masks, different random masks will have some overlapping elements and some disjoint elements, following the theorem below:
Theorem 2.1 Subspace Overlap. Generate k memory subspaces W i E [k] by random mask with 1't 's ratio p. so each memory has p- - active trained parameters. For any two subspaces W e and W '' i + j; E [k),, there are ( active parumeters that are overlapped. For all subspaces, there are p² WW overlapped active parameters.
The theorem shows that larger p will cause more overlap of subspace parameters, and the proof is in Appendix C. We find that this overlap is helpful in playing the role of "anchors" for knowledge merging (See Figure 5 and Appendix B.5). However, knowledge conflicts also exist in the overlapped parameters, so we leverage the recent task arithmetic model merging technique
[45] to
5

---

Table 2: Main editing results for QA setting (ZsRE dataset), T: Num Edits.
Q4
Method
T
100
T
1000
Rel.
Gen.
Loc.
Avg.
Rel.
Gen.
Loc.
Avg.
Rel.
Gen.
Loc.
Avg.
Rel.
Gen.
Loc.
| Avg.
LLaMM
FT-L
0.57
0.52
0.96
0.68
0.48
0.48
0.76
0.57
0.30
0.27
0.23
0.27
0.19
0.16
0.03
0.13
FTW
0.96
0.95
0.02
0.64
0.82
0.01
0.83
0.74
0.08
0.55
0.76
MEND
0.95
0.93
0.98
0.95
0.26
0.28
0.28
0.27
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ROME
0.85
0.80
0.99
0.88
0.64
0.23
0.22
0.04
0.16
MEMIT
0.84
0.81
0.99
0.88
0.58
0.58
0.85
0.67
0.02
0.02
0.02
0.02
0.04
0.04
0.02
0.03
MEMIT-MASS
0.84
0.81
0.99
0.88
0.75
0.72
0.97
0.81
0.76
0.68
0.85
0.76
0.69
0.65
0.62
0.65
DEFER
0.68
0.58
0.56
0.61
0.65
0.47
0.36
0.49
0.20
0.12
0.27
0.20
0.03
0.03
0.74
0.27
GRACE
0.99
0.36
1.00
0.78
0.96
0.16
1.00
0.71
0.96
0.15
1.00
0.70
0.93
0.08
1.00
0.67
WISE
0.98
0.92
1.00
0.97
0.94
0.88
1.00
0.94
0.90
0.81
1.00
0.90
0.77
0.72
1.00
0.83
Kistral-78
FT-L
0.58
0.54
0.91
0.68
0.39
0.39
0.50
0.43
0.11
0.10
0.02
0.08
0.16
0.13
0.01
0.10
FI-EWC
1,00
0.99
0.01
0.82
0.72
0.09
0.54
MEND
0.94
0.93
0.98
0.95
0.01
0.01
0.02
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ROME
0.79
0.77
0.98
0.85
0.05
0.02
0.04
MEMIT
0.81
0.79
0.99
0.86
0.46
0.45
0.61
0.51
0.00
0.00
0.01
0.00
0.04
0.04
0.02
0.03
MEMIT-MASS
0.81
0.79
0.99
0.86
0.74
0.71
0.97
0.81
0.73
0.71
0.88
0.77
0.73
0.70
0.62
0.68
DEFER
0.64
0.54
0.79
0.66
0.53
0.43
0.29
0.42
0.28
0.17
0.26
0.24
0.02
0.02
0.67
0.24
GRACE
1,00
0.36
1.00
0.79
1.00
0.15
1.00
0.72
1,00
0.15
1.00
0.72
1.00
0.02
1.00
0.67
WISE
0.98
0.97
1.00
0.98
0.92
0.89
1.00
0.94
0.87
0.80
1.00
0.89
0.70
0.67
1.00
0.79
relieve the conflicts. First, we compute the edit weight shift vectors T. [rè Then, we use Ties-Merge to merge the edit vectors into one:
W
Wold E [A]].
Wrr f- W. + Ties(Tei W.).
(8)
Ties-Merge consists of three steps: i) trim: trim the redundant parameters for each task vector; ii) elect the sign: elect the signs of each parameter; ii) disjoint merge: compute the disjoint mean for each parameter which has the same and correct signs [45]. By Ties-Merge, different subspaces of knowledge are integrated into one with fewer conflicts. We study the effects of different merging techniques in Table 11 of Appendix B.2.
Routing and retrieving among several side memories. One single side memory has its limited knowledge capacity [43]. For the lifelong editing stream, we can produce several side memories and retrieve them via activation score routing. We compute different activation indicator scores of side memories and retrieve the top-1 during inference. This design is named WISE-Retriews which enables a more challenging lifelong editing scenario. For WISE with only one side memory, it is notated as WISE-Merge. For most of the experiments, we use WISE-Merge by default, and we compare WISE-Retrreve in Table 6 and Figure 6. The pseudo-code of our method can be found in Algorithms and 2.
3 Experiments
3.1 Experimental Settings and Evaluation Metrics
In the experiments, we compare the performance of different baselines and WISE in sequentially editing LLM models hundreds to thousands of times. In practice, we augment x, by generating 10 random token sequences of length 10 using fe, enhancing editing generalization//aaatttion to diverse contexts. We ensure that this augmentation with random tokens is applied across all baselines (See Appendix B. we ablate the contribution of Random Token).
Datasets and Models. We choose trending au- toregressive LLM models I.LAMA---7B [13], Mistral-7B [52], and GPT-J-6B [53, 54] for
Table 3: Dataset statistics for main results. Locality Data is the irrelevant data of the editing process. T the number of samples. Pre-edit is the unedited model's performance on each dataset.
evaluation. The dataset details are in Table 3. Following [10]. we evaluate WISE on the closed-book question-answerinn
SETTING
EDITING DATA
T
Provedii (LLaMA/Mistaa)) LOCALITY DATA
(QA) dataset ZsRE [46], and also evaluate its ability to cor- rect Hallucination in GeIPCheckGPT [48]. The
QA Halle. OODD Gen.
ZsRE [46]
1,000
0.360359 ACC
Ng (47]
SeHCheesGaFT (49)
600
27 4/9.9 PFL
RedPajma (89)
Temporal (50)
100
0.56 6-ACC (GPT-)
Pile [51)
Temporal dataset [50] is employed to test the out- duitt--distribution (OOD) generalization of editing. Since Temporal comprises emerging entities 2019-2019, we avoid using the latest LLMs in OOD
experiments. Instead, we follow the original literature of the Temporal dataset [50] and adopt GPT-J-
6B as the base model, which is pretrained on the Pile [51] with a cutoff in 2020. Implementation details and editing examples for each dataset and can be found in Appendix A.

---

Table 4: Main editing results for Hallucination setting (SelfCheekGPPT dataset). T: Num Edits.
Htalactnation
-LaRE-2-76
Miatral-T1
T- 10
T
T00
7- (3)
T-1
7- 10
T-
100
T 0000
Methol
Rel. (PP) ()
Too: (T)
Rel (1)
071
(2)
Tex: (1)
Rel (1)
Loc. (1)
Rel. C.)
Lxc, (T)
Rel. (1)
Loc. (T)
Rel. (1)
Lec. (7)
Rel. (
Lec. (T)
PRL
11.05
12.57
14:31
35..6
11:18
103033
1431
1590-93
0102
FREWE
2.50
11.34
3.61
1..00
2.80
11.5
4.25
0124
3333
1089
4.23
0.17
3.46
MEND
5.65
0187
11:01
0.35
10.00
0.88
1911.90
0.0
3.6
0.36
83.11
0.01
35134.91
000
ROMME
1.68
1199
3.04
8.91
98.15
106
0.98
1.85
1983
103.75
001
17
MEMIT
1.50
1.0
2,36
0.57
76.65
0.65
107.61
002
1.64
1.80
15.89
0.89
57.23
0.4
132.50
0/2
MASIT-MASS
1.60
1.00
1.61
0.59
1.18
0.56
13.47
0.94
1.64
1.80
2.78
0.99
322
0.97
7.5
0.5
DEFER
1.29
01.23
3.64
18.26
8,91
18.39
19.16
0.12
4.78
1648
7.30
14225
9.54
044
24.15
0.13
GRAAEE
2.21
1.00
1.67
1.40
9.67
1.80
9.34
1
1.39
1.80
3.97
1.80
9.53
1.00
9.57
1.00
WISE
1.91
1.00
1.34
1.40
1.84
1.40
3.12
0.99
1.40
1.80
2.56
0.94
1.31
0.98
5.31
0.93
The baselines include methods of continual learning and model editing. We compare WISE against direct fine-tuning FT-L with an additional KI. divergence loss [18], and continual learning fine-tuning based on Elastic Weight Consolidation (FT-EWC) [20]. We also compare WISE to other model editors, including 1) GPT-style editors based causal tracing: ROME [18], MEMIT [19], and MEMIT-MASS (a batch-editing version of MEMIT); 2) hypernetwork- editors: MEND [31] and 3) the latest memory--aseed editors: DEFER (inspired by SERAC [32] for inference routing) and GRACE [10]. Details on all comparisons are found in Appendix A.2.
Baselines.
Metrics. Each edit example includes an edit descriptor (i.e., query) Xe, its paraphrase prompts X.' (if available) for testing generalization. and an unrelated statement Xloc for testing locality. For the editing dataset Dodn = {e., }+}} } with I edits, we evaluate the final post-edit model fer after the T-th edit example (xr. We evaluate the model editor's 's reliability and generalization using the metrics Rel. (a.k.a Edit Success Rate [10I) and Gen. (Generalization Success Rate [55]). while Loc. (I (Localization Success Rate [55]), defined as the post-edit model should not change the output of the irrelevant examples Xinc, assesses specificity. We report these metrics and their mean scores, which are formally defined as:
Rel. + (Jo (x2)) = =.). Gen.
I Σ 1(fr (x)) = y'). Loc.
I T Σ (Sor (x()) = f0n (x(W))
(9)
where (-) is the indicator function. Notably, for the Hallucination dataset, following [10), we use the perplexity (PPL.) to verify the locality, and there is no proper metric for generalization.
3.2 Main Results
Competitive Performance of WISE. The competitive performance of WISE is evident in Table 2 and 4, which compare its results with eight baselines on the QA (ZsRE) and Hallucination (SeliCheckGPT) settings. In general, we observe the followings: O WISE outperforms existing methods on multiple tasks after long editing sequences; @ direct editing of long-term memory (ROME, MEMIT, etc.) creates conflicts with prior pretraining knowledge, resulting in poor locality; and retrieving working memory and modifying activations (GRACE, DEFER, etc) struggle to generalize to diverse queries.
In the QA setting, with T = 1000, WISE achieves average scores of 0.83 and 0.79 on ILaMAA and Mistral, respec- tively, reflecting improvements of 18% and 11% over the nearest competitor. This demonstrates WISE's outstand- ing stability and effective management of long-sequential edits. While methods like MEND and ROME are compet- itive early in editing, they show clear shortcomings as the edit sequence extends. Directly editing long-term memory (e.g., MEMIT, FT-EWC, MEND) results in a significant decline in Loc. When T E (100, 1000), this indicates that these methods cannot preserve LLMs' knowledge structure and significantly impair the model's generalization ability. GRACE excels in Loc. and Rel. (close to 1.00), however, sacrifices generalization in continual editing. A possible reason is that token representation may not be suitable for measuring semantic similarity in autoregressive LMs, leading to paraphrase X failing to achieve similarity matching with any CodeBook Key in GRACE (detailed in Appendix B) Overemphasis
Table 5: 00D results for Temporal dataset. GPT-J-6B is used.
73
0DD Lot. [Rl.
000 Gra. 1ox: [Arg
ool Editky
0.56
0.27
0.39
(0.55
0,21
10.39
FTE
[.8]
0.22
0.40
ROME
D.99
0.00
066
0.25
(085
0.50
083
013
MEMITSAASS
0.2
099
0.55
0.78
0.27
0.97
0.67
DEFER
0.3
0.36
0.51
0.36
006
WISE
1.99
0.36
0998
8.76
(095
0.37
1,00 6.76
preserving and precisely adapting training data (working memory) hampers adaptability to new contexts. In a nutshell, most previous methods struggle to balance Rel., Gen., and Loc., particularly in long- form-form editing tasks. In addition, the results of GPT-J-6B can be found in Figure 9 in the Appendix.
WISE also surpasses the baselines on the Hallucination dataset, maintaining the lowest perplexity scores of 3.12 and 5.21 at T 600, with Loc. remaining above 0.93. We similarly observe
7

---

Perfermace Meg!
Subepace Cerrlap
1.0
Geardltasee
5.6
Localay
30
8.11
07
3.6
2.
82
5.0
1-20
0621 6.70
0.97
40
6.7
Figure 4:
Analysis of locating
Figure 5: Analysis of different mask ratios p and subspaces k for WISE. Left: Avg. performance of Rel., Gen., and Loc.; Right: the subspace overlap probability in Thcorem 2.1. 7sRE, LLaMA-2-7B.
FEN layer of side memory for WISE. ZsRE, LLaMA- 2-BB.
significant PPL increases for FT-L, MEND, and ROME in long-context editing tasks, while GRACE's performance is lackluster in LLM long texts (possibly due to the limited fitting capacity of the very small active trained parameters \761 of GRACE).
Out-of- of--sti-iiitttion Evaluation. Ideally, model editing needs generalize distributionally from formulaic editing examples to natural texts [50], where the distributional shift involves complexity rather than conventional domain shift [56]. Following [50], we evaluate the OOD generalization of editing methods on emerging entities using the temporal updating dataset, Temporal. Editing examples and evaluation metrics are provided in Appendix A.I. As shown in Table 5, WISE effectively handles out-of- of-dissributton generalization tasks (achieving the best OOD Gen. and overall performance). DEFER delivers mediocre performance on OOD Gen. due to the limited capacity of the auxiliary model[14). During the fine-tuning phase, GRACE and MEMIT focus on the representation u* of a single input token after W. (GRACE: last token, MEMIT: last subject token). However, regarding us the editing carrier encounters two problems: I) the training objective is not aligned with the pretraining phase, and 2) the single representation limits the search scope of gradient descent, making it difficult to handle OOD generalization. WISE, on the other hand, avoids these challenges.
3.3 Further Analysis
Visualization of WISE's Routing Activation. To demonstrate the effectiveness of memory routing, we record the activation values (x) of 1000 (QA, ZsREy600 (Halluc.) queries during the inference stage via knowledge merging into single side memory. As shown in Figure 3, the purple horizontal line represents the activa- tion threshold recorded during the editing phase. Almost all unrelated queries show low activations with values less than 10 in ZsRE and less than 20 in Halluc.; meanwhile, WISE accurately routes the editing prompt and unseen paraphrases into the side memory. This ensures editing locality and prevents excessive shifts from the pre-training distribution during lifelong editing.
QA (mee)
40
200
400
600
Woo
1000
Ballocination (wolkteookqp))
600
440
1000
200
300
400
550
600
Figure 3: Activations of the memory routing module of WISE when vary- ing T. X- X-axis: Num edits. LLaMA-7B.
Localization Analysis of WISE's Side Memory. To validate the benefits of editing mid-to-late layers, we select decoder layers from early, intermediate, mid-to-late, and late stages. As shown in Figure 4. the ablation results reveal that editing critical layers like the early and final layers (0, 1, 31) is ineffective, even resulting in a very low Loc. value of 0.096, which indicates a failure to recognize the editing scope. This may occur because the early layers represent fundamental grammatical information, and the final layer directly controls the decoding procedure, leading to poor editing of advanced language functions. Editing in the intermediate layers is suboptimal but still shows a markable improvement compared to early layers, possibly because intermediate layers start to integrate basic grammatical information with more complex semantic data. Notably, the mid-to-late layers demonstrate exceptional editing performance; for instance, selecting layer 26 results in an 80% success rate and generalization while maintaining 100% locality. This empirically supports our claim in Section 2 that the redundant mid-to-late layers [39] are ideal side memory layers and confirms the hierarchical nature of information processing in Transformer LLMs [57, 58].
Analysis of p and for WISE. We analyze the important hyperparameters of WISE: the mask ratio p and the number of subspaces k in Figure 5, On the left figure, for l 2, the best p is 0.2,
8

---

satisfying k* p 0.4 < 1, which implies the effectiveness of our subspace design that higher knowledge density will cause better generalization. When scaling k, we observe an increasing demand of p. From Theorem 2.1, the probability of subspace overlap is p², and we hypothesize that this overlap is important as an anchor for model merging. Interestingly, from the right figure, it can be observed that the optimal cases always have the p closest to 0.03. This shows an inherent tradeoff between merge anchor and merge conflicts, and the subspace overlaps around 0.03 are optimal for the best performances. Such experiments indicate that 20% FFN parameters can accommodate at least 500 edited samples. When "mask memory exhaustion" " occurs, we can allocate new mask parameters to store new knowledge. Using retrieve when knowledge isn't 't full and merging needed to save memory, achieves true lifelong model editing.
Scale Up to 3K of Edits. We scale the num- ber of continual edits to 3K in Table 6, We compare WISE-Merge, keeping one side mem- ory by multi-time merging, and WiSE-Retrieve, keeping several side memories by routing and retrieving among different side memories. For Retriveer,eve, we show an upper bound "ora- cle", which always identifies the correct routing path. We observe that the WISE series maintains high scalability, consistently outperforming the strongest baselines including MEMIT-MASS and GRACE. WISE-Retrieve based on top-1 activa- tion retrieval demonstrates the best results in 3K edits, showing the effectiveness of well-organized memory subspaces and routing strategies during editing. We note that the "oracle" exhibits marginal performance decline when scaling the edits from 2K to 3K, yet demonstrates remarkable perfor- mance across all metrics. This underscores the potential of WISE to handle extremely long continual edits, contingent upon substantial improvement in the retrieval of side memories. Additionally, an appropriate replay of edits can further improve retrieval accuracy, as detailed in Appendix B.3.
Table 6: Scaling to 3K edits of ZsRE. -LLMM-2-77
T
2000
T 3000
Method
Rel.
Gen.
Loc
Avg.
Rel.
Gen. Loc | Avg.
GRACE
0.96 0.03
1.00
0.66
0.96
0.03
1.00
0.66
MEMIT-MASS
0.64 0.58
0.59
0.58
0.53
0.53
WISE- WISE-Retrieve
0.66 0.63
1.00
0.76
0.58
0.56
1.00
0.71
0.68 0.64
0.77
0.61
0.58
0.73
0.77 0,72 1.00
0.83
0.75 0,70
1.00
0.82
Contribution of Router designs in WISE. Without the router strategy, all inputs either pass solely through the main or side memory. To further validate its effectiveness, we conduct additional ablations with Ly. WISE's performance on ZsRE is shown in Table 7. We observe the expected decrease in Loc. w.o. La. such as dropping from 1.00 to 0.72 at T=1000, re- veals the router's effectiveness in identifying editing scopes, minimizing side effects, and retaining a substantial amount of pre-training knowledge.
Table 7: Ablation study of Router (compared with Table 2). LlaMA.
WISE J.,
Rel.
Avg.
T- 1
1.00 0.96
0.93
40.07
0.96
4000
T= 10
0.93 0.90
0.88
-012
0.90
-0.04
T- 100
0.92 0.85
0.81
40.19
0.86
4004
T= 1000
0.84 0.79
0.72 -.28
0.78 -005
Inference Time Analysis of WISE. Figure 6 shows the inference time of a single instance for LLaMA after t [0, 3000] editing steps, measured across 10 trials of each setting. Consistent with our expectations, we find that WISE-Merge incurs a con- stant inference delay (about 3%) as the editing stream expands. RetreReetriee, due to the introduction of retrieval routing, shows an increase in inference time as the number of edits increases, with time cost increment of about 7% after 3K edits. Knowledge merging ensures that WISE-Merge only brings constant additional costs (0.64% extra parameters and 4% extra GPU VRAM, as detailed in Appendix B.7), contrasting with past memory-based works that continuously demand more available memory [10, 32].
Inference Time (1.0x)
w/o Bdiin
1:85
MGBE Merue
1.20
MOSE Rot leve
1-1
1.00
0.96
800
1000
1500
2000
2500
3000
Figure 6: Inference time of WISE when varying T. ZsRE, LLaMA-2-7B.
Related Works
Memory and Knowledge Injection of LLMs. LLMs have long-term (episodic) and working mem- ory [24, 25, 27]. Long-term memory is stored in model parameters, updatable via (re)pretraining [53], finetuning [59], and model editing [14]. Working memory resides in neuron activations, utilized during inference [24]. In-context learning and retrieval-based editing methods like GRACE contribute to working memory [60, 10]. However, whether finetuning or retrieval is debated [61, 62]. Also, current knowledge injection methods often suffer from computational overhead [13, 10], catastrophic forgetting [63], and overfitting [64]. Methods like MemorryL.MM [28]. SPALM [27], NKB [65], and Memoria [25] are proposed to improve the memories from the architecture design perspective.

---

Model Editing of LLMs. Model editing encompasses constrained finetuning, locating errating-and--e and-aditing meta-learning and retrieval-based methods. ROME identifies factual associations and edits efficiently using MI.P-baee memories [18], extended by MEMIT for mass-editing [19]. T-Patcher adds neurons
for edits in L.LMs' feew-forward layers [11]. Meta-learning methods like MEND decouple finetuning gradients to generalize edits [31], complemented by MALMEN addressing cancellation effects [15]. Retrieval- methods like SERAC and GRACE improve working memory for editing [32, 10]. From single to mass editing and static to lifelong editing, model editing evolves to meet realistic demands. The latest efforts in lifelong editing such as LTE [66], MALMEN [15], and RECIPE [67] require extensive training with domain-specific edits before specific editing, yet we cannot predict the domain of upcoming edits in the editing flow and accessing these data is often impractical or unrealistic. It potentially increases the risks associated with retraining.
Model Merging Model merging [68], also known as model fusion [69, studies how to aggregate different models' knowledge into one by parameter merging. However, in the research of linear mode connectivity, it is found that different minima of neural networks can hardly be merged into a generalized one even if trained on the same datasets from the same initialization (but with different random seeds) [71, 72]. The main reason is considered to be the permutation invariance property of deep neural networks, which means that the positions of neurons can be permuted without affecting the network function [71]; as result, different minima reside in different loss basins [72]. To improve linear mode connectivity and model merging, methods like optimal transport [70. 73], re-basin [72], and training-time timee alignment [44] are developed. For the applications, model merging techniques can help to improve the generalization of federated learning [74, 75] and enable knowledge aggregation of different-task models in a task arithmetic way [76, 77]. Recently, methods like task arithmetic in tangent space [77]. TIES-Mergihn Merging [45], Ziplt! [78], and ColD fusion [79] have been proposed for
deep model fusion of pretrained foundation models, such as CLIP, ViT, and large language models.
Specifically, TTES-Merging [45] consists of trim, elect sign & merge pipeline, which inspires the merge process of side memories in our paper.
For detailed related works, please refer to Appendix D.
5 Limitations and Broader Impacts
Although WISE shows promising results in lifelong editing, it also has some limitations. One limitation is addressed in Table 6 that the side memory retrieval has room for improvement to reach the oracle. Also, in Figure 6, the inference time of WiSE-Retrieve increases with ever-growing editing streams. However, the current limitations cannot outweigh the merits of WISE in that it currently reaches better performance general for lifelong model editing. We bridge the gap between long-term and working memory, it may inspire further work memory design for model editing even LLM architecture. However, the application of such technologies should be guided by ethical considerations. Malicious users may attempt to edit LLMs to propagate hate, highlighting the need for safeguards to prevent abuse and mitigate harmful outcomes. Some current model editors update the model's weights directly, making edits hard to trace and withdraw. WISE uses modular and non-destructive side memory, allowing users to discard it if edits are unnecessary or harmful, without modifications to the main LLMs.
6 Conclusion
In this paper, we point out the impossible triangle of current lifelong modeling editing approaches that reliability, generalization, and locality can hardly be achieved simultancously. We find the reason behind this is the gap between working and long-term memory. Therefore, we propose WISE, consisting of side memory and model merging, to remedy the gap.
Acknowledgements
We would like to express gratitude to the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NS- FCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F03001 1), Yongjiang Talent Introduction Programme (2021 1-16-G. SMP-Zhipu.Al Large Model Cross- Disciplinary Fund, Ningbo Science and Technology Special Projects under Grant No. 20237212, Information Technology Center and State Key Lab of CAD&CG, Zhejiang University. We gratefully acknowledge the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.
10

---

References
[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
[2] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Be- yond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536 2022.
[3] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022.
[4] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du. Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, 02//2333.18223 2023.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
[6] Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 98 18-9830, 2022.
[7] Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12)11-38, 2023.
[8] Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. Challenges and Risks of Bias in Large Language Models (October 26, 2023), 2023.
[9] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491-6506, 2021.
[10] Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghas- semi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2023.
[11] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patchher One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, 2023.
[12] Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 02223348-23666 2021.
[13] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0928 2023.
[14] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportu- nities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222-10240. 2023.
[15] Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language model via meta learning. In The Twelfth International Conference on Learning Representations, 2023.
11

---

[16] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy V. Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. CoRR, abs/2004.00345,, 2020.
[17] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In France-F Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[18] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
[19] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory transformer. In The Eleventh International Conference on Learning Representations, 2023.
[20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Btrabsa-BBarwinns et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -3226, 2017.
[21] George A Miller, Galanter Eugene, and Karl H Pribram. Plans and the structure behaviour. In Systems Research for Behavioral Science, pages 369-382. Routledge, 2017.
[22] Alan Baddeley. Working memory and language: An overview. Journal of communication disorders, 36(3):189-208 2003.
[23] Keisuke Fukuda and Geoffrey F Woodman. Visual working memory buffers information retrieved from visual long-term memory. Proceedings of the National Academy of Sciences, 1 114(20)5306-5311, 2017.
[24] Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1774-1793, 2023.
[25] Sangjun Park and Jin YenYe Bak. Memoria: Hebbian memory architecture for human-like sequential processing, arXiv preprint arXiv:2310.030: 2023.
[26] Charles Packer, Vivian Fang, Shishir Patil, Kevin Lin, Sarah Wooders, and Joseph Gonzalez. Memgpt: Towards Ilms as operating systems. arXiv preprint arXiv:2310.08560, 2023.
[27] Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguisties, 9:362-373, 2021.
[28] Yu Wang, Xiusi Chen, Jingbo Shang, and Julian McAuley. Memoryllm: Towards self-updatable large language models. arXiv preprint 2iti:2400.0462244 2024.
[29] Joseph B Hellige. Hemispheric asymmetry: What's right and what's left, volume Harvard University Press, 2001.
[30] Richard B Ivry and Lynn C Robertson. The two sides of perception. MIT press, 1998.
[31] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. International Conference on Learning Representations, 2022.
[32] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-basee model editing at scale. In International Conference on Machine Learning, pages 158 15817-15831. PMLR, 2022.
12

---

[33] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In France-r Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[34] Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? In The Twelfth International Conference on Learning Representations, 2024.
[35] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David Traum, and Lluís Märquez, editors. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy, July 2019, Association for Computational Linguistics.
[36] Yulia Otmakhova, Karin Verspoor, and Jey Han Lau. Cross-linguistic comparison of linguistic feature encoding in BERT models for typologically different languages. In Ekaterina Vy- lomova, Edoardo Ponti, and Ryan Cotterell, editors, Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 27-35, Seattle, Washington, July 2022. Association for Computational Linguistics.
[37] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguisties.
[38] Yung-Sun Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengeheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024.
[39] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint (eX.i:22443308833.. 2024.
[40] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 17456-17472. Curran Associates, Inc., 2022.
[41] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pages 1597-1607. PMLRR, 2020.
[42] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[43] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. 2024.
[44] Zexi Li, Zhiqi Li, Jie Lin, Tao Shen, Tao Lin, and Chao Wu. Training- time neuron alignment through permutation subspace for improving linear mode connectivity and model fusion. arXiv preprint arXiv:2402.01
2024.
[45] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2023.
[46] Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia, editors, Proceedings of the 2 1 Conference on Computational Natural Language Learning (CoNLJ. 2017), pages 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics.
13

---

[47] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le,, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguisties, 7:452-466 2019.
[48] Potsawee Manakul, Adian Liusie, and Mark Gales. SeLiCheckGPT: Zero-resource black- box hallucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004-9017, Singapore, December 2023. Association for Computational Linguistics.
[49] Together Computer. Redpajama: an open dataset for training large language models. 2023.
[50] John Hewitt, Sarah Chen, Lanruo Lora Xie, Edward Adams, Percy Liang, and Christopher D. Manning. Model editing with canonical examples, 2024.
[51] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
[52] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock. Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[53] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training.
[54] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: : h/t::/gitttt thub.
www.iinnooooo/mmsiiiiio www.oo//eeeeeeetnoonoreeeeeeeeee trrnsfoomer-- May 2021.
[55] Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. 7ekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01288, 2024.
[56] Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust language modeling. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 4227-4237, Hong Kong, China, November 2019. Association for Computational Linguistics.
[57] Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, and Sebastian Schuster. Coloring the blank slate: Pre-training imparts hierarchical inductive bias sequence-to-seqqunce models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 1352-1368, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[58] Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. Grokking of hierarchical structure vanilla transformers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computationnal Linguistics (Volume 2: Short Papers), pages 439-448, Toronto, Canada, July 2023. Association for Computational Linguistics.
[59] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhte Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[60] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824-24883 2022.
14

---

[61] Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05954 2023.
[62] Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context learning: A fair comparison and evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12284-12314, 2023.
[63] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.0874 2023.
[64] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 15:38274-38299 2022.
[65] Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, and Zhifang Sui. Neural knowledge bank for pretrained transformers. Natural Language Processing and Chinese Comput- ing: 12th National CCF Conference, NLPCC 2023, Foshan, China, October 12-15, 2023, Proceedings, Part H. page 772-783, Berlin, Heidelberg, 2023. Springer-Verlag.
[66] Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, and Wei Wang. Learning to edit: Aligning ilms with knowledge editing, 2024.
[67] Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue. Lifelong knowledge editing for Ilms with retrieval-augmente continuous prompt learning, 2024.
[68] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's ' mergekit: toolkit for merging large language models. arXiv preprint drXiv:2403.132577 2024.
[69] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309. 15698, 2023.
[70] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
[71] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representutions, 2022.
[72] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023.
[73] Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. In The Twelfth International Conference on Learning Representations, 2024.
[74] Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In International Conference on Machine Learning, pages 19767-19788. PMLR, 2023.
[75] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaza- eni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
[76] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
[77] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
15

---

[78] George Stoica, Daniel Bolya, Jakob Brandt Bjomeer, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations, 2024.
[79] Shachar Don- Yohyy, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 788-806, 2023.
[80] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-
1901, 2020.
[81] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenA/ blog, 188):, 2019.
[82] OpenAl and the Co-authors. Gpt-4 technical report, 2024.
[83] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
[84] Ohad Shamir and Tong Zhang. Stochastic gradient descent for non- non-smooth optimization: Con- vergence results and optimal averaging schemes. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pages 71-79, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[85] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[86] Nils Reimers and Iryna Gurevych. Sentence- Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNPP) pages 3982-399 3992, Hong Kong, China, November 2019, Association for Computational Linguistics.
[87] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE:
Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6894-6910 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[88] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-67, 2020.
[89] Tian Yu Liu, Matthew Trager, Alessandro Achille, Pramuditha Perera, Luca Zancato, and Stefamo Soatto. Meaning representations from trajectories in autoregressive models. In The Twelfth International Conference on Learning Representations, 2024.
[90] Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya, and Jacob Andreas. Subspace regu- larizers for sho- class incremental learning. In International Conference Learning Representations, 2022.
[91] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Workshop on Efficient Systems for Foundation Modely ICML2023, 2023.
16

---

[92] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infii-attention. arXiv preprint arXiv:2404.07143, 2024.
[93] Matthew Sotoudeh and A Thakur. Correcting deep neural networks with small, generalizing patches. In Workshop on safety and robustness in decision making, 2019.
[94] Ankit Singh Rawat, Chen Zhu, Daliang Li, Felix Yu, Manzil Zaheer, Sanjiv Kumar, and Srinadh Bhojanapalli. Modifying memories in transformer models. In International Conference on Machine Learning (ICML), volume 2020, 2021.
[95] Shuaiyi Li, Yang Deng, Deng Cai, Hongyuan Lu, Liang Chen, and Wai Lam. Consecutive model editing with batch alongside hook layers, 2024.
[96] Ce Zheng, Lei Li, Qingxiu Dong. Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1862-4876, Singapore, December 2023. Association for Computational Linguistics.
[97] Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Pengliang Ji, and Xueqi Cheng. Decoding by contrasting knowledge: Enhancing iims' confidence on edited facts, 2024.
[98] Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang. and Hao Wang. Continual learning of large language models: A comprehensive survey, 2024.
[99] Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey, 2024.
[100] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence,
44(7):3366-338, 2021.
[101] Bill Yuchen Lin, Sida I Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, and Scott Yih. On continual model refinement in out- cit-of--istrbbution data streams. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3128-3139, 2022.
[102] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
[103) Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019.
[104] Thomas Henn, Yasukazu Sakamoto, Clément Jacquet, Shunsuke Yoshizawa, Masamichi Andou, Stephen Tchen, Ryosuke Saga, Hiroyuki Ishihara, Katsuhiko Shimizu, Yingzhen Li, et al. A principled approach to failure analysis and model repairment: Demonstration in medical imaging. In Medical Image Computing and Computer Assisted Intervention-MICAO 2021: 24th International Conference, Strasbourg. France, September 27-October 1, 2021, Proceedings, Part III 24, pages 509-518. Springer, 2021.
[105] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092-28103 200, 2021.
[106] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[107] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference on Computer Vision pages 63 631-648. Springer, 2022.
17

---

[108] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1399119,,0 2022.
[109] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Amool Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00800 2020.
[110] Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Curtis Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. In International Conference on Machine Learning, pages 34431-34455. PMLR, 2023.
[111] Yi Dai, Hao Lang, Yinhe Zheng, Fei Huang, Luo Si, and Yongbin Li. Lifelong learning for question answering with hierarchical prompts. arXiv e-prints, pages arXiv 2208, 2022.
18

---

Appendix
In the Appendix, we introduce more details along with additional experimental results, discussions, and related works:
Appendix A: Experimental setups (cf. Section 3).
Appendix B: More experimental results (cf. Section and 3).
Appendix C: Proof of the Theorem 2.1 (ef. Section 2).
Appendix D: Additional discussions and more related works (cf. Section 4).
A Implementation Details
A.1 Description of Datasets
Table 8: Bolded text refers to the edit labels Ye- Locality example X is an unrelated query.
(a) ZsRE, question-answering editing dataset example.
(b) Hallucination editing dataset example. In original data [10], there is no paraphrase ze' so the measurement of Gen. metric is ignored here.
Xe. Ye Which continent is Berkner Island in? South America
Xe. Ve This is a Wikipedia passage about heinz christian pander. Heinz Christian Pander (1794 1865) was a German anatomist and embryologist who was born in Riga, Latvia. He studied medicine at the University of Dorpat and later at the University of Berlin. In 1820, he took part in scientific expedition to Bokhara as naturalist.
Xloc
who gets the
golden boot if its a tie? shared
X Ye
On which continent is Berkner Island lo- cated? South Amer- ica
Xloc
Tired and restlessly, drifting in and out of sleep. Hearing crashing and banging, thinking the roof will cave in. Not alert enough to quite know what it was, I yelled loudly for whoever was making those noises such an hour to stop. They heard and listened, I' I'm guessing
The 7sRE question-answering task [46] is extensively studied within the model editing literature [18, 19, 31, 15, 11], where each record contains an editing statement X., paraphrase prompt X,, and a locality prompt Xiuc We use the same train/test split as [31] (163196/19088) Notably, only MEND requires fitting hypernetwork on the training set; other methods discard the training set and perform edits and evaluations on the test set. In practice, we randomly sample 1K and 3K records from the test set form the edit in Section 3.2 and 3.3.
ZsRE
Hallucination We utilize the same dataset GRACE, SeliCheckGPT [48], assess the ability of Model Editors to mitigate hallucinations in autoregressive LMs. This set- ting involves editing highly inaccurate sentences (sourced from GPT-3 [80]) and replacing them with correspond- ing sentences from actual Wikipedia entries. This dataset aligns more closely with real-world deployment scenar- ios where models trigger "unexpected behaviors," and the token length of edits is significantly longer than in past datasets, making it a more challenging editing setting. Un- like GRACE, which used GPT2-XL XL (1.5B) [81], our main experiments deploy larger LLMs, LLaMA and Mistral, Figure 7: Hallucination length statistics. measure retention of pretraining data (Xioe) from the base model: Red- Pajama [49], public version of L.L.MA's pretraining data. Some of the exceptionally long editing samples cannot
140-
120.
100-
Frequeercy
80
200 250
300
350
400
Ranges
both with 7B parameters,
be accommodated As shown in Figure?? 7, the original dataset provided by GRACE, after tokenization with LLAMATO- KENIZER, has length distributions ranging from [17,390] The dimension of a single MLP layer in is (000, 4096)* Theoretically. fine-tuning an input of length 390 with default
NVIDIA A800 (80GB) due to resource limitations.
'nttps://nugginggaes co/meta llama/ amaa/Llmma-2 21-.aaammmm27722 2-
19

---

Table 9: Temporal OOD dataset example. Bolded text refers to the edit labels Ye and Yood.
Xe. Ye
Self-driving cars, also known as autonomous vehicles, are vehicles that are capable of navigating and operating without human intervention. These innovative vehicles rely on a combination of advanced sensors, artificial intelligence, and computer algorithms to interpret their environment and make real-time decisions. With the potential to significantly impact numerous industries and sectors, self-driving cars have the ability to revolutionize transportation by enhancing safety, improving traffic flow, and increasing energy efficiency. However, challenges related to regulatory frameworks, ethical considerations, and public acceptance still need to be addressed before widespread adoption becomes a reality.
Xloc
Apple has a new peach with the release of its 3.0GHz, 8-core Intel Xeon-basee Mac Pro. The 8-core Mac Pro is powered bu two quad-core Intel Xeon Člov ertownprocessors running at 3.0GHz. Apple also released a quad-core Mac Pro featuring two Dual-Core Intel Xeon Woodcrestproceessoss.
Xe: Y ood Self-driving cars, also known autonomous cars or driverless cars, are vehicles capable of traveling without human input. These cars utilize a range of sensors, including optical and thermographic cameras, radar, lidar, ultrasound/sonar, GPS, odometry, and inertial measurement units, to perceive their surroundings. By interpreting sensory information, control systems in the car are able to create a three-dimensional model of its environment. Using this model, the car can then identify the best navigation path and develop strategies for managing traffic controls and obstacles. As self-driving car technology continues to advance, it is expected to have a significant impact on various fields such as the automotive industry, health, welfare, urban planning, traffic, insurance, and the labor market. The regulation of autonomous vehicles is also becoming an increasingly important topic of discussion.
full precision and the Adam optimizer would require (390+4+4+4) * (11008 * 4096 4) 4 7B
100.36GB of VRAM (for activations, gradients, first-order, and second-orde optimizers), exceeding the memory capacity of the NVIDIA A800. Consequently, we excluded excessively long samples (limiting tokenized lengths to 254) and ultimately retained 906 editing instances (compared to 1392 in GRACE). To facilitate a fair comparison with MEND, we specifically allocated a training set for MEND, with final train/test split of 306/600. All methods were edited and evaluated on the test set.
Temporal [50] sources the prefix x. from the first paragraph an entity's Wikipedia page and samples a paragraph y. discussed by GPT-4 [82] about the emerging entity x.: which is usually noisy but may contain helpful information. These are presented as editing prompts to Model Editors. For nut-of-distribution (OOD) generalization to complex natural contexts (not fitted), Y Yood is taken from the actual Wikipedia suffix of x.- This setup is utilized to evaluate the OOD generalization of Model Editors centered around single canonical example. Consistent with previous work [10], the out-of- scope data Xlsc is derived from the Pile [51], the pretraining corpus of GPT-J-6B. Examples from the dataset can be seen in Table 9. To measure the OOD generalization of editing methods for emerging entities, we perform model editing using standardized simple examples and then evaluate this behavior on more complex instances. Following [50], in natural setting, no single correct continuation exists. Thus, we also use probability threshold based evaluations, such as 80%, where the editing success rate evaluates whether the loss Ixx_ Yx for an example falls below
(0.8),)), as indicated in the for- mula below. The intuition behind this is that many other plausible alternative continuations may exist.
1
OOD Gen.
[1(To, (Xe, Nows) < <5)}.
(10)
T
A.2 Descriptions of Compared Model Editors
FT-L. All other layers of the L.MSs remain frozen, and only single MLP layer is fine-tuned through autoregressive loss [18]. Additionally, we impose an Lox norm constraint to prevent the parameters from deviating too far from the pretrained distribution.
FT-EWC. Flastic Weight Consolidation (EWC) has been demonstrated to mitigate catastrophic forgetting by updating weights using Fisher information matrix, which is computed from past edits,
20

---

multiplied by a scaling factor À [20]. Following [10], we omit the constraints of the Loe norm in this implementation.
MEND. MEND [31] transforms the gradients obtained from standard fine-tuning using a hyper- network that converts gradients decomposed into low rank (rank=1) into new gradients, which are then applied to the target layer for parameter updates. During the training phase, a small auxiliary hypernetwork receives editing examples (Xe, Ye), and X MEND's training loss comprises the standard autoregressive loss combined with the KL divergence loss of the model's output on X before and after editing. This hypemetwork plays a crucial role during the editing procedure.
ROME. ROME [18] uses causal analysis to pinpoint knowledge within specific MLP layers and modifies the entire matrix through least squares approximation. It operates under the strong assumption that the MLP is the primary module for storing knowledge [33], and it injects a single piece of knowledge into the MLP each iteration using a Lagrangian remainder.
MEMIT. Similarly, based on the assumption that the FFN serves as a knowledge key-value store, MEMIT [19] manipulates parameters of specific layers directly through least squares approximation. Unlike ROME, which updates a single layer, MEMIT is a multi-layer updating algorithm that supports simultaneous updates of hundreds thousands of facts. For sequential model editing tasks, MEMIT requires immediate on-the-fly fly repairs when the model makes errors, expressed as fe, (fer XT, Yr), involving multiple operations the original model.
MEMIT-MASS. Unlike sequential editing, MEMIT supports modification of multiple knowledge fragments in batch mode, named MEMIT-MASS. Suppose we collect streaming errors as (X, 3)) and inject them collectively into the MLP, only involves a single editing operation the original model as fer MEMIT (fe. X. )). Although this approach loses the capability for on-the-fly repairs, we still include this baseline in our experiments.
{(xo: (1): y1).... (xr.yr)
DEFER. In GRACE, a reimplementation of SERAC [32] is utilized, denoted as DEFER. For new inputs, DEFER includes a network g (corresponding to the scope classifier in SERAC) that predicts whether to: 1) trust the prediction of the L.MSs, or 2) trust the prediction of the new model. Here, the new model is configured as a single-layer linear network with a sigmoid activation function, corresponding to the counterfactual model in SERAC. During the editing process, g and are fine-tuned jointly.
GRACE. GRACE [10] utilizes discrete KEY- VALUE codebook and maintains the codebook throughout the editing flow by adding, expanding, and splitting KEYs. During the inference phase, retrieves the nearest KEY and determines whether to replace the activation of the hidden layer output.
A.3 Training Details and Hy Hyperparreeters
Except for MEMIT-MASS, the batch size for all methods is consistently in sequential editing scenarios. All experiments are conducted using 3 NVIDIA A800 GPUs, with all tasks reproducible on a single A800. Editing ZsRE takes approximately hours, while Hallucination requires around hours. To ensure fair comparisons, unless otherwise specified (for some methods like MEND, ROME, and MEMIT, we follow the original literature by selecting the last few layers or using causal analysis to identify the target layers), the default target layers for editing on LLaMA, Mistral, and GPT-J J are __or___roo-_. - weight, model. layers [27] .mlp. down__prooorooop weight, and transformer h [21] .mpppppp...... c_fc, respectively.
model.layere [27]
For FT-L, we utilize reimplementation from ROME 1, employing the Adam [83] optimizer with consideration of learning rates at le-5, le-4, and 5e-4, and conducting gradient descents for 50 iterations, ultimately reporting the best results at learning rate of 5e-4.
For FT-EWC, we follow the reimplementation in GRACE and its default settings, setting the learning rate at 1e-2, the Aewe penalty factor at 0.1, and the number of replay instances at 10.
For the training phase of MEND, we adhere to the original paper, setting the learning rate at le-4, iterating 100K times, and employing early stopping at 30K, ultimately achieving an accuracy of 0.95 on the training set. Notably, we target the last few MLP layers as per the original literature, such as model.layers [i] mlp down_projj wei ghttt model. .layers [i] .mlpp...........mm
mmw.gate_pppp. gate_proj. weight, model.layerr [i] .mmp.uppppp.. up_proj . weight in LLaMA, where E [29, 30,31].
For ROME and MEMIT, we follow the original literature on GPT-J using the default configurations, specifically the fifth layer and layers [3,4,5.6,7,88. In LLaMA and Mistral, additional causal analysis is conducted to pinpoint the layers storing knowledge. As shown in Figure 8, an increasing trend in
Phttps://githe com//mmen001/rmee
21

---

Caussi effect stakes at early siee with Arth or modules severel
Effect of single state on
Increcct Effect
Effect single
Averg indirect
Effect with Altn severed
Fiflect with severed
Effect with MLP severed
20.0%
1.000%
07%
6.0%
15
20
25
15
20
30
L1aM6-2-7B Layer which the single midden state is restored
Mistral 7B Layer which the single hidoen state restored
Figure 8: Mid-layer MLPs play a crucial mediating role in LLaMA 2- and Mistral-7B
1.3
14
Canerall mattoe
ROME
02
62
Number Cort
1083
Rumber Cantinall
T>
3783
1o
Contbuu
TEB
3800
Humber Continaa Cota
1991
Figure 9: GPT-J-6B, ZsRE, continual editing,
the Average Indirect Effect of the MLP is observed across layers [4,5,6,7,8], suggesting that the model recalls factual knowledge here and passes the matured token distribution via residual connections to the last MLP. Thus, in LLaMA and Mistral, ROME edits the fifth layer, while MEMIT edits layers [4,5,6,7,8].
For DEFER, the original literature uses learning rate of 1.0; however, we found it unfit for LLaMA and Mistral, with severe fluctuations in model loss. Therefore, we experiment with learning rates of 7e-5, 7e-4, and le-3, and ultimately report using 7e-5 (optimal).
Table 10: WISE hyper-parameters during editing and merging.
Hlyper-Paramette
Values
For GRACE, we strictly follow the original literature, setting the learning rate at 1.0, and using replace_lase _lst to only re- place the activation of the last token in autoregressive scenarios. After observing failures in generalization, we adjust various Gun values and discuss this more in Appendix B.
Optimizer LR n Mask Ratio p
SGD
1.0
0.2
5.0
B
20.0
For WISE, the hyperparameters for the QA and Hallucination tasks are identical. We find that a learning rate of 1.0 with the SGD [84] optimizer is good approach for stable train- ing. The hyperparameters designed in the knowledge editing phase include the random masking probability p and the routing threshold guidance a, 3, y. In the knowledge merging phase, hyperparameters include the number of merges k and the merging weights X for each MLP (we discuss the impact of p and & in Section 3.3). Theoretically, as the importance of knowledge in any MLP is considerable, we always average with A 1/k across all experiments. These are shown in Table 10.
Y
10.0
Merge Weights À
0.5
Knowledge shards k
2
A.4 Pseudo Code of WISE
The pseudo-code of the WISE editing stage is in Algorithm 1, and the of the WISE inference stage is Algorithm 2.
More Experimental Results and Analyses
B.1 On the Pitfall of GRACE: Generalization Collapses in Decoder-only LLMs
Here, discuss why GRACE exhibits poor generalization when editing decoder-only LMs.
As shown Figure 10, we continuously edit 15 samples (x) using GRACE and observe the nearest codebook Key for their paraphrases
and unrelated queries Xloc, as well as the governed Deferral radi e of those Keys. When overlapping Keys exist, GRACE reduces the Deferral radii to split this Keys and then adds new codebook entry, resulting in exponentially decaying of radii during the editing process. Though e initialized from high finit: will be small and ineffective after continuous edits. From Figure 10, observe that GRACE is more likely to have conservative
22

---

Algorithm 1:
WISE Editing Stage
The initial LLM model fe,, the targeted FFN layer, the edit dataset Dedi whose length is 7, the irrelevant dataset Din, the subspace mask ratio p. the numher of subspaces k; whether WISF-Retrrive.
Input:
Output: The final LLM model fer after I'' edits.
1: Generate & random masks M, [k] of ratio pt if WISE-Retrieve, copy the side memory several times;
2: for each edit (X,, Y:) E Dedit t [T] do
3: Edit Yx) in the corresponding memory subspace by Ledit = log pw., (y:(x)) Lai
4: Update the activation threshold:
min(e, A (x;))
5: if All the k subspaces of a side memory are full then
6: Use Ties-Merge in Equation 8 to update the final side memory;
7: if WISE-Retriews then
8:
Move to another copy of side memory Wer; 9: end if
10: else
11: if Current subspace M, is full then
12: Move to another subspace of side memory M
13: end if
14: end if
15:
end for
16:
return Obtain the final LLM model fer-
Algorithm 2: WISE Inference Stage
The edited LLM model fer, the activation threshold c, the test dataset Diest whether WISE-Retrieve.
Input:
Output: The model's output.
1: for each query X; Diest do
2: if WISE-Retrieve then
3: Get the value of activation Aust - [|A(xx;)) (Wer Wllll for each side memory and
select the one with the maximal value of Aacti
4: else
5:
Get the value of activation Auc ||A(x;;) (W
W.)l2s
6:
end if
7:
if Ame >> c then
8:
Use the side memory W o² generate the output as in Equation 9: else
10: Use the main memory W, to generate the output as in Equation 6.
11: end if
12:
end for
strategy that sets smaller Deferral radii during editing. Smaller Deferral radii will cause to fail to hit the codebook (the distance to the nearest Key is farther than its Deferral radií) but let X successfully far away from the radii, resulting low generalization and high locality. Also, we observe that the Deferral radii method is not effective under any Gnii; for all tested Enit values 1.0, 3.0, 10.0, and 500.0, they all have low generalization and high locality.
This suggests that in autoregressive LMs, the distribution of the last token cannot effectively represent semantics: whereas in encoder-only and encoder-decoder architectures, capturing semantic infor- mation through vector representation has been extensively studied [85-87]. This is consistent with the degree of generalization shown by GRACE when anchoring the T5 [88] Encoder layer. Some related works [89] also indicate that autoregressive models, semantic similarity measures based on averages of output tokens underperform, recommending the use of score distributions over text continuations to represent semantic distances.
B.2 Impact of Knowledge Merging Strategies for WISE
23

---

Eiik
DistOk, It Noarest Key k)) bu:
Distitec Iha Nearost Key Kg) buy
12 Distance
ofR,
30
11
12
13
14
Elik
DistiKe. Its Nourwal Key
Distifov Nearwet Kny kg)
12 Distance
20
Cul defferal radras ofk:
bxl defforal radrus of
10
2
10
14
Eist 10.
Distic, Key
Distitee:
ke)
L2 Distance
20
Fgg dedteral radius
Dej detforal
10-
10
11
12
13
14
Exit 500.
Disting Its Nearest Key kt)
Distikge: Its Nearest Key Kg)
12 Distance
Aug defieral of
Ax
12
10
11
12
13 14
Figure 10: Investigation the query x and its distance to the nearest Key k, as well as the deferral radius of that Key. Red and Blue respectively represent the paraphrase query X and the unrelated query Xiec, with the hatch representing the radius of the nearest Key. We observe that when conflicts occur (hit the codebook Key but with different Edit Target ye). the deferral radius decays exponentially. This results in GRACE being unable to encompass the paraphrase and maintain high locality, regardless of how Emn is adjusted. ZsRE, LLaMA 2--BB.
Here, conduct in-depth study of the knowledge merging strategies for WISE, exploring various merging approaches including (i) Linear, which uses simple weighted average; (fi) Slerp, which spherically interpolates the parameters of two models; (iri) Ties, a component used in the main experiments of this paper that resolves merging disturbances through TRIM ELECT SIGN; (iv) Dare: which follows Bernoulli distribution to delete redundant parame- ters and rescale the remaining ones: (v) Dare Ties. which combines dare and the sign consensus algorithm TIES; and (vi) Sign, an ablation component of Ties that addresses directional conflicts all utilizing the official implementation from MergeKit [68] We randomly sample 100 edits from
Table 11: Varying Merging Strat- egy. ZsRE, LLaMA- 1laMA-2-785 2-
Methods
RRll.
Gen.
Loc.
Avg.
Linear
.63
.61
.93
.72
Slerp
.62
.64
.91
.72
Dare
.68
.63
.92
.74
Dare_ Ties
.67
.63
.83
.71
Ties
.85
.81
94
.87
Sign
.80
.76
.97
.84
ZsRE, retaining a fine-tuned MLP every 50 edits (merging 2 MLPs). As shown in Table 11, we
hhttps://gis com comm/acee-aaaea http://rreeeaa/eeggtetetee

---

observe that ignoring the direction of parameter updates (Linear, Slerp, Dare) leads to significant decline in editing performance, underscoring the importance of addressing knowledge conflicts in overlapping parameters. The success of Sign also reaffirms this point. Meanwhile, the randomly masked knowledge shards exhibit a non-redundancy, indivisible nature. This demonstrated by the significantly weaker performance of Dare_ _iie compared to Ties/Sign, indicating that removing parameter updates can lead to the loss of edited knowledge or even potential "anchors".
B.3 Analysis of Retrieving Top-1 Activation
Edit Success (ES.) %
Routing Suceess (preeg)1) %
80
100
80-
WISE Retrieve
WISE Retricntsnase
WISE Retrieve
WISE Retrieve xc. Lecnno
WISE Retrieve w. Leweno
3000
3000
(a) Average Rel. and Gen.
(b) Retrieval Acc. by Top-1 Activation
Comparing editing results of Wist-(Retreeve, Retrievegracte Retrieve [ when varying T. (a) shows the simple average of Rel. and Gen. (ES.), while (b) shows retrieval accuracy, i.e., whether the Top-1 Activation routes the correct MLP (prec@1) 1). X-axis: Num edits. ZsRÉ. L1aMA-2-7B.
Figure 11:
WISE-Retrieve retains each knowledge-sharding memory and retrieves through Top-1 Activation. However, shown in Table 6 and Figure 11b, the retrieval accuracy still has significant room for improvement; specifically, when T reaches 3K, the accuracy of routing to the correct MLP drops to around 60%, indicating the specificity between side memories is insufficient. One possible reason is that when sampling the edits from a single dataset (ZsRF), the editing instances (X,, Ye) all belong to the same domain. This leads to some very similar instances being captured by multiple expert side memories (resulting in high activations for all side memories), introducing more retrieval failures.
Therefore, to improve the specificity of side memory and reduce the probability of routing errors, we attempt to add a new constraint Luemo to Equation 5. For knowledge-sharding memory W,, we randomly replay instances (Xm: ym) from the edit set Dw, of past shard W jej ensuring that W; remains inactive for Xm.
L' = La + max (xc(Xm)) - a), s.t. X. E Dw,
Luxuu
As shown in Figure 11b, this replay behavior increases the specificity between side memories, main- taining nearly 88% retrieval accuracy at T 3K. Figure la also shows that WISE-Retriews w. Limem improves Edit Success (ES.) by 8.39 % compared to WISE-Retrieve, providing a promising direction for future work. With finer-grained activation management, we might be able to bridge the performance gap between Retrieve and Dracle.
B.4 Case Study
In Table 12, we present bad cases of using WISE to edit the LLaMA-2-77 on the ZsRE dataset and mitigating these failures critical for future work in model editing. We observe that in i) errors occur only in part of the tokens, and these eтrоrs constitute a large proportion of the bad cases, indicating that the edits have not been sufficiently fitted. ii) displays cases where the entire output is incorrect, and factual failures indicate difficulties in retaining memory of parameters for some rare entities (such as Persian iia, Lib). iv) presents cases of generalization failure, for example in éud), where the model answered "English" but did not fully follow the ground truth, indicating significant room for improvement the accuracy of generalized edits. Meanwhile, in iii) we surprisingly find that even when WISE errs on the Edit Prompt, can correctly answer its paraphrase itib) "The kind
25

---

Table 12: Failure cases of using WISE to edit 1--aMA22-78 drepresents errors in part of the tokens, Xrepresents complete output errors (i.e., factual failures), and / /indicates the expected exact match.
Prompt
Edit Target
Post-Edi Output
ia) By which person Lahti Town Hall has been designed?
Aki Kaurismäki
Wime Kaurismäki
ib)
Which is the architect of Lahti Town Hall?
Wime Kaurismäki
ic)
Which corporation was USS Leedstown (APA-56)
Lockheed
Leez Shipbuilding
created by?
Shiphuilding
id) Which company manufactures the USS Leedstown
Leez Shiphuilding sh
(APA-56)?
kia)
Which language is Garowe Principles written in?
Persian
Dutchian
(bb) what language does the monthly football magazine Garowe Principles report?
Somian
tic) What year the service entry date for Panzer 58?
1957
1953 x
tid) What was the year Ponzer 58 was commissioned?
1953 x
iia) What was Gemma Bosini's range?
mezzo-srano
Wzo-srano
iisb) The kind of voice of Gemma Bosini is what?
mezzo-srano /
iva) In which state is Qalch Lan located?
Golestan Province
Golestan Province /
iub) What state is Qaleh Lan in?
Lestan Province X
inc) In which language Garowe Principles monthly football magazine reporting?
American English
American English /
iud) What language are Garowe Principles written in?
English English
of voice of Gemma Bosini is what?". This indicates that WISE can handle contextual information correctly in some cases but falls short in specific editing instructions, suggesting that optimizing editing instructions (modifying the editing context) may be a direction for improvement.
B.5 Importance of Knowledge Anchor When Merging Models
Table 13: Analysis of Merging w.o. and w. "knowledge anchor" (KA). T 1000. 7sRF.
Here, we discuss the effects of independent (ensured by non-overlapping masks) vs partially overlapping param- eters within MLP subspaces shown Table 13. is observable that, despite vary- ing mask ratios and the number of subspaces k, partial overlap (w. KA) consistently outperforms independent configurations (w.o. KA) in terms of Reliability (Rel.) and Generalization (Gen.). For example, at p/k of 5/0.20, there is relative improvement of 9% and 7% respectively. This demonstrates that the overlapping regions contribute as "anchors" for knowledge fusion, facilitat- ing information transfer across different subspaces. Moreover, the shared parameters provide natural regularization [90] mechanism, helping synchronize model behavior across different subspaces.
LLaMA-2-7B.
editing performance, as
KA
KA
Rel. Gen.
Loc: Avg.
Rel.
Gen.
Loc, Avg.
2/0.50
0.74
0.73
1.00
0.82
0.77
0.72
1.00
0.83
3/.33
0.68
0.75
1.00
0.82
5/0.0
0.64 0.61
1.00
0.75
0.73
0.68
1.00
0.80
B.6 Ablation Study of Random Prefix Token
Efffet of Random Prefix Token (PT)
Sell PT)
Gwn (wcs.
Gwn ("])
0.8
0.6
0.2
0.0
10
1000
T: Num Edits
Figure 12: Ablation studies on Random Prefix Token (PT) of WISE. Light/Dark colors indicate Editing Sucess w.o./w PT addition. ZsRE, LlaMA-2-7B
26

---

As described in Section 3.1. we employ random prefix token augmentation to enable the editing knowledge cope with various contexts. That is, for a single x,, expands into (prefix,,X,, X2). The prefix is derived from tokens that are randomly generated by the original LM fe. serving as an economical data augmentation method. We observe that the editing success rate compromised (Figure 12). Specifically, for instance, at T=1000, Rel. and Gen. decreased by 0.15 and 0.17, respectively. By utilizing randomly generated prefix tokens, the model is able to learn broader range of linguistic features, thereby exhibiting greater robustness in practical applications. We believe that access to the "data generator" can deepen the model's memory of editing samples.
B.7 Parameter Efficiency
pontsmmttooo (1.00
1.9
The key to lifelong model editing is maintaining con- stant or slowly increasing computational costs as the num- ber of edits expands. Here, we provide quantitative analysis using LLaMA-2-7B as an example. Suppose we select model layeer [27] wwwppdp.pmppp.... down_proj [oojjwwesg weight as side memory. In that case, the theoretically added parame- ters are 11008 x 4096 x 0.18 GB, which accounts for 0.64% of the original LLaMA's 7B x 4 = 28 GB (ignoring the VRAM required for input activations). As shown in Figure 13, practice, WISE-Merge increases VRAM by 4% compared to the original LLaMA and remains constant over time. WISE-Retriive, instead of merging, uses retrieval routing, meaning the computational cost increases over time, but this increase is gradual and can easily handle thousands or tens of thousands of inputs. Additionally, if we partially merge side MLPs (combining WISE-Retriens and WISE-Merge), we can further reduce the computational demands of WISE-Retrieve.
1.55
Rviling Megy
1.00
1.65
1.00
0.55
1900
Figure 13: Computational costs.
C Proof of Theorem 2.1
Theorem C.1 Subspace Overlap. Generate k memory subspaces W [k] by random mask with I's ratio p. so each memory has p. WWrl active trained parameters, For any two subspaces W; and w + j; i, E [k), there are p² ( active parameters that are overlapped. For all k subspaces, there are pa W.r overlapped active parameters.
Proof: We aim prove the Subspace Overlap theorem by induction.
Let W represent the i-th memory subspace generated by a random mask with a sparsity ratio of p. where E [A]] Each memory subspace W contains p- |Warl active trained parameters.
We start by considering the case of two memory subspaces, W or and W u's where * j and j, E [k:]
Let P(parameter sampled) = p be the probability that parameter is sampled in one mask generation event.
1. For a single mask generation, the probability that a specific parameter is sampled is p. We denote this probability as P(sampled) p.
2. Considering two independent mask generation events, the probability that the same parameter is sampled in both masks the product of their individual probabilities, i.e., p², This derived from the independence of the events. Mathematically:
P(sampled in both masks) Pssampled) P(sampled) p p= p².
3. Extending this logic, for k independent mask generation events, the probability that specific parameter is sampled in all masks is pt Mathematically:
P(sampled in all & masks) P(sampled) P(sampled) ... P(sampled) = p².
A times
Now, let's calculate the number of parameters overlapped in two random masks:
The total number of parameters in W,r is [.,l
Thus, the number of parameters overlapped two random masks, Wi and W is p² W.rl- Extending this to l random masks, the number of parameters overlapped in all l masks is pt W This concludes the proof.
27

---

D Detailed Related Works
Memory and Knowledge Injection of LLMs The memories of LLMs can be divided into long- term (episodic) memory and working memory (short-term) [24, 25, 27]. Long-term memory refers to the knowledge stored in the model's parameters, which can be updated by (re))rrtriining [53], finetun- ing [59], and model editing [14]. Working memory is stored in sustained accivationssrpppesenaaaoooos
of neurons, which will be awakened during inference time [24]. In-context learning (ICL) a kind
of working memory [60], also along with retrieval-based editing methods like GRACE [10]. How to reinforce memory and inject/update knowledge for LLMs is a fundamental question [28, 61, 62]. ICL or finetuning? Different works show different conclusions. In [62]. the authors find that few-shot finetuning is more generalizable than ICL, especially for out-o--distribution data. In [61]. the authors contrast finetuning with retrieval-augmented generation (RAG) in terms of knowledge injection and find that RAG is better in most cases, and combining both will produce the best results. However, finetuning and pretraining are cemputationnexpensiii
[13, 10] and usually suffer from catastrophic forgetting [63] and overfitting [64]. For ICL and RAG, the working memory is sometimes not controllable, the model may not follow the information of the contexts [24], and the context window is limited [91, 92], and there are works addressing these issues by training controllable ICL. [24], long-context [91, 92], and recurrent memory architecture design [28]. SPALM is proposed to add language models with storage modules that resemble both working and long-term memories [27].
Model Editing of LLMs Model editing can be summarized as the following lines of research. Constrained finetuning: Preliminary model editing uses constrained finetuning to update parameters based new examples [93, 94]. Locate-and-edit: ROME [18] locates the factual associations in autoregressive LLMs and conducts accurate and efficient edits by taking MLPs as key-value memories. Then, MEMIT [19] extends ROME from single-editing to mass-editing. COMEBA-HK [95] identifies the Local Editing Scope and extends MEMIT for sequential editing. In addition, T-Patcher [11] targets the last fred-forward layer of LLMs, adding an additional neuron for each edit. Meta learning: Recent meta- veeaa-lerning methods use hypernetworks for aiding editing. MEND [31] learns a hypernetwork that can decouple the finetuning gradients into the gradient updates that generalize the edits and won't 't damage the performances on unrelated inputs. To remedy the cancellation effect of MEND, MALMEN [15] uses hypernetwork to produce the weight shifts of editing and formulates the weight shift aggregation as the least square problem. netrieval-basee methods: Instead of directly editing the model parameters, retrieval-based methods aim to improve the working memory of LLMss to enable model editing. IKE [96] uses context-edit facts to guide the model when generating edited facts. DeCK [97] employs contrasting knowledge decoding, which enhances the confidence of in-context-based editors in the edited facts. SERAC [32] (a modified version dubbed as DEFER [10)) records edit items in a file and trains additional scope classifier and counterfactual model to detect, retrieve, and generate the edit-related results. Though the editing retriever and generator neural networks, they are too small to have the power of LLMs. GRACE [10] adopts a discrete codebook of edits for retrieving and replacing the edits' layer representations during inference. From single editing [18] to mass editing [15, 19), and from static editing to sequential [1 I] (continual) or lifelong editing [10], model editing is developing to meet more realistic demands.
Continual Learning Continual learning [98, 99] tackles the catastrophic forgetting problem in deep learning models with new knowledge [100], and recent research has focused various methods in this area. One such method is continual finetuning, where LLMs are refined over time with the arrival of new instances. For instance, comprehensive study by [101] explores continual finetuning extensively. However, it has been observed that regularizing finetuning with continual learning techniques such as Elastic Weight Consolidation [20], Experience Replay [102], and Maximally Interfered Replay [103] can lead to a rapid decay in performance on previous tasks, although it aids in retaining some memory of past inputs. This suggests that editing, as opposed to vanilla continual finetuning, presents unique challenges, especially considering that edits are unlikely to be evenly distributed [104]. One promising direction within the realm of continual learning is the adoption of key-value methods, inspired by advancements in computer vision [105, 106]. Recent studies have showcased the effectiveness of continual prompt-leariigg for NLP [107, 108], particularly in applications like text retrieval [109]. Notably, discrete key-value methods have been shown to excel in handling shifting distributions [110], with some recent efforts extending their application to question answering [1 [1]1]. These methods cache values to ensure that inputs remain within the distribution for downstream encoders, thus facilitating the incorporation of longer-term memory, provided there are adequate computational resources.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Section Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section Limitations
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Assumptions in Section 2.3.2 and Proofs in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report the setup throughout the paper as well as the Section 3.1 and Appendix A
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: We use publicly available datasets (Appendix A), Code and Data are also provided in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ mublic/guides. https//oooeeaaaaiinnnneneee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.ao https/ccccbpcccccpeeeeeeeess gww.s//////eeeeeeeeeeeeeeee.se Swtp://ooseeSssseeeeeeeee wwriiisssoippyyyiyo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: In Appendix A, we provide detailed descriptions of data splits and the proposed method's hyperparameters and baselines' hyperparameters. Additionally, in Section 3.3, we diseuss how select them for the proposed method.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The LLMs only have one checkpoint of the corresponding size, e.g.. LLaMA-2-78 so we only edit once for each setting. But we test our method and base- lines under various models, settings, and datasets, therefore, the statistical significance of the experiments can be verified and supported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix A.3, B.7 and Section 3.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://ewwrrs..ccuuueeeeeeeeeeee Coublicc/Ehicsciii ines?
Answer: [Yes]
Justification: We use publicly standard datasets that do not contain information about individual people or offensive context to our knowledge. Ethical considerations discussed in Section 5.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section 3.1 and Appendix A. We use publicly available artifacts.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwwde. com condaatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
33

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "LodMB0kams", "reviewer": "Reviewer_KKAN", "summary": "A fundamental question in lifelong model editing of large language models (LLMs) is where the updated knowledge should reside in the model's memory. This paper identifies an inherent challenge in editing either long-term memory (direct model parameters) or working memory (non-parametric knowledge through neural network activations/representations by retrieval). It reveals that achieving reliability, generalization, and locality simultaneously in lifelong editing settings is highly challenging in existing approaches. Editing long-term memory by directly modifying parameters can lead to conflicts with irrelevant pre-trained knowledge or previous edits, compromising reliability and locality. In contrast, retrieval-based working memory edits struggle to enable the model to understand and generalize the updates effectively. To address these challenges, this paper proposes WISE, a method designed to bridge the gap between different types of memory. WISE employs a dual parametric memory scheme, comprising a main memory for pre-trained knowledge and a side memory for edited knowledge. Edits are made exclusively in the side memory, with a router trained to determine which memory to access for a given query. For continual editing, we introduce a knowledge-sharding mechanism leveraging Ties-merging, where different sets of edits are stored in distinct parameter subspaces and subsequently merged into a shared memory without conflicts.", "questions": "N/A", "limitations": "The paper appropriately addressed limitations and broader societal impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow and nicely clarifies the challenges in lifelong model editing of LLMs in threefold: reliability, locality, and generalization, along with sufficient observations/analyses and references.\n\nThe basis for selecting later (mid-to-late) layers for side memory duplication and training sounds reasonable, and the quantitative analysis provides a reasonable demonstration of this choice.\n\nThe routing idea between long-term memory and side memory also sounds reasonable, grounded by statistical analyses.\n\nThe edit knowledge (sub)-isolation via sharding with random masks is interesting and seems to be effective in alleviating the inherent limitations that merging multiple models degenerates performance due to knowledge overlap.\n\nIn the end, the proposed method outperforms existing (lifelong) model editing baselines with improved versatility.", "weaknesses": "During the training phase, the model requires excessive memory since it requires copying multiple side memories with the same dimensionality as the original FFN weights in the model. Though it partially alleviates this issue by omitting copying side memory for earlier layers, it still needs substantial additional capacity.\n\nGradient masks for subspace memorization are randomly generated regardless of the density of target knowledge. It would be great if the model adaptively controls \\rho according to the difficulty of incoming knowledge or information quantity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer KKAN\n\nThanks for recognizing the value of our work. Your comments are highly aligned with our paper, and we hope the following comments could address your questions:\n\n> 1. Response to “During the training phase, the model requires excessive memory …”\n\nThanks for the comment. However, it is notable that the side memory that we copy is quite small, without substantial additional capacity.\n\n- First, the memory that we copy is one linear module in one certain FFN layer, e.g., the `model.layers[34].mlp.down_proj` in **LLaMA-2-13B-chat**, so the number of parameters is quite marginal (e.g., 0.64% of the whole model).\n- Second, we introduce two variants of WISE: WISE-Merge (with only one side memory) and WISE-Retrieve (with several side memories), and both of them have limited and controllable side memories.\n    - For WISE-Merge, it only introduces marginal and *constant* additional costs (**0.64%** extra parameters and **4%** extra GPU VRAM).\n    - While WISE-Retrieve has several side memories, the experimental results show that the additional costs are also limited and controllable. Importantly, Figures 6 and 12 of our submission show that at T=3000, when there are six side memories in WISE-Retrieve, it introduces only approximately 7% inference latency and memory overhead. WISE-Retrieve also benefits from the additional memories and demonstrates higher reliability (Table 6).\n\n> 2. Response to “It would be great if the model adaptively controls $\\rho$ according to the difficulty of incoming knowledge.”\n\nYour suggestion regarding adaptive control of $\\rho$ based on the difficulty or information content of incoming knowledge is insightful. We briefly discuss the insights and practicality.\n\n- Recent literature [1] highlights that language models typically store approximately 2 bits of knowledge per parameter. [2] demonstrates that the performance of LoRA [3] and DoRA [4] tends to plateau and decline with an increase in trainable parameters. They underscore the importance of appropriately scaling parameters to fit each editing sample.\n- In our experiments, we observed that utilizing a mere **0.1%** of side memory parameters (about 0.04M) often resulted in unsuccessful edits per sample. Future work could explore adapting $\\rho$ by identifying significant gradient values [5] or estimating the information entropy of incoming knowledge before input, aiming to enhance the robustness of WISE. We are glad to incorporate similar ideas in the future improved version of WISE.\n\n---\n\n[1] Allen-Zhu, Zeyuan, and Yuanzhi Li. \"Physics of language models: Part 3.3, knowledge capacity scaling laws.\" arXiv preprint arXiv:2404.05405 (2024).\n\n[2] He, Haoze, et al. \"Sparse Matrix in Large Language Model Fine-tuning.\" arXiv preprint arXiv:2405.15525 (2024).\n\n[3] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" ICLR 2022\n\n[4] Liu, Shih-Yang, et al. \"Dora: Weight-decomposed low-rank adaptation.\" ICML 2024\n\n[5] Damai Dai, et al. “Knowledge Neurons in Pretrained Transformers.” ACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bf6GYIhvMQ", "reviewer": "Reviewer_5h8i", "summary": "This paper proposes WISE which uses a side memory and some model merging techniques to perform lifelong knowledge editing.", "questions": "1. **Experimental Results**: In MEMIT[1], they achieve 96.7 on Efficacy and 89.7 on Paraphrase (see Table 1) with 10,000 edits, but in Table 6, MEMIT-MASS has 0.64 in Rel. and 0.58 in Gen. under 2000 edits and 0.58 in Rel., 0.53 in Gen. under 3000 edits. What happened to the model to contribute to the discrepancy between the current results and the results reported in the paper?", "limitations": "The limitation of ever-expanding memories may need to be mentioned in the limitation section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. \n2. The experiments are somewhat comprehensive. \n3. The paper is working on the continual editing problem, which is important.", "weaknesses": "1. **Ever-expanding memory**: As mentioned in line 212-213, \"One single side memory has its limited knowledge capacity\". Thus, to perform lifelong knowledge editing, it seems that infinite side memories would be needed. In this sense, lifelong knowledge editing is still not solved as with limited side memories, only a limited number of edits can be performed. As the ever-expanding external memory is needed, why do we need to store all this knowledge in the memory instead of just storing the knowledge in the form of raw text and performing knowledge retrieval from this knowledge base? \n2. **Experimental Results**: In the paper MEMIT [1], they scale the batch editing to 10000 examples. There are around 20000 examples in zsRE dataset. However, in the main table (Table 2) only up to 1000 edits are studied. \n3. **Sequential Editing Cases**: Two possible scenarios in sequential editing are: (1) There would be conflicting knowledge overtime. For example, The president of some country may continue changing every a few years. How does WISE perform in this case compared to others? (2) Multi-hop edits as discussed in [2]. However, it seems that both of these situations are not discussed in the paper. \n\n[1] MASS-EDITING MEMORY IN A TRANSFORMER.   \n[2] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer 5h8i**\n\nWe thank the reviewer for acknowledging that our paper is well-motivated and the experiments are comprehensive. We kindly address your questions as follows.\n\n> 1. Response to “Why not store the knowledge in the form of raw text and perform knowledge retrieval (RAG)?”\n> \n\nThanks for your valuable comments. **We kindly refer to the general response for the concern of “ever-expanding memory”.** Then, we will elaborate on the following aspects and explain why WISE is irreplaceable compared to raw text retrieval:\n\n1. **Lifelong Perspective on Knowledge Update**\n    - **Fact Reasoning/Utilization**: Let us consider a simple example: Suppose we aim to edit A: \"change *Oliver Twist*'s author to *Shakespeare*\" and then to edit B: \"change *Shakespeare*'s nationality to *French*\". If we query \"In which country was *Oliver Twist* written?\", a similarity-based retrieval would likely only retrieve A. Using raw text cannot establish generalized knowledge associations both externally and within the model, resulting in weak generalization and poor reasoning abilities. Even recent literature has begun using knowledge graph construction for GraphRAG to mitigate this dilemma, but it is inevitably limited by information extraction technology's inability to establish precise associations.\n    - **Adaptive Chameleon or Stubborn Sloth [1]**: LLMs demonstrate a strong confirmation bias toward parametric memory when faced with both supportive and contradictory evidence to their parametric memory [1]. This means that RAG faces additional challenges in optimizing raw text to be coherent and convincing.\n    - **Efficiency of Inference**: Retrieval itself is a massive engineering system; common embeddings, retrieval, rankers, and synthesis all cause significant delays in online systems.\n2. **Synergy Between WISE and Raw Text Retrieval**\n    - WISE and RAG are independent; WISE updates internal model knowledge as parametric knowledge, which can be combined with RAG (non-parametric knowledge) to achieve more robust knowledge updates.\n\nTo validate these points, we conduct comparative experiments on the MQuAKE dataset (evaluation on $x_e$'s multi-hop query). For **ICE** (In-Context Editing, i.e., RAG), we choose `all-mpnet-base-v2` and retrieve the Top-1 cosine similarity editing fact to construct the prompt: `[Updated Information]: \\n {retrieved-editing-fact} \\n [Query]: {query}`. Results are shown in **Figure A** of the ***Rebuttal PDF*** **in general response**.\n\nWe first observe that **WISE + ICE consistently outperforms** WISE or ICE alone, proving our point that combining the two achieves better multi-hop reasoning performance. Secondly, as T increases to 1000 and 2000, **WISE demonstrates the advantage of utilizing editing facts x**, combining and associating knowledge to enhance reasoning. \n\n> 2. Response to “MEMIT scales the batch editing to 10000 examples.”\n> \n- As shown in Appendix A.2, MEMIT-MASS does not belong to sequential edits but rather to parallel/batch edits (reviewer fviW also points out this), **losing the capability for on-the-fly repairs.** The main table also demonstrates MEMIT's sequential editing performance, akin to ROME, where the model is fully compromised by T=100.\n- Lifelong editing was first proposed in GRACE. We follow GRACE's settings and dataset, reporting editing performance at T=1000 in the main table, with analysis extending to T=3000 in Section 3.3. Comprehensive analysis from Table 6 and Figure 11 shows that WISE is capable of scaling up to more edited samples. We recommend using WISE to respond to each mistake online promptly. After accumulating a sufficient number of mistakes, fine-tuning the original model on all accumulated mistakes allows for the removal of side memories. Then, WISE can be reused for on-the-spot repairs. The ability to scale to thousands/ten thousands of edits can meet the requirements of real applications for deployed online models.\n\n> 3. Response to the issue about “the discrepancy between the current results and the results reported in the MEMIT paper”.\n> \n\nThis discrepancy arises from **different metrics yielding different results**. We follow GRACE's metrics. As shown in Equation 9 of our paper: $\\text{Rel.} = \\frac{1}{T}\\sum\\limits_{t=1}^{T} \\mathbb{1}(f_{\\Theta_{T}}(x_e^t) = y_e^t)$. In MEMIT paper 5.2.2, Efficacy Success (ES) is defined as the probability of the new object $o'$ being greater than the original object $o$: $E[P_{G} [o_i' | p(s_i, r_i)] > P_{G}[o_i | p(s_i, r_i)]]$. We argue that determining editing success based on **real output alignment is stricter (thus, smaller value)** and more aligned with real-world scenarios.\n\n> 4. Response to the issue about “Knowledge Conflict in sequential editing”.\n> \n\nIn Appendix B.3, we discuss the accuracy of Retrieving Top-1 Activation, demonstrating the efficacy of WISE in identifying the correct side memory. We propose an inspection module as a remedy for WISE: \n\n- First, for a given editing knowledge, use routing activation to *check whether it matches past editing sequences and identify the corresponding side memory*.\n    - If so, cover previous knowledge through re-editing this side memory using current new knowledge in a tiny subspace (e.g., selecting a 0.01 mask parameter), which will rewrite the previous conflicting knowledge while minimally affecting other editing knowledge.\n    - Otherwise, conduct vanilla WISE steps, i.e., introduce new parameter subspaces for inserting new knowledge.\n\n> 5. Reponse to “Editing results of Multi-hop query (MQuAKE dataset).”\n> \n\nWe have supplemented these results in **Figure A** of the ***Rebuttal PDF*** **in general response**. The discussion regarding these results is shown in the second response in your thread. We will further refine this experiment and update the manuscript.\n\n---\n\n[1] Jian Xie, et al. “Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts.” ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZgFWv1clzm", "reviewer": "Reviewer_pT3T", "summary": "The paper studies the problem of editing (updating) knowledge of LLMs in a lifelong learning scenario. Authors design WISE, a multi-level memory system designed to store updates to the model. The proposed design contains of the main memory and a number of side-memories, with a mixture-of-expert-like router to choose between them. Authors also propose a procedure for merging the main (mid-term) memory into side-memories effectively committing it to the long-term memory. Authors evaluate the proposed technique on small variants of popular LLMs, achieving good average performance on ZsRE and a number of other datasets (SelfCheckGPT, Temporal). The paper also contains several side-analyses such as visualizing the router behavior, scaling to 3K edits, and speed benchmarks.", "questions": "I do not have any insightful questions, so I will instead use this section for minor comments.\n\n### Typos, missing citations, minor concerns\n\n> L389 Malicious users may attempt to edit LLMs to propagate hate, highlighting the need …\n\nThere are numerous other scenarios for a potential misuse of this technology. To name a few: \ncensorship, particularly for non-democratic state actors\nmisinformation, of a non-hate-inducing kind\n\nFor the record: i do NOT mean that the paper requires an ethical review. Most of these attack vectors are already possible with prior work. It would be unrealistic to expect a full sister study on ethics. Fortunately, we have AI ethics researchers.\n\n\n> L34 should satisfy the following properties [ 14 , 15, 11 ]\n\nI believe this terminology were originally introduced earlier in Sinitsyn et al (2020) [ https://openreview.net/forum?id=HJedXaEtvS ] and De Cao et al (2021) [ https://aclanthology.org/2021.emnlp-main.522/ ], including for language models. Though the term “LLM” specifically did not exist back then. \n\n> L25 parameters, computes, and data\n\nWhile using “compute” as a noun is not yet well studied, I have seen it mostly used as an uncountable noun (like “data” instead of “datas”). Please use your own judgment though.\n\n> L98 (definition formula for D_edit)\n\nTo the best of my knowledge, using | for nested definition of Xe,Ye is rarely used (or understood) by ML practitioners. Consider defining them separately\n\n\n\n\nOverall, if this paper ends up accepted, I respectfully ask that authors reduce the total negative (vertical) space used in formatting by exiling some of the less important analyses to appendix or, if all else fails, by using the extra content page granted for the final version.", "limitations": "To the best of my knowledge, authors have sufficiently addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem of updating LLM parameters is important and a practical improvement in this direction can reduce the total cost of using LLMs (both economic and environmental)\n\n- The paper contains a variety of experimental configurations and baselines, including several models of different capability (and creation date), different datasets, and numerous baseline algorithms. This would normally be expected of a NeurIPS submission, but sadly, often not the case. The only possibly unexplored dimension is model size: testing with 6-7B models may not reveal some caveats that arise only in larger ones.\n\n- As a minor but helpful touch, the provided code contains clear instructions on running the code and well-defined dependency versions. This helps future researchers reproduce and build on top of this work.\n\n\nThe paper is also generally well written and reasonably well structured, though I got the impression that authors tried to squeeze a lot of information into few pages. If this paper ends up accepted, I respectfully ask that authors reduce the total negative (vertical) space used in formatting by exiling some of the less important analyses to appendix or, if all else fails, by using the extra content page granted for the final version.", "weaknesses": "I have two main concerns, though they are not significant ones.\n\nFirst WISE is a rather complicated system with a lot of moving parts: router type, merging strategy, main/side memory sizes, where to introduce this module into an LLM, and how many times, how best to allocate the memory size between components. Authors provide some ablation analysis (e.g. Appendix B.2 and below), but many of these ablations are missing.\n\nMy second (very minor) concern is that authors experiment only with small LLMs (sic), which leaves out the possibility that WISE behaves unexpectedly with larger and higher-capability ones (at the minute of writing this, Llama-3-70B, Qwen-2-72B, Nemotron et al.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer pT3T\n\nThanks for your valuable feedback. We appreciate the opportunity to address your concerns.\n\n> 1. Response to “many of ablations are missing.”\n> \n\nThanks for the comment. All components of WISE can be summarized as router, merging strategy, locating side memory, and side memory sizes. Indeed, except for the merging strategy detailed in Appendix B.2, these components are either ablated (if applicable) or analyzed in the main body of the paper. We discuss the independent contributions of these components as follows:\n\n- **Router**\n    - As shown in Figure 3, we present visualizations of WISE router activations. The purple horizontal line in the figure represents the minimum activation threshold $\\epsilon$ recorded during the editing phase. Nearly all irrelevant queries (blue) show lower activations, while inputs within the editing domain ($x_e$ or its paraphrased form $x_e'$) exhibit higher activations (red), underscoring the efficacy of the router. Without the router strategy, all inputs either pass solely through the main or side memory. To further validate its effectiveness, we conduct additional ablations with $L_a$. WISE's performance on ZsRE is shown in **Table A** of the ***Rebuttal PDF*** **in the general response**.\n    - Observing the expected **decrease in Loc.** w.o. $L_a$ , such as dropping from 1.00 to 0.72 at T=1000, reveals the router's effectiveness in identifying editing scopes, minimizing side effects, and retaining a substantial amount of pre-training knowledge.\n- **Merging strategy**\n    - We provide detailed ablations in Appendix B.2. Table 10 shows that simple linear/slerp interpolation performs poorly, whereas Ties and Sign excel. It also effectively demonstrates a) the necessity of non-redundancy of masked knowledge shards; b) we must resolve conflicts during merging to ensure maximal retention of all past edited samples.\n- **Main/side memory sizes**\n    - The main memory is part of the original LLM, hence its size is not under our control. Side memory replicates a layer of FFN from the LLM, initialized with its weights. Theoretically, it requires 0.64% of the original LLM's parameters (using LLaMA-2-7B as an example).\n- **Where to introduce this module into an LLM, and how many times**\n    - *Where to introduce*: Extensive literature [1,2] suggests LLMs encode high-level semantics in mid-to-late layers and handle more advanced linguistic phenomena. As shown in Figure 4, we select early, intermediate, mid-to-late, and late stages, finding superior editing performance in mid-to-late layers, possibly due to the formation of more complex semantics/knowledge. Overall, we provide guiding recommendations for \"Where to introduce.\" Additionally, we attempt to generalize to the 13B model, as shown in **Table B** of the ***Rebuttal PDF***, confirming these findings.\n    - *How many times*: In Figure 5, we discuss knowledge density (closely related to \"how many times\"), finding optimal $\\rho^k$ consistently near 0.03, with good performance within the interval $\\rho$ =[0.3, 0.5] (exhibiting robustness). This suggests that even without prior knowledge of incoming knowledge count (T), current experiments indicate that 0.2 FFN parameters can accommodate at least 500 edited samples. When \"mask memory exhaustion\" occurs, we can allocate new mask parameters to store new knowledge. Using retrieve when knowledge isn't full and merging as needed to save memory, achieves true lifelong model editing.\n\n> 2. Response to “WISE's behavior with larger and higher-capability LMs (e.g. LLaMA-3-70B, Qwen-2-72B)”.\n> \n\nThanks for your suggestions. In this paper, we explore the latest (at least at the submission stage) and popular LLMs (LLaMA-2-7B, Mistral-7B), observing significant and consistent improvements with WISE across multiple experimental settings. Regarding LLMs of the 70B scale, we lack sufficient resources to conduct such experiments. However, to the best of our abilities, we attempt to observe WISE's performance on **LlaMA-2-13B-chat**, as shown in **Table B** of the ***Rebuttal PDF*** (choosing the mid-to-late layer: `model.layers[34].mlp.down_proj`):\n\nExperimental results validate WISE's efficacy on the 13B-chat model, even surpassing editing performance on the 7B model at T=1000. This enhances WISE's scalability across model sizes, and we plan to further refine experiments and attempt editing larger 70B models. Once again, thank you for your suggestions.\n\n> 3. Response to “Typos, missing citations, minor concerns”\n> \n\nMany thanks for your kind attention and carefulness. We make the following responses.\n\n- Ensuring safe and responsible practices in WISE is of paramount importance. Model editing, which can suppress harmful language generation [3-5], could help address these concerns.\n- We have added citations for Sinitsyn et al. (2020) and De Cao et al. (2021) in the updated version of the paper and removed the incorrect term \"compute\" in L25.\n- We have also added a reference to Table 7 when introducing D_edit, which provides specific cases of x_e and y_e. Thank you for pointing out these issues.\n- We will remove all \\vspace in the CR phase to ensure the paper's formatting is more consistent.\n\n---\n\n[1] Tianjie Ju, et al. ”How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study.” *LREC-COLING 2024*\n\n[2] Yung-Sung Chuang, et al. ”DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models.” ICLR 2024\n\n[3] Xinwei Wu, et al. “DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models.” EMNLP 2023\n\n[4] Mor Geva, et al. “Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space.” ACL 2022\n[5] Mengru Wang, et al. “Detoxifying Large Language Models via Knowledge Editing.\" ACL 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5XsxiqhtA", "reviewer": "Reviewer_fviW", "summary": "Introduces a new technique, WISE, to edit/insert facts in a autoregressive language model. The proposed approach gives a balanced performance in terms of reliability, generalization, and locality; improving upon the existing methods. WISE edits facts in a side-memory that is equivalent to one of the MLP down projection layers. A router is trained to decide if a specific query should be routed to the side-memory or the corresponding layer in the main LM.", "questions": "1. The activation $\\textbf{a}$ used in routing correspond to which token in the prompt? Is it the subject last token like ROME/MEMIT or the last token of the prompt like GRACE? \n    * One of the reason GRACE achieves poor generalization is because it edits the last token of the prompt and thus very sensitive to the prompt paraphrases. The notation used in the paper suggests that WISE also uses the last token of the prompt (clarify this if I am wrong). However WISE seems to achieve an impressive generalization. Is that because the parameters of $\\mathbf{W}_{v'}$ are directly edited for better routing? Also, does WISE use only one prompt $x_e$ to edit a specific fact, or does it use multiple paraphrases of $x_e$ during the training phase?\n\n2. Parameter Overlap: Theoren 2.1 seems to counts parameter overlap as the number of parameters that are common between *all* the $k$ side-memories as $\\rho^k|\\mathbf{W}|$. But if a parameter is common between only two side-memories, it is also a overlap, right? And, in that case, the number overlaps would be $\\Big(\\binom{k}{2}\\rho^2 - \\binom{k}{3}\\rho^3 + \\binom{k}{4}\\rho^4 \\cdots \\Big) \\cdot |\\mathbf{W}|= \\sum_{i=2}^k (-1)^i \\binom{k}{i}\\rho^i \\cdot |\\mathbf{W}_{v'}|$.\n    * I also didn't follow the point on the overlapping playing the role of \"*anchors*\" for knowledge merging, even after refering to Appendix B.5, which just restates the same point. Can you please clarify this?\n    * I also didn't follow the claim made in line 191-192 => *\"different shards of edits have different random masks, and due to the (sub)orthogonality of random masks, different shards will not conflict with each other\"*.\n\n3. Although WISE claims to be sequential (line 222), it doesn't seem to be the case, as you expect access to $n$ edits to distribute among $k$ shards (line 179-180). Or, is the loss $L_{edit}$ optimized for each of the edits sequentially? Also, how do you perform the merging if you don't have access to all the edits beforehand? If WISE was truly not done sequentially, then it should only be compared with methods that permit parallel/batch edits.\n\n4. As you increase the number of edits $T$, do you also increase the number of sharts $k$? Or, do you always keep $k = 2$ reported in Table 9? Also since the title of the paper focuses on *Lifelong* model editing, I wonder if it is reasonable to increase $k$ rather than choosing a fixed $k$.", "limitations": "Limitations have been adequately addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* WISE is a novel approach that balances reliability, generalization, and locality while editing facts in a LM.\n* Clear motivation and solid technical contributions.", "weaknesses": "Needs more clarity on some of the technical details. See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer fviW\n\nThanks for your valuable feedback. We appreciate the opportunity to address your concerns.\n\n> 1. Response to “The activation $\\mathbf{a}$ used in routing corresponds to which token in the prompt?”\n> \n\nAs shown in Equation 3, the activation used for classification is derived from the entire prompt, meaning $\\Delta_{\\text{act}}$ represents the **average activation across all prompt tokens**. This increases the probability of correctly classifying the paraphrase $x_e'$.\n\n> 2. Response to “Impressive generalization of WISE.”\n> \n\nThanks for the comment, we suppose there are three key reasons behind WISE's impressive generalization:\n\n- By averaging activations across all prompt tokens, we enhance the model's ability to identify the scope of edits, allowing paraphrase $x_e'$ to pass through side memory while unrelated inputs $x_i$ pass through main memory.\n- $\\mathbf{W}_{v'}$ is directly modified (Long-term Memory), copying FFN parameters from the original LLM and memorizing new editing samples with a small portion of parameters. During inference, side memory and earlier layers are packaged together, enhancing overall representation capabilities beyond activation-based methods.\n- We employ random text augmentation (avoiding leakage of paraphrase inputs). Initially utilized in ROME: For a single $x_e$, it expands into (prefix_i, $x_e$). The prefix is derived from tokens randomly generated by the original LM, serving as an economical data augmentation method.\n\n> 3. Response to the issue of “parameter overlap”.\n> \n- **Why $k$-memories’ overlap:** Because during model merging, we merge several parameter subspaces into one, not pair-wise merging, we consider the $k$-memories’ overlap.\n- **Explanation of the “anchors”:** For vanilla mode merging, full models are fine-tuned to merge, so the overlap is 100% in the original Ties-Merge. While in our random subspace, only a small proportion ($\\rho$) of parameters are updated in a subspace. Therefore, when conducting model merging, it is necessary to have overlapped parameters to rescale and rectify (e.g., trim/select sign in Ties-Merge) different model pieces.\n    - We also made an ablation study. In Table 12, \"w. KA\" (generate random subspaces with replacement) consistently outperforms \"w.o. KA\" (generate disjoint subspaces by random sampling without replacement), demonstrating that anchors improve performances.\n- **(Sub)orthogonality of random masks:** Since the proportion of subspace overlap is small (e.g., 0.03), most of the parameters are disjoint, which means that the knowledge in different subspaces will cause little conflicts; here, we note this as sub-orthogonality. There is a tradeoff: avoiding knowledge conflicts requires subspaces to have most parameters disjoint, but the parameters cannot be totally disjoint, otherwise the model merging will be poor, so we intuitively observe nuances in the tradeoff where 0.03 of overlap can realize the best result across different $\\rho$ and $k$.\n\n> 4. Response to “Does it use multiple paraphrases of $x_e$ during the training phase?”\n> \n\nSorry for the confusion caused, we will elaborate on this point as follows.\n\n- As mentioned above, we employ LM-generated token augmentation during the training phase to enable the LM to remember knowledge updates across different contexts. This approach is also employed in ROME, MEMIT, FT, etc., as documented in the supplementary material.\n- However, we acknowledge the oversight regarding GRACE, which accepts only bs=1 inputs. In the rebuttal phase, we've completed these experiments, applying the 11 \"augmented data\" instances generated by LMs to GRACE. When epsilon_init = 500.0, the results of GRACE are shown in **Table C** of the ***Rebuttal PDF*** **in the general response.**\n    - Overall, GRACE demonstrates generalization (e.g., Gen. is 0.28 higher at T = 1 compared to Table 1). However, at T = 1000, due to the inability of the last token (as mentioned in Appendix B.1) to represent semantics, its generalization performance remains inadequate. We will update the main table with GRACE's experimental results.\n- We conducted an ablation study on random text augmentation, the final performance is in **Figure B** of the ***Rebuttal PDF.***\n    - We observe that **the editing success rate is compromised**. We believe that access to the \"data generator\" can deepen the model's memory of editing samples, and this constant overhead is worthwhile. This finding will also be added in the Appendix.\n\n> 5. Response to “Perform merging though we don't have access to all the edits beforehand.”\n> \n\nIn Figure 5, we discuss knowledge density, finding optimal $\\rho^k$ consistently near 0.03, with good performance within the interval $\\rho=[0.3, 0.5]$ (exhibiting robustness). This suggests that even without prior knowledge of incoming knowledge count (N), current experiments indicate that 0.2 FFN parameters can accommodate at least 500 edited samples. When \"mask memory exhaustion\" occurs, we can allocate new mask parameters to store new knowledge. Using retrieve when knowledge isn't full and merging as needed to save memory, achieves true lifelong model editing.\n\n> 6. Response to “The number of shards k when T increases.”\n> \n\nWe do not only test the editing performance for k=2. As shown in Figure 5, k=2 and $\\rho$=0.2 exhibit optimal performance at the scale of T=1000, which is why it was reported in the main table. Figure 5 also presents results for k=[2,3,4,5,6], revealing interesting conclusions about knowledge density. Furthermore, in Table 6, we extend the number of editing instances to 3K, resulting in 6 merges (3000 / 500). Please refer to General Response 1, where we discuss that as k increases, single side memory has its limited knowledge capacity. Therefore, combining WISE-Retrieve with timely merges and fine-tuning after accumulating sufficient mistakes can meet the requirements of real applications for deployed online models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We thank all the reviewers for their time and for providing constructive comments to enhance the paper. We appreciate that reviewers recognized:

- Our paper is well-written, easy to follow, and has clear motivation (balancing Reliability, Generalization, and Locality in lifelong model editing) (Reviewers KKAN, 5h8i, fviW).
- Experimental contribution via robust and comprehensive experimental setup (several models, different datasets, and numerous baseline algorithms) (Reviewers 5h8i, pT3T).
- Solid technical contributions (reviewer fviW) and sufficient analyses (Reviewers KKAN, pT3T).

---

We include the following additional experiments in the **Rebuttal PDF of this general response**, hoping it could relieve your concerns. Specifically, the added results are as follows.

1. (**"Scale to Larger LMs (13B-chat)" experiments**) [*Reviewer pT3T*] WISE demonstrates exceptional scalability, surpassing even the performance of editing the 7B model. (Rebuttal PDF Table B)
2. (**"Ablation study on Router" experiments**) [*Reviewer pT3T*] In addition to visualizing router activations in Figure 3, we further ablate the router, demonstrating its ability to identify editing scope to ensure robust local editing. (Rebuttal PDF Table A)
3. (**"Fact Reasoning/Utilization on MQuAKE" experiments**) [*Reviewer 5h8i*] When the number of edits reaches 1000 or more, WISE (Parametric Memory) establishes superior knowledge associations, surpassing in-context editing (i.e., raw text retrieval). (Rebuttal PDF Figure A)
4. (**"Ablation study on Random Prefix Token (RPT)” experiments**) [*Reviewer fviW*] WISE economically generates random tokens from the original model, enhancing editing generalization/adaptation to diverse contexts. (Rebuttal PDF Figure B and Table C)

---

### **Common Questions.**

> 1. Response to the issue of “Ever-expanding memory” [Reviewer 5h8i] and “Lifelong model editing” [Reviewer fviW].

**Answer:** 

- **The editing scope of (lifelong) knowledge editing:** As introduced in the literature, knowledge editing (i.e., model editing) serves as a knowledge injection method to bridge the gap between finetuning and timely online requirements. Also, lifelong knowledge editing (i.e., continual/continuous editing) focuses on the scenario where there are growing and sequential knowledge edits. It is worth noting that if the number of edits reaches a specific point, conducting finetuning using the accumulated data is fine, and our WISE can provide a more elegant approach by merging the side memories into the main memory by model merging techniques. In addition, in previous lifelong model editing literature, the maximal editing scope of GRACE is 3000 (T=3k) in their experiments, while in our paper, we also follow this setting.
- **Our WISE’s advantages:** Comprehensive analysis from Table 6 and Figure 11 of our original manuscript shows that WISE is capable of scaling up to more edited samples. Additionally, our method is an AI-native memory that explicitly builds knowledge connections in model parameters; therefore, it has great generalization compared with retrieval-based methods, e.g., GRACE. Also, our WISE is orthogonal to RAG (in-context editing, ICE). In Figure A of the ***Rebuttal PDF***, it is found that our method can surpass RAG/ICE and reach a higher point when combining it.
- **Remedy and future improvement:** We will give some suggestions on the memory management of WISE as a remedy for ever-expanding side memories. Akin to LRU (Least Recently Used) in Operation System, we can use similar ideas to delete or clean some side memories that are rarely accessed. In addition, merging the side memories back into the main memory by model merging techniques is also an approach; though the knowledge conflict problem may exist when doing so, the merged memory may also serve as a better initialization for finetuning the model on the editing dataset.

Meta-review: This paper identifies the fundamental challenges in lifelong model editing of LLMs, where reliability, generalization and locality cannot be realized together in the lifelong editing settings. It then proposes a new method to bridge the gap between different types of memory. It develops a dual parametric memory scheme, where a main memory is for pre-trained knowledge and a side memory for edited knowledge. A router is trained to determined which memory to access for a given query. A knowledge sharing mechanism with Ties-merging is introduced for continual editing.

All the reviewers recognized that this paper is well written with comprehensive experiments. The overall proposed designs make sense and are sufficiently supported by the experiments. In addition, all the concerns raised by the reviewers are addressed in the rebuttal period. The authors are encouraged to take the feedback into account in revising the paper (e.g., making it less dense for clarity of presentation).
Decision: Accept (poster)