Title: Initializing and Retrofitting Key-Value Adaptors for Traceable Model Editing
Venue: NeurIPS 2024
Accepted: no
Keywords: natural language processing

model editing

language model

key-value adaptor
Abstract: As the insight of knowledge storage in language models deepens, the ability to perform CRUD (Create, Read, Update, Delete) operations on language models becomes increasingly indispensable for satisfying the demands of managing rapidly updating knowledge. Considering the high cost of fine-tuning language models, model editing methods with low cost are usually required to manipulate models' knowledge. Evident suggests that modules carrying knowledge in a Transformer module are primarily the MLP blocks, thus we propose \textbf{iReVa}, a method that explicitly initializes and retrofits key-value pairs into MLP blocks to construct a new mapping of a piece of knowledge without damaging the irrelevant knowledge. In comparison to existing methods, iReVa reveals better interpretability and stronger capacity for carrying traceable edits. Experiment results on series of GPT series models show our prominent performance on edit success and generalization without influencing specificity. We also perform the first attempt at conducting knowledge withdrawal test of iReVa. Our codes are available at github.com/thartvigsen/grace.

Abstract

As the insight of knowledge storage in language models deepens, the ability to perform CRUD (Create, Read, Update, Delete) operations on language models becomes increasingly indispensable for satisfying the demands of managing rapidly updating knowledge. Considering the high cost of fine-tuning language models, model editing methods with low cost are usually required to manipulate models' knowledge. Evident suggests that modules carrying knowledge in a Transformer module are primarily the MLP blocks, thus we propose iReVa , a method that explicitly initializes and retrofits key-value pairs into MLP blocks to construct a new mapping of a piece of knowledge without damaging the irrelevant knowledge. In comparison to existing methods, iReVa reveals better interpretability and stronger capacity for carrying traceable edits. Experiment results on series of GPT series models show our prominent performance on edit success and generalization without influencing specificity. We also perform the first attempt at conducting knowledge withdrawal test of iReVa. Our codes are available at github.com/timberflow/iReVa.git.

1 Introduction

Language Models (LMs) [1] are becoming imperative tools for consulting in real-world scenarios. One significant reason for the prevalence of LMs is their ability to answer factoid questions. For example, when we ask an LM with the question ' Who is president of America ? ', it returns the answer ' Joe Biden '. Even though a mass amount of knowledge is stored in the LMs, we still face the issue of out-of-date and missing knowledge [2, 3]. Alternatively, some knowledge may change over years and some domain-specific knowledge may be absent from the LMs.

To bridge the gap, the task of model editing is introduced to edit the knowledge in LMs, which targets at conducting change to the parameters of LMs and inject certain knowledge to them [4]. The difficulty of this task lies in the manipulation to the LMs, where the knowledge is implicitly stored in dense vectors. A naive solution to model editing is fine-tuning a LM with the new knowledge, whereas the cost is climbing with the surging size of LMs. More recent studies propose to directly update the models' weights in mastery phase [5, 6] via either teaching a hyper-network to learn the change of the weights or locating-then-editing knowledge neurons [7, 8, 9, 10]. While the editing methods above are efficient in updating knowledge in LMs, they encounter the difficulties of differentiating the existing and new knowledge, which makes the editing hard to control. Methods like life-long model editing [11], MELO [12], and T-Patcher [13] propose to learn the representation for new knowledge and merge this information with the original models.

However, these methods still conform to the paradigm of learning the batch edit [13, 14] as a 34

whole without modeling edit parameters in a traceable way, which can not conform the edit success 35

to each edit and have a lack interpretability to the editing. In contrast, we propose a method of 36

I nitializing and R etrofitting K E yV alue A daptors ( iReVa ), an editing method that inserts a key-value 37 adaptor to indicate the mapping of an edit data pair and further retrofit the adaptor with multiple 38 objectives. Moreover, to prevent the unnecessary change to the irrelevant knowledge, we elaborately 39 design activation mechanism for the knowledge neurons. Experimental results on series of GPT-like 40 models show that iReVa is able to outperform the SOTA results by around 9% and 6% average score 41 improvement on zsRE-10K and PARAREL-10K, respectively. Moreover, iReVa is able to perform 42 knowledge withdrawal in almost perfect condition. 43

Our contributions are summarized as follows: 1) We introduce a novel editing method which initializes 44 and retrofits a key-value adaptor for traceable model editing, which is compatible to most LMs. 2) 45 Our method outperforms recent baselines on model editing tasks with noticeable margins based on 46 various evaluation metrics. 3) We validate the interpretability and generalization capabilities of our 47 method by conducting further analysis such as knowledge withdrawal test and generalization test. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

2 Related Work

2.1 Insight of Knowledge Storage in Language Models

As pre-trained LMs show strong abilities to answer factoid questions. Discussion about how LMs store knowledge has emerged. [2] introduced the perspective of treating LMs as knowledge bases and proved its plausibility, which attracted the subsequent attention towards the exploration on the form of knowledge incorporated by LMs. The opinion pointed out by [15] indicates that factual knowledge is stored in two-layer-FFN network of a Transformer due to the similar form as keyvalue memories. This opinion was followed by [16], which further derives the coefficient between final prediction and knowledge neurons in MLP blocks. In contrast, [9], through a casual-tracing experiment, posed viewpoints that knowledge is stored in self-attention module. [7] further validates that the weight update is concentrated on parameters in self-attention module when we train models with new knowledge. Our editing method is built upon the former hypothesis and we focus on the editing to the MLP blocks.

2.2 Editing LMs by Manipulating Knowledge

With the frequent update of the knowledge, the demand of model editing increases. Diverse studies 63 have been proposed. By analogy with human knowledge acquisition, we can categorize the editing 64 into three distinct phases. In recognition phase [17], methods such as ERAC and IKE [8, 18] solved 65 the problem by importing additional memories in the form of a relevant contexts or prompts. In 66 association phase [6], parameter-efficient tuning [19, 20, 12, 11] inserts low-rank adaptors or prefix 67 token embeddings to fine-tune new knowledge and combine them to the original models. There are 68 also some studies directly changing the weights of Transformers in mastery phase [5]. For example, 69 [7] proposed KE and [8] proposed MEND to predict the updated parameters of a model with a trained 70 hyper-network. Furthermore, ROME [9] and MEMIT [10] compute the weight update explicitly with 71 proper representations of knowledge queries and values. However, none of them focuses on traceable 72 model editing, which allows more flexible manipulation of the knowledge. 73

74

3 Problem Formulation

We follow the previous studies [21, 12, 11] to formulate the task. Suppose we are given a base model 75 that could be a pre-trained language model f Φ parameterized by Φ , model editing aims at editing f Φ 76 with a dataset D in = { ( x 1 , y 1 ) , ..., ( x i , y i ) ..., ( x n , y n ) } , where ( x i , y i ) denotes the edit input-output 77 pairs. Initially, for x i ∈ D in , the base model makes prediction ˆ y i = f ( x i ) but ˆ y i = y i . In this case, 78 we change f Φ by editing its parameters to Φ ∗ . A good model editing to f Φ ∗ should satisfy: 1) for 79 any x i ∈ D in , the edited model f Φ ∗ should output desired predictions, that is f Φ ∗ ( x i ) = y i ; 2) for 80 any input out of the scope of D in , which is denoted as D out , the edited model f Φ ∗ should retain 81 the original predictions, that is f Φ ∗ ( x i ) = f Φ ( x i ) ; 3) the edit of ( x i , y i ) towards f Φ ∗ should not 82 influence any prior edits x <i ∈ D in . 83

̸

84

Key interpretability:

𝑥

!

:

The Divine Comedy

is written by ?

…

𝑥

"

:

Who is the

president of America ?

Value interpretability:

𝑦

!

:

𝑦

#

:

…

𝑦

"

:

𝑦

$

:

New York City

𝑥

!

𝑥

"

𝑥

#

𝑥

$

𝑥

%

𝑥

&

𝑜

i

𝑥

!

𝑥

%

𝑥

&

𝑥

%

𝑥

"

𝑥

%

𝑥

&

0

𝑥

%

𝑥

!

(a) Training process

key weights

K &

value weights

V

&

𝑉 &

𝑦

!

:

𝑦

"

:

𝑦

#

:

𝑦'

$

Bidden

𝑦

!

(b) Inference process for in-scope edit

(c) Inference process for out-of-scope edit

𝐾 (

[𝑉 ⊕ 𝑉

& ]

⊺

𝑔

()*

([𝐾 ⊕ 𝐾

( ]

⊺

i)

original knowledge

neurons

new knowledge

neurons

𝑦

%

:

New York City

4 Method

To develop an editing method that supports traceable edits to knowledge neurons, we introduce 85 a novel method ' iReVa ' that i nitializes and R etrofits k E yV alue A daptors for traceable model 86 editing. The pre-trained LM f Φ usually contains Transformer blocks, which consist of intertwined 87 self-attention and feed-forward layers. The prior studies [15] have shown that the inside MLP blocks 88 are commonly deemed as the neurons for storing implicit knowledge. Our method is able to insert new 89 knowledge but without damaging the irrelevant knowledge in the models by inserting and retrofitting 90 the key-value adaptors to these blocks. 91

Figure 3 depicts the architecture of our proposed method. For a two-layer-FFN MLP block in the l -th 92 layer of the original model f Φ , we denote the weights of the first FFN layer as K l ∈ R d 1 × d 2 and the 93 second FFN as V l ∈ R d 2 × d 1 . Assume a hidden state h l ∈ R d 1 is an input of the FFN of l -th layer, 94 the above block processes the input as follows: 95

where g act is the activation layer and h l +1 ∈ R d 1 is the input of the next Transformer block. Here, K l 96 and V l emulate neural memories, where keys capture input patterns and values are stored knowledge 97 to be retrieved. When there comes an input vector, it first computes a distribution over the keys, then 98 retrieve the expected knowledge. As the process is just the same for each layer, we can choose any of 99 the layers to edit, we omit l for simplicity in the following description. 100

Our method inserts a key-value adaptor into the existing MLP block. Specifically, we update Φ by 101 inserting a new knowledge neuron to store the edit. Two matrices ˆ K ∈ R d 1 × n and ˆ V ∈ R n × d 1 102 perform as the key-value pair to memorize n edited knowledge, where the knowledge is well-indexed 103 by n dimensions. Therefore, Equation 2 becomes: 104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

where ⊕ denotes concatenation. As we can see, the key-value adaptor appends more information to o , which could overwrite the original output. And original parameter set Φ is extended to Φ ∗ with the new included parameters ˆ K and ˆ V . Therefore, we aim to find a good key-value adaptor for model editing that can collaborate with the original knowledge neurons. Considering the independence of the above two function terms and the potential more flexible combination to the output, we relax the formulation of the adaptor to ADAPTOR( i ; ˆ K , ˆ V ) = α ˆ V ⊺ g act ( ˆ K ⊺ i ) , which may be a more expressive function with a scaling factor α [19]. Next, we will introduce how to find such an optimal adaptor which not only satisfies the edit success but also preserves the original model behavior.

4.1 Initial Key-Value Adaptors for In-Scope Editing

Given an edit ( x i , y i ) ∈ D in , we first initialize its knowledge neuron ˆ k 0 ∈ R d 1 and ˆ v 0 ∈ R d 1 . For ˆ k 0 , we initialize each key to the x i using the cached input i predicted by f Φ ( x i ) at layer l , which results in a high probability of matching to the input pattern. For ˆ v 0 , we initialize it using the weights corresponding to y i from the last layer of f Φ . Specifically, f Φ ( x i ) takes charge of generating the next token which can be deemed as the prediction to x i . Thus, we extract the corresponding column of the ground truth token y i from the weights W ∈ R d 1 ×| V | for generating the next token distribution, where | V | and d 1 are the sizes of the vocabulary and dimension of the last layer, respectively 1 After initialization, we build a mapping from x i to y i in a Transformer.

4.2 Retrofit Adaptors for Model Editing (Training Phase)

To prevent the effect of the inconsistent scaling brought by built-in parameters in Equation 1, we first normalize i to ensure that its mean value is close to 0 before it is fed into the adaptor. Given ( x i , y i ) , we can have the initialized key-value adaptor as follows:

To avoid the inserted key-value adaptor distracts the original knowledge stored in the existing neurons, we propose to use the activation functions that can activate the memory with a large matching value and ignore the memory with a small matching value. When we deploy the adaptor to models, the activation function usually remains consistent with the base model. Furthermore, we apply a hyper-parameter margin θ > 0 , which allows memory to be active if x > θ , otherwise inactivate. For example, we use GeLU [22] for GPT [23] series model and our activation function can be denoted as:

The motivations behind the above design in our activation function are two-fold: First, the activation function works as a neuronal inhibitor to inhibit the activation of new knowledge neurons, which retains the original output in most cases. Second, the involvement of the margin further raises the bar to activate the new knowledge neurons. If a certain input is out of the editing scope, it fails to match any memory, all inserted neurons will be inhibited after the activation function as shown in Figure 1.

In practice, edit input x i is shown in the form of a sequence of tokens such as '{ the , capital , of , China , is }' and y i is the single-token answer ' Beijing '. This indicates that we have a sequence of hidden states { h 1 , h 2 , ..., h s } corresponding to input x i = { w 1 , w 2 , ..., w s } . To avoid damaging the original behavior of the edit model, the edit block merely works on the final token, which is the last token before generation:

̸

where i j is the input corresponding to the j -th hidden state h j in the sequence. As a result, only 142 when the entire input sequence is fed into the model, the new knowledge is activated, which not 143 only prevents the dramatic change to the original model but also benefits the gradient update to the 144 key-value pairs 2 . 145

Fine-tuning adaptors with multiple objectives . While the above initialization effectively builds the 146 mapping from a certain edit input to the edit output, its impact on irrelevant knowledge may lead to 147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

catastrophic forgetting [24] issue, which is caused by the extending key-value pairs of the adaptor.

In other words, we expect ADAPTOR( i ; ˆ K , ˆ V ) could dominate the output for each x i ∈ D in but maintain unchanged prediction for x i ∈ D out and x <i ∈ D in . Inspired by the elastic weight consolidation for neural networks [25], we set optimization goals to retrofit Φ ∗ with the consideration of the following perspectives.

(1) To maximize the prediction of y i from the last layer, we maximize the probability of the ground truth edit output given the edit input:

(2) Even though L edit enables models to fit the mapping from x i to y i effectively, it may push our adaptor far from the initialization, which may damage the initialized key distribution and lead to overfitting. Hence, we propose an additional term to prevent the dramatic change of the update of ˆ k :

(3) Importantly, to prevent the fine-tuning from changing the irrelevant knowledge, we sample some out-of-scope edit data to form D out 3 and retain the original outputs from the model:

Hence, we comprehend each aspect to form the final objective to retrofit the key-value adaptor:

where a , b are hyper-parameters denoting the importance of the different objective aspects. It is worth noting that we edit one knowledge neuron at one time, but we still support sequential editing by iteratively inserting key-value pairs. During training, all parameters except for ˆ k and ˆ v for the current edit are frozen. That is, we freeze the prior edit knowledge neurons and simply update the neuron inserted for current edit. This procedure repeats until we have conducted edit over the entire dataset. Compared with parameter high-efficient tuning methods [19, 26], which injects the new knowledge into a pre-trained LM as a whole, iReVa focuses on editing parameters in a traceable manner. In other words, we can locate the edited knowledge neurons. At the end, we display the training procedure of iReVa in Algorithm 1.

Algorithm 1 Training Procedure of iReVa

4.3 Activate Max-Matching Key in Adaptor (Inference Phase) 170

As we iteratively append ˆ k and ˆ v to the knowledge neurons. The above procedure will sequentially 171 generate mappings from the edit input to the edit output. Eventually, we obtain two concatenated 172 matrices ˆ K ∈ R d 1 × n and ˆ V ∈ V n × d 1 . During inference, we further control the amount of active 173 neurons and highlight the max-matching memory. To this end, we introduce a max-pooling layer to 174 extract the memory with the maximum matching score: 175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

where j = argmax t ( ˆ K ⊺ t i ) and ˆ K t denotes the j -th column of ˆ K . As we can see, when there comes a new input, this layer will highlight the inserted knowledge neurons with the highest similarity score to the input as shown in Figure 1. It is worth noting that we exclude the max-pooling layer during the training procedure because this may impede the back-propagation due to the inactivation of the neurons.

5 Experimental Setup

5.1 Datasets

We perform extensive experiments on two modeling editing tasks: zsRE [8] is a commonly used model editing tasks derived from question-answering benchmark. Totally 19 , 086 examples are included, each example includes a source question, paraphrase question and corresponding answer. We construct another PARAREL [27] dataset. Each sentence in PARAREL is derived from a triplet ( s, r, o ) and the object o was replaced with a ' [MASK] ' token and a paraphrased version is also involved. To apply PARAREL in model editing task, we selected those sentences that end with ' [MASK] ' token to conform to the format of next-token-prediction 4 . For both datasets, we sample irrelevant question-answer pair from NQ to evaluate the preservation to out-of-scope editing. We test 10 K edit in a batch and denote them as zsRE10 K and PARAREL10 K , respectively.

5.2 Baselines

We compare our iReVa with 6 advanced baselines that support batch editing: NO EDITING denotes we do not modify the base model and utilize its original prediction; FT [28] is the simple fine-tuning with a constraint on the key-value adaptor. MEMIT [10] and ROME [9] are two methods employing a casual analysis to detect the most significant hidden states. They view the editing as a minimum optimization and edit the weight directly, which is effective in batch edit; MEND [8] applies rank-one decomposition to divide the model into two rank-one matrices, which is able to carry mass knowledge in the dense metrics; MELO [12] activates specific LoRA block corresponding to specific queries for multiple edits, which support large-scale editing in just one process.

5.3 Evaluation Metrics

We follow the commonly-used evaluation metrics [9, 10] to measure the effect of our editing method.

Edit Success (ES) measures the models' prediction accuracy on edited data x i ∈ D in by calculating ES = 1 N ∑ N i =0 I ( y i = f Φ ( x i )) , which represents whether the new knowledge is successfully injected into the base model.

Generalization (Paraphrase Success, PS) measures the models' prediction accuracy on paraphrase questions provided by benchmarks. We compute paraphrase success with the same formulation but for x i in paraphrase questions set. Paraphrase success indicates whether the model can recognize similar expressions and provide edited answers.

Specificity (Neighborhood Success, NS) measures the models' prediction accuracy on irrelevant questions. Different from D out , these questions are only used for preventing data leakage. We compute neighborhood success with the same formulation but for x i in neighborhood questions set. Neighborhood success manifests the capability of solving catastrophic forgetting and preserving irrelevant knowledge stored in model.

Score is the average of three aforementioned metrics.

5.4 Implementation Details

Regarding editing datasets, we pre-process the edit input-output pairs following existing studies [8]. If the multiple tokens form a single prediction, we decompose the multiple tokens into multiple data pairs by greedily appending the previous token in the edit output at the end of the edit input 5 . For model selection, we conduct the experiments on GPT2-XL (1.5 Billion parameters) [29] due to its wide application on existing model editing studies. We trained iReVa on a single NVIDIA

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

A800 80G GPU. On two evaluated benchmarks, we set a = 1 e -3 , b = 1 e -3 , α = 2 e -1 , and iReVa is applied in 47 -th ( 48 layers totally) layer inspired by the assertion in [15]. For the margin in activation function, we set θ = 0 . 75 for zsRE, θ = 0 . 65 for PARAREL. During training, we conduct experiments on GPT2-XL with setting learning rate as 5 e -2 , batch size as 1 , epoch number as 5 . We set the learning rate as 5 e -3 for GPT-NEO-2.7B . More implementation details of baselines is displayed in Appendix A.7. We re-implement the comparable baselines using the same configuration reported in existing studies.

6 Results and Analyses

6.1 Comparisons to Existing Methods

Table 6.1 exemplifies performances of iReVa and baselines on zsRE and PARAREL with 10 K edits in batch. As we can see, iReVa outperforms all baselines on average scores with noticeable margins. Even without retrofitting, our method is able to outperform the SOTA results by around 9% and 6% average score improvement on zsRE-10K and PARAREL-10K, respectively. Among all the baseline methods, FT achieves good results on ES and PS, this indicates that fine-tuning is simple but effective to inject knowledge but it could easily distract the irrelevant knowledge, resulting in a poor NS. Whereas other baselines can not guarantee the editing success in a batch, resulting in poor ES and PS. In comparison, iReVa achieves impressive results on all the evaluation metrics. It achieves close to 100% ES without detriment to the original NS. We observe a slight improvement from the results of iReVa to iReVa+ L on zsRE-10K dataset, it verifies our rationale deduce for the initialization of key-value pairs. However, the improvement brought by fine-tuning is not maintained on PARAREL-10K, we suspect this is because the involvement of irrelevant knowledge brings in little unexpected noise with possibility.

6.2 Edit Withdrawal Test

Compared with the existing editing methods, our method has the unique advantage of interpretability and traceability, that is we can clearly identify the edit for each newly inserted key-value pair. This provides a chance to conduct an edit withdrawal test. Specifically, we test, after editing on 10 K examples, if iReVa is able to withdraw certain edits and recover the original output from the base model without much loss. To this end, we inhibit corresponding knowledge neurons as withdrawing the edit, which is denoted as f -ˆ k Φ ∗ . For evaluation, we introduce two metrics, namely Retrieve Success and Consistency . They are formulated as RS = 1 N ∑ N i =0 I ( f Φ ∗ ( x i ) = f -ˆ k i Φ ∗ ) and Con = 1 N ∑ N i =0 I ( f Φ ( x i ) = f -ˆ k i Φ ∗ ) , respectively. The evaluation result on zsRE-10K is shown in Table 6.2. The results which are close to 100% proves that iReVa can explicitly manipulate the activation of knowledge neurons and easily withdraw the updated knowledge. It is worth noting that this test is not applicable to any other editing methods as their edited parameters are untraceable. This is the first attempt at conducting more flexible knowledge editing.

̸

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

6.3 Efficiency Analysis

We discuss the spatial and time complexities of iReVa. Regarding time complexity during inference, iReVa only insert the adaptor in a single l -th layer and the insertion only affects the final token prediction of the input. With i ∈ R 1 × d 1 , ˆ K ∈ R d 1 × n , ˆ V ∈ R n × d 1 , the extra time consumption is O ( d 2 1 n ) , which is unrelated to the input length and number of layers. Regarding spacial complexity, as we insert two vectors for each edit in a single layer, the extra spacial consumption is O (2 nd 1 ) . In practice, for GPT2-XL with 1 . 5 B parameters, the adaptor merely possesses 0 . 08 B parameters with 10 K edits. There is no additional spacial complexity involved in the training phase, given that only 2 d 1 parameters are learnable for each edit. We empirically record that 10 K edits with iReVa cost 7 . 5 / 1 . 6 hours (fine-tuning/without fine-tuning) with a single NVIDIA A800 GPU, compared to 9 . 16 hours for ROME and 5 . 4 hours for MEMIT.

6.4 Ablation Study

Table 6.4 shows iReVa's performance on zsRE-10K when we iteratively remove sub-modules: (1) w/o activation function denotes that we remove the activation function proposed in Equation 6. (2) w/o max-pooling denotes that we involve all knowledge neurons during inference instead of the design of Equation 12. (3) w/o L rec denotes that we train iReVa without initialization and set a = 0 in Equation 11. (4) w/o L irr means we do not apply L irr by setting b = 0 in Equation 11. As we can see, all the modules contribute to the good results. In comparison, the activation function is important to preserve the out-of-scope edit. Without activation function, we can attain better results on ES and PS, but NS will decrease sharply. We also find that the influence of max-pooling is significant, which may attribute to noisy data added by a large amount of active but irrelevant knowledge neurons. Besides, excluding L rec will lead to an observable drop on the three metrics because we discord the effective initialization on ˆ K and ˆ V . Finally, disabling L irr may induce a marginal improvement in ES and PS, but at the cost of a reduction in NS.

6.5 Generalization Capabilities of iReVa

Layer generalization . To evaluate the effect of iReVa in various layers, we iteratively apply iReVa and the other two baseline editing methods to different layers of GPT2-XL , which consists of 48 layers in total. Figure 6.5 illustrates the influence of three metrics on different layers with intervals. The tendency shows that the edit in the higher layer results in better editing results. This indicates that LMs' final prediction primarily depends on the information retrieved from higher layers and the knowledge stored in lower layers may be overshadowed. For ROME and MEMIT, apparently, they show distinct generalizations in edit layer. Their ES and PS peak at middle layer like 17 or 22 , which proves that the layer generalization is remarkably relevant to the characteristics of different methods. Even though MEMIT achieves good performance in NS when the edit happens in lower layers, overall iReVa outperforms the baselines regarding the comprehensive evaluation metrics.

LMs generalization . We also test iReVa on different LLMs as base models, Figure 6.5 shows iReVa's generality on different backbones. We apply a larger LM GPT-NEO-2.7B [30] and smaller LM GPT2-LARGE [29] to evaluate the effect of iReVa on LMs with different sizes. Both

100

100

30

25

80

Success

80

NeighborhoodSuccess

EditSuccess

20

60

60

Paraphrases

15

40

40

10

20

iReVa

20

iReVa

iReVa

一

ROME

ROME

5

ROME

MEMIT

MEMIT

MEMIT

0

Q

0

0

10

20

30

40

0

10

20

30

40

0

10

20

30

40

EditLayer

EditLayer

EditLayer

GPT-NEO-2.7B and GPT-LARGE contain two-layer-FFN MLP blocks. IReVa can be deemed as a 295 plug-in module for general LMs, which can be applied to more LMs. From the figure, we observe 296 that iReVa can achieve the best average score on both LMs, which shows its general effect. 297

100

25

80

NeighborhoodSuccess

80

ParaphraseSuccess

20

Edit Success

60

60

15

40

40

10

iReVa

20

iReVa

5

iReVa

20

一

ROME

ROME

ROME

MEMIT

MEMIT

MEMIT

0

100

101

102

103

104

100

101

102

103

104

100

101

102

103

104

Editquantity

Editquantity

Editquantity

Edit quantity generalization . We discuss the influence on iReVa's performance with the variation 298 of edit quantity, we simply increase the number of edits in the batch and evaluate ES, PS and NS. 299 Figure 6.5 shows the tendency of three metrics along with the comparison to baselines ROME and 300 MEMIT. As we can see, iReVa is robust to the number of edit in the batch. It consistently surpasses 301 the other baselines when dealing with the various number of edits. MEMIT performs poorly even 302 with a small number of edits. ROME drops dramatically as the edit number grows. 303

304

7 Conclusions

In this paper, we propose iReVa, a model editing method with traceable knowledge storage, which 305 inserts edit key-value adaptor into the MLP module of a transformer model explicitly. iReVa displays 306 prominent abilities of edit success, generalization and specificity and outperforms baselines with an 307 observable margin. Besides, iReVa first successfully demonstrates its capacity on the knowledge 308 withdrawal. For further research, we will focus on generalize iReVa to more LM architectures. 309

310

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, 311 Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are 312 few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 313

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. 314 Miller, and Sebastian Riedel. Language models as knowledge bases?, 2019. 315

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language 316 models know? Transactions of the Association for Computational Linguistics , 8:423-438, 317 2020. 318

319

320

321

322

323

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models, 2024.

324

325

326

327

328

329

330

331

332

N Jayashri and K Kalaiselvi. Knowledge acquisition-scholarly foundations with knowledge management. International Journal of Advanced Studies of Scientific Research , 3(12), 2018.

Jérôme Seymour Bruner. The process of education. 1960.

Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models, 2021.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale, 2022.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass333 editing memory in a transformer, 2023. 334

Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh 335 Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors, 2023. 336

Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron337 indexed dynamic lora, 2023. 338

Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 339 Transformer-patcher: One mistake worth one neuron. In The Eleventh International Con340 ference on Learning Representations , 2023. 341

Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit 342 Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, 343 and visualizing model beliefs. arXiv preprint arXiv:2111.13654 , 2021. 344

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers 345 are key-value memories, 2021. 346

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons 347 in pretrained transformers, 2022. 348

Jérôme Seymour Bruner. The course of cognitive growth. American Psychologist , 19:1-15, 349 1964. 350

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 351 Can we edit factual knowledge by in-context learning?, 2023. 352

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, 353 Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv 354 preprint arXiv:2106.09685 , 2021. 355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

395

396

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory-based model editing at scale, 2022.

Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 , 2020.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation , pages 109-165. Academic Press, 1989.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open , 2023.

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models, 2021.

Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models, 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06) , volume 2, pages 1735-1742. IEEE, 2006.

A Appendix

A.1 Detailed Description of Initialization of Key-Value Adaptor

Wedescribe how we initialize k and v in detail. Given the input x i = { w 1 , w 2 , ..., w s } , we first obtain the corresponding embeddings for each token, such that x i = { w 1 , w 2 , ..., w s } . After encoded via l Transformer layers, we obtain a sequence of hidden representations as input { h l 1 , h l 2 , ..., h l s } . In the two-layer-FFN MLP block of l -th layer, after self-attention and layer norm, we have the hidden representation of the last token as:

We extract i l +1 s as the initialization of ˆ k 0 . Subsequently, { h l +1 1 , h l +1 2 , ..., h l +1 s } are further processed 392 via the higher layers. In the last layer, we make prediction based on the hidden representation in L -th 393 layer, which can be denoted as: 394

where W ∈ R d 1 ×| V | and each column denotes the representation of a token. We extract the column corresponding to the ground truth edit out token y i , that is ˆ v 0 = W [: ,y i ] .

A.2 Discussion of Back Propagation of Key-Value Adaptor 397

Recall the knowledge neurons of our key-value adaptor are: 398

Given L , the gradients are computed as: 399

where g ′ act is the derivative of the activation function. We have multiple observations of the gradients: 400 First, we would like the newly inserted neuron to be activated initially, namely g act > 0 . Otherwise, 401 the gradients are close to 0 and the neurons are likely to be dead. This is the reason why we initialize 402 the ˆ k and ˆ v with the consideration of having a high matching value of k ⊺ i . Second, when we update 403 ˆ k and ˆ v , they are unrelated to k and v , which makes it possible to isolate the irrelevant knowledge. 404

For the knowledge neurons without our key-value adaptor, we have the propagation: 405

As we can see, excluding the key-value adaptor in the neuron makes the gradients simply derived from k and v , which maintains the original knowledge in the neurons.

A.3 Influence of θ and a

The influence of θ is illustrated in A.3. The figure shows the trade-off between the three metrics smoothly. The primary affected metric is Neighborhood Success , and Edit Success and Paraphrased Success exhibit a slight downward trend. For a , we find that merely Paraphrase Success peaks while a = 1 e -2 , meanwhile Edit Success and Neighborhood Success do not continue to improve with the increase of a .

100

100

90

80

80

Metrics

60

EditSuccess

70

EditSuccess

Met

60

ParaphraseSuccess

40

50

20-

40

30

0

0.0

0.2

0.4

0.6

10-5

10-4

10-3

10-2

10-1

100

Valueofθ

Coefficient of reconstruction loss a

The gradients of i are computed as: 406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

A.4 Sample off-scope examples for iReVa

To enhance iReVa's Specificity, we generate 3 kinds of irrelevant questions q for each ( x, y ) ∈ D in to minimize ˆ K ⊺ i · x out , where x out is the representations of q . These questions are listed as follows: a) Randomly generated questions produced by feeding base model with a bos (begin of sentence) token. b) Questions generated by base model with feeding the subject s of the x provided by the benchmark. c) Questions sampled from other examples in training dataset, whose opinion is similar to contrastive learning [31]. During iReVa training, we generate 2 questions in a), 2 questions in b) and 6 questions in c) for each training example.

A.5 Pre-processing procedure of zsRE

Shown in 2, we split each ( x, y ) pair into multiple ( x ′ , y ′ ) to ensure y ′ is a single-token edit out. This procedure is also applied in the evaluation of zsRE and PARAREL, which measures the ( i +1) -th token of edit-out prediction accuracy given edit-in and i prefixes of edit-out.

Algorithm 2 Pre-processing Procedure of PARAREL 1: Input Raw dataset zsRE D , tokenization function encode ; 2: Init D ′ = [] ; 3: for ( x, y ) ∈ D do 4: Init tokens = encode( y ) ; 5: for i ∈ { 0 , 1 , 2 ... len(tokens) -1 } do 6: D ′ .append (( x +tokens[: i ] , y [ i ])) ; return D ′

A.6 Pre-processing Procedure of PARAREL 427

Algorithm 3 Pre-processing Procedure of PARAREL

;

This section details the pre-process method on close text dataset PARAREL [27]. PARAREL contains 428 34 types of relations r , with an average of 900 question bags b per relation, totaling 27,738 distinct 429 questions q . And for each question bag, around 9 rephrased versions are recorded with a sole answer 430 a . 431

The entire pre-process algorithm is shown in 3. To make PARAREL applicable for next-token432 prediction task, we reserve the sentences that end with special token ' [MASK] '. After a round of 433 filtering, we removed question bags b with only 1 valid sentence that ends with ' [MASK] ' for both 434 Edit Success and Paraphrase Success need to be computed. During this filtering, we collect the 435 subject of question s bag by calculating the longest common sub-array of all q ∈ b tokenized by 436 GPT2Tokenizer [29] simultaneously for specific methods require the subject of a question. The 437

next screening occurs at b whose subject s is an empty string or identical to b [0] . With residual 438 question bags b ′ , we choose b ′ [0] as the source question and a randomly sampled question from b ′ [1 :] 439 as the paraphrase question. 440

441

442

443

444

Empirically, we believe PARAREL is harder than zsRE because the average token length of edit target is shorter, thus model can't give more empirical predictions based on given prefix of the target, which is mentioned in A.5. In other word, the account for first-token prediction may influence the difficulty of datasets noticeably.

445

446

447

448

449

450

451

452

453

454

455

A.7 Implementation Details of Comparable Baselines

A.7.1 Fine Tuning(FT)

We implement fine tuning on two feed-forward networks( mlp.c_fc, mlp.c_proj ) at the layer of 46 with GPT2-XL . Base model is trained for 20 epochs with lr = 1 e -4 , batch size = 32 .

A.7.2 MEND

We do not load the pre-trained MEND [8] weight, but apply MEND directly. Hyper-parameters of MEND keep consistent with the configuration of MEND's open-source code.

A.7.3 ROME, MEMIT

ROME [9] and MEMIT [10]'s setups on GPT2-XL also remain identical to the source code. On GPT-NEO-2.7B , we alter the edit layer to 5 for ROME and {3,4,5,6,7,8} for MEMIT.

A.7.4 MELO

Due to larger edit amount and different backbone for zsRE, we modify several configurations to 456 make MELO [12] comparable to our methods. For MELO's code book, we enlarge the number of 457 blocks (clusters) to 100. Besides, we rewrite MELO's training loss to make it compatible with causal 458 decoder. 459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [TODO]

Justification: [TODO]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [TODO]

Justification: [TODO]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper poses no such risks.

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer:

[TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer:

[TODO]

764

Justification:

[TODO]

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [TODO]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the tracable sequential model editing challenge by plugging in additional model components to a transformer MLP blocks. The proposed approach adds additional model components for each edit, allowing for traceability for each edit.\n\nSTRENGTHS:\n•\tThe results indicate that it is a strong approach compared to relevant literature and its performance is relatively stable when scaling it to thousands of edits.\n•\tThe approach allows for \"separability of each edit\" which in turn allows for additional operations such as edit updation or deletion as showcased in the Edit withdrawal experiment. \n•\tThe edit withdrawal experiment is both unique and intriguing, as the concept of removing edits appears to be a novel area of exploration.\n\nWEAKNESSES:\n•\tThe overall approach does not appear to be novel, as it closely resembles T-Patcher. Both iReVa and T-Patcher rely on inserting neurons for editing and using neuron activations to control when to run the patch/adopter. Furthermore, analysis of the editing approach across different layers reveals the same pattern as discussed in the T-Patcher paper which involves adding additional model components in the final layer for optimal results.\n\n•\tExperiments with T-Patcher are missing from the comparisons to the existing methods section. Given its similarity, T-Patcher should be included for comparison.\n\n•\tAlthough T-Patcher performs editing in batches, it still uses a single patch of neurons for each edit, making its editing similarly traceable. Thus, the paper's claim of a \"stronger capacity for carrying traceable edits\" seems unfounded.\n\n•\tThe Edit Withdrawal Test section is hard to understand. How exactly was the experiment conducted? Were all edits removed or only a limited set? Detailed experimentations for this section are needed as it is the only use case of traceability explored in the paper.\n\n•\tEditing techniques that rely on code books with playback vectors e.g. GRACE would allow for edits to be removed. The authors should make it clear that the withdrawal test is not possible for the editing techniques that they have chosen for comparison.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces iReVa, a novel method for model editing that explicitly initializes and retrofits key-value pairs into MLP blocks of transformer models to perform CRUD (Create, Read, Update, Delete) operations on LMs. iReVa aims to update knowledge in LMs without damaging irrelevant knowledge, offering better interpretability and traceability compared to existing methods. The method is validated through experiments on GPT series models, showing significant improvements in edit success and generalization without affecting specificity.\n\nSTRENGTHS:\nProvision of the first attempt at conducting knowledge withdrawal tests for model editing methods.\n\nThe paper includes a comprehensive analysis of iReVa's performance, including knowledge withdrawal tests and generalization tests.\n\niReVa's approach to model editing is innovative, focusing on retrofitting key-value adaptors into MLP blocks for traceable model editing\n\nWEAKNESSES:\nThis paper could benefit from a more detailed comparison with other model editing methods, especially those focusing on lifelong learning and continual editing [1][2].\n\nIt does not discuss the computational efficiency of iReVa in terms of inference time or memory, which is crucial for real-world applications.\n\nThe reliance on the hypothesis that factual knowledge is stored in MLP blocks may be limiting [3], and the authors could explore the broader implications of this assumption.\n\nThe method's applicability to other types of tasks, such as erasing hallucinations, is not validated.\n\nThere is a noticeable absence of experimental validation on other recent and updated models such as GPT-J (used by ROME etc.), LLaMA.\n\nThe technical novelty of iReVa is somewhat limited, as it builds upon existing concepts like MEMIT [4] and key-value memory structures in MLPs [2].\n\nThe absence of a strategy for selecting the adaptor layer may hinder the method's rapid migration and application to various language models。\n\nEquation 3 requires clarification, why 'i' and 'o' in Equation 3 are both passed through SELF_ATTEN again?\n\nReferences\n\n[1] Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors, Hartvigsen et al,\nNeurips 2023.\n\n[2] Transformer-Patcher: One Mistake worth One Neuron, Huang et al, ICLR 2023.\n\n[3] What does the Knowledge Neuron Thesis Have to do with Knowledge? Niu et al, ICLR 2024\n\n[4] Mass-Editing Memory in a Transformer, Meng et al, ICLR 2023.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper introduces a novel method called iReVa for knowledge editing. iReVa initializes and retrofits key-value pairs into MLP blocks to create a new mapping of knowledge without affecting related information. Compared to existing methods, iReVa offers better interpretability and a stronger ability to make traceable edits.

STRENGTHS:
1. The proposed methods demonstrate great performance compared to other baselines under the batch editing scenarios.

WEAKNESSES:
1. The color in the figure is not obvious to discriminate between the original knowledge neurons and new knowledge neurons.
2. The computation of the proposed method is similar to T-Patcher, I'm curious about the difference between them. The proposed methods are designed to tackle the batch edit, but it seems it still needs to add one neuron for each example.

QUESTIONS:
1. Line 57-59 focus on self-attention, while the author selects MLP to further investigate, the logic here is a bit confusing. Is there something I missing?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on model editing at a low cost. Evidence suggests that modules carrying knowledge in a Transformer module are primarily the MLP blocks. Therefore, the authors propose a method, namely iReVa, to initialize and retrofit key-value pairs into MLP blocks in a Transformer for explicitly inserting new knowledge. Specifically, they insert new neurons in the MLP blocks for each piece of knowledge. Each neuron is initialized with the embedded key and value derived from the input-output pair, respectively. To prevent dramatic change to the irrelevant knowledge, iReVa further retrofits the key and value by fine-tuning with multiple objectives. Compared to the existing methods such as MEND, ROME, MEMIT, and MELO, iReVa reveals better interpretability and stronger capacity for carrying traceable edits. The experiments on zsRE-10K and PARAREL-10K datasets reveal that iReVa has superior performance regarding edit success, generalization, and specificity. Further edit withdrawal test indicates that iReVa can explicitly manipulate the activation of neurons and easily withdraw the edits.\n\nSTRENGTHS:\n1.\tThis paper focuses on modeling editing, which has significant applications in the era of LLMs. It can be applied to alleviate the hallucination issue of LMs and resolve the out-of-date as well as missing knowledge in an LM.\n2.\tThis paper introduces a novel editing method with key-value adaptors for traceable model editing. The proposed method makes sense to me. The initialization with embedded key and value derived from the input-output pair can easily make precise edits to the model. Further retrofitting refines the adaptors to satisfy the task.\n3.\tFor experiments, the author has comprehensively shown the superiority of their method in the perspectives of edit success, generalization, and specificity. And more analyses reveal the generalization of iReVa. Particularly, the edit withdrawal test in Section 6.2 is well-designed, which shows the effect of traceable edits and could provide a potential solution for dynamic knowledge maintenance for LMs.\n4.\tOverall, this paper is well-written and easy to follow.\n\nWEAKNESSES:\n1.\tThe discussions on the limitations and broader societal impacts of iReVa are not included in the paper. I have some questions about the application scope of the proposed method. Please see the questions below.\n\nQuestions\n1.\tCould iReVa lead to a dramatically increasing number of parameters? Let’s see if there are millions of knowledge for editing, how can you potentially insert all the knowledge into LMs with iReVa? \n2.\tAfter you change a piece of knowledge, can the reasoning still be conducted for the edited knowledge? For example, if we have edited the president of America, could some reasoning questions like ``Who is the wife of the president of America” also be resolved with the new knowledge?\n3.\tTypo: ``evident’’ in line 6 should be ``evidence’’. Please check.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nNo, the author should discuss the limitations of the proposed method such as the application scope, the potential risks, and future improvement to indicate how robust the results are to violate the\nassumption. I would like the author to add such information during the rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}