Initializing and Retrofitting Key-Value Adaptors for Traceable Model Editing

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:59:08

Abstract: As the insight of knowledge storage in language models deepens, the ability to perform CRUD (Create, Read, Update, Delete) operations on language models becomes increasingly indispensable for satisfying the demands of managing rapidly updating knowledge. Considering the high cost of fine-tuning language models, model editing methods with low cost are usually required to manipulate models' knowledge. Evident suggests that modules carrying knowledge in a Transformer module are primarily the MLP blocks, thus we propose \textbf{iReVa}, a method that explicitly initializes and retrofits key-value pairs into MLP blocks to construct a new mapping of a piece of knowledge without damaging the irrelevant knowledge. In comparison to existing methods, iReVa reveals better interpretability and stronger capacity for carrying traceable edits. Experiment results on series of GPT series models show our prominent performance on edit success and generalization without influencing specificity. We also perform the first attempt at conducting knowledge withdrawal test of iReVa. Our codes are available at github.com/thartvigsen/grace.

Main paper: 
 introduction: 
 Language Models (LMs) [1] are becoming imperative tools for consulting in real-world scenarios.One significant reason for the prevalence of LMs is their ability to answer factoid questions. For example, when we ask an LM with the question "Who is president of America ?", it returns the answer "Joe Biden". Even though a mass amount of knowledge is stored in the LMs, we still face the issue of out-of-date and missing knowledge [2, 3] . Alternatively, some knowledge may change over years and some domain-specific knowledge may be absent from the LMs.To bridge the gap, the task of model editing is introduced to edit the knowledge in LMs, which targets at conducting change to the parameters of LMs and inject certain knowledge to them [4] . The difficulty of this task lies in the manipulation to the LMs, where the knowledge is implicitly stored in dense vectors. A naive solution to model editing is fine-tuning a LM with the new knowledge, whereas the cost is climbing with the surging size of LMs. More recent studies propose to directly update the models' weights in mastery phase [5, 6] via either teaching a hyper-network to learn the change of the weights or locating-then-editing knowledge neurons [7, 8, 9, 10] . While the editing methods above are efficient in updating knowledge in LMs, they encounter the difficulties of differentiating the existing and new knowledge, which makes the editing hard to control. Methods like life-long model editing [11] , MELO [12] , and T-Patcher [13] propose to learn the representation for new knowledge and merge this information with the original models.However, these methods still conform to the paradigm of learning the batch edit [13, 14] as a whole without modeling edit parameters in a traceable way, which can not conform the edit success to each edit and have a lack interpretability to the editing. In contrast, we propose a method of Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Initializing and Retrofitting KEy-Value Adaptors (iReVa), an editing method that inserts a key-value adaptor to indicate the mapping of an edit data pair and further retrofit the adaptor with multiple objectives. Moreover, to prevent the unnecessary change to the irrelevant knowledge, we elaborately design activation mechanism for the knowledge neurons. Experimental results on series of GPT-like models show that iReVa is able to outperform the SOTA results by around 9% and 6% average score improvement on zsRE-10K and PARAREL-10K, respectively. Moreover, iReVa is able to perform knowledge withdrawal in almost perfect condition.Our contributions are summarized as follows: 1) We introduce a novel editing method which initializes and retrofits a key-value adaptor for traceable model editing, which is compatible to most LMs. 2)Our method outperforms recent baselines on model editing tasks with noticeable margins based on various evaluation metrics. 3) We validate the interpretability and generalization capabilities of our method by conducting further analysis such as knowledge withdrawal test and generalization test.2 Related Work 
insight of knowledge storage in language models: 
 As pre-trained LMs show strong abilities to answer factoid questions. Discussion about how LMs store knowledge has emerged. [2] introduced the perspective of treating LMs as knowledge bases and proved its plausibility, which attracted the subsequent attention towards the exploration on the form of knowledge incorporated by LMs. The opinion pointed out by [15] indicates that factual knowledge is stored in two-layer-FFN network of a Transformer due to the similar form as keyvalue memories. This opinion was followed by [16] , which further derives the coefficient between final prediction and knowledge neurons in MLP blocks. In contrast, [9] , through a casual-tracing experiment, posed viewpoints that knowledge is stored in self-attention module. [7] further validates that the weight update is concentrated on parameters in self-attention module when we train models with new knowledge. Our editing method is built upon the former hypothesis and we focus on the editing to the MLP blocks.With the frequent update of the knowledge, the demand of model editing increases. Diverse studies have been proposed. By analogy with human knowledge acquisition, we can categorize the editing into three distinct phases. In recognition phase [17] , methods such as ERAC and IKE [8, 18] solved the problem by importing additional memories in the form of a relevant contexts or prompts. In association phase [6] , parameter-efficient tuning [19, 20, 12, 11] inserts low-rank adaptors or prefix token embeddings to fine-tune new knowledge and combine them to the original models. There are also some studies directly changing the weights of Transformers in mastery phase [5] . For example, [7] proposed KE and [8] proposed MEND to predict the updated parameters of a model with a trained hyper-network. Furthermore, ROME [9] and MEMIT [10] compute the weight update explicitly with proper representations of knowledge queries and values. However, none of them focuses on traceable model editing, which allows more flexible manipulation of the knowledge. 
problem formulation: 
 We follow the previous studies [21, 12, 11] to formulate the task. Suppose we are given a base model that could be a pre-trained language model f Φ parameterized by Φ, model editing aims at editing f Φ with a dataset D in = {(x 1 , y 1 ), ..., (x i , y i )..., (x n , y n )}, where (x i , y i ) denotes the edit input-output pairs. Initially, for x i ∈ D in , the base model makes prediction ŷi = f (x i ) but ŷi ̸ = y i . In this case, we change f Φ by editing its parameters to Φ * . A good model editing to f Φ * should satisfy: 1) for any x i ∈ D in , the edited model f Φ * should output desired predictions, that is f Φ * (x i ) = y i ; 2) for any input out of the scope of D in , which is denoted as D out , the edited model f Φ * should retain the original predictions, that is f Φ * (x i ) = f Φ (x i ); 3) the edit of (x i , y i ) towards f Φ * should not influence any prior edits x <i ∈ D in .𝑥 ! : The Divine Comedy is written by ? … 𝑥 " : Who is the president of America ?Value interpretability: We interpret the inference phase by giving some explicit examples (Please note we omit some neurons during inference due to the space limit.). When the query falls in the in-scope edit, our key-value adaptor will be activated and retrieve the corresponding knowledge. When the query falls in the out-of-scope edit, our key-value adaptor is inactive and the model retrieves knowledge from the original memory.EQUATION 
method: 
 To develop an editing method that supports traceable edits to knowledge neurons, we introduce a novel method "iReVa" that initializes and Retrofits kEy-Value Adaptors for traceable model editing. The pre-trained LM f Φ usually contains Transformer blocks, which consist of intertwined self-attention and feed-forward layers. The prior studies [15] have shown that the inside MLP blocks are commonly deemed as the neurons for storing implicit knowledge. Our method is able to insert new knowledge but without damaging the irrelevant knowledge in the models by inserting and retrofitting the key-value adaptors to these blocks.Figure 3 depicts the architecture of our proposed method. For a two-layer-FFN MLP block in the l-th layer of the original model f Φ , we denote the weights of the first FFN layer as K l ∈ R d1×d2 and the second FFN as V l ∈ R d2×d1 . Assume a hidden state h l ∈ R d1 is an input of the FFN of l-th layer, the above block processes the input as follows:EQUATIONEQUATIONEQUATIONwhere g act is the activation layer and h l+1 ∈ R d1 is the input of the next Transformer block. Here, K l and V l emulate neural memories, where keys capture input patterns and values are stored knowledge to be retrieved. When there comes an input vector, it first computes a distribution over the keys, then retrieve the expected knowledge. As the process is just the same for each layer, we can choose any of the layers to edit, we omit l for simplicity in the following description.Our method inserts a key-value adaptor into the existing MLP block. Specifically, we update Φ by inserting a new knowledge neuron to store the edit. Two matrices K ∈ R d1×n and V ∈ R n×d1 perform as the key-value pair to memorize n edited knowledge, where the knowledge is well-indexed by n dimensions. Therefore, Equation 2 becomes:EQUATIONwhere ⊕ denotes concatenation. As we can see, the key-value adaptor appends more information to o, which could overwrite the original output. And original parameter set Φ is extended to Φ * with the new included parameters K and V. Therefore, we aim to find a good key-value adaptor for model editing that can collaborate with the original knowledge neurons. Considering the independence of the above two function terms and the potential more flexible combination to the output, we relax the formulation of the adaptor to ADAPTOR(i; K, V) = α V⊺ g act ( K⊺ i), which may be a more expressive function with a scaling factor α [19] . Next, we will introduce how to find such an optimal adaptor which not only satisfies the edit success but also preserves the original model behavior.Given an edit (x i , y i ) ∈ D in , we first initialize its knowledge neuron k0 ∈ R d1 and v0 ∈ R d1 . For k0 , we initialize each key to the x i using the cached input i predicted by f Φ (x i ) at layer l, which results in a high probability of matching to the input pattern. For v0 , we initialize it using the weights corresponding to y i from the last layer of f Φ . Specifically, f Φ (x i ) takes charge of generating the next token which can be deemed as the prediction to x i . Thus, we extract the corresponding column of the ground truth token y i from the weights W ∈ R d1×|V | for generating the next token distribution,where |V | and d 1 are the sizes of the vocabulary and dimension of the last layer, respectively 1 After initialization, we build a mapping from x i to y i in a Transformer.To prevent the effect of the inconsistent scaling brought by built-in parameters in Equation 1, we first normalize i to ensure that its mean value is close to 0 before it is fed into the adaptor. Given (x i , y i ),we can have the initialized key-value adaptor as follows:ADAPTOR(i; K, V) = α(v 0 ) ⊺ g act (( k0 ) ⊺ i).To avoid the inserted key-value adaptor distracts the original knowledge stored in the existing neurons, we propose to use the activation functions that can activate the memory with a large matching value and ignore the memory with a small matching value. When we deploy the adaptor to models, the activation function usually remains consistent with the base model. Furthermore, we apply a hyper-parameter margin θ > 0, which allows memory to be active if x > θ, otherwise inactivate. For example, we use GeLU [22] for GPT [23] series model and our activation function can be denoted as:EQUATIONThe motivations behind the above design in our activation function are two-fold: First, the activation function works as a neuronal inhibitor to inhibit the activation of new knowledge neurons, which retains the original output in most cases. Second, the involvement of the margin further raises the bar to activate the new knowledge neurons. If a certain input is out of the editing scope, it fails to match any memory, all inserted neurons will be inhibited after the activation function as shown in Figure 1 .In practice, edit input x i is shown in the form of a sequence of tokens such as "{the, capital, of, China, is}" and y i is the single-token answer "Beijing". This indicates that we have a sequence of hidden states {h 1 , h 2 , ..., h s } corresponding to input x i = {w 1 , w 2 , ..., w s }. To avoid damaging the original behavior of the edit model, the edit block merely works on the final token, which is the last token before generation:EQUATION)where i j is the input corresponding to the j-th hidden state h j in the sequence. As a result, only when the entire input sequence is fed into the model, the new knowledge is activated, which not only prevents the dramatic change to the original model but also benefits the gradient update to the key-value pairs2 .Fine-tuning adaptors with multiple objectives. While the above initialization effectively builds the mapping from a certain edit input to the edit output, its impact on irrelevant knowledge may lead to catastrophic forgetting [24] issue, which is caused by the extending key-value pairs of the adaptor.In other words, we expect ADAPTOR(i; K, V) could dominate the output for each x i ∈ D in but maintain unchanged prediction for x i ∈ D out and x <i ∈ D in . Inspired by the elastic weight consolidation for neural networks [25] , we set optimization goals to retrofit Φ * with the consideration of the following perspectives.(1) To maximize the prediction of y i from the last layer, we maximize the probability of the ground truth edit output given the edit input:EQUATION(2) Even though L edit enables models to fit the mapping from x i to y i effectively, it may push our adaptor far from the initialization, which may damage the initialized key distribution and lead to overfitting. Hence, we propose an additional term to prevent the dramatic change of the update of k:EQUATION(3) Importantly, to prevent the fine-tuning from changing the irrelevant knowledge, we sample some out-of-scope edit data to form D out3 and retain the original outputs from the model:EQUATIONHence, we comprehend each aspect to form the final objective to retrofit the key-value adaptor:EQUATIONwhere a, b are hyper-parameters denoting the importance of the different objective aspects. It is worth noting that we edit one knowledge neuron at one time, but we still support sequential editing by iteratively inserting key-value pairs. During training, all parameters except for k and v for the current edit are frozen. That is, we freeze the prior edit knowledge neurons and simply update the neuron inserted for current edit. This procedure repeats until we have conducted edit over the entire dataset.Compared with parameter high-efficient tuning methods [19, 26] , which injects the new knowledge into a pre-trained LM as a whole, iReVa focuses on editing parameters in a traceable manner. In other words, we can locate the edited knowledge neurons. At the end, we display the training procedure of iReVa in Algorithm 1.Algorithm 1 Training Procedure of iReVa 1: Input In-scope editing pairs D in ; out-of-scope editing pairsD out ; Original model f Φ ; Iteration number T 2: Initial Φ * ← Φ 3: for (x i , y i ) ∈ D in do 4: Initial k ← i; v ← W [yi,:]▷ Initialize key-value adaptor as shown in Section 4.1 5:Φ * ← Φ * k v 6:for t = {1, 2, .., T } do k ← Adam( k, ∇ L k) 9: v ← Adam(v, ∇ L v) return f Φ *As we iteratively append k and v to the knowledge neurons. The above procedure will sequentially generate mappings from the edit input to the edit output. Eventually, we obtain two concatenated matrices K ∈ R d1×n and V ∈ V n×d1 . During inference, we further control the amount of active neurons and highlight the max-matching memory. To this end, we introduce a max-pooling layer to extract the memory with the maximum matching score:EQUATIONwhere j = argmax t ( K⊺ t i) and Kt denotes the j-th column of K. As we can see, when there comes a new input, this layer will highlight the inserted knowledge neurons with the highest similarity score to the input as shown in Figure 1 . It is worth noting that we exclude the max-pooling layer during the training procedure because this may impede the back-propagation due to the inactivation of the neurons.5 Experimental Setup 
datasets: 
 We perform extensive experiments on two modeling editing tasks: zsRE [8] is a commonly used model editing tasks derived from question-answering benchmark. Totally 19, 086 examples are included, each example includes a source question, paraphrase question and corresponding answer.We construct another PARAREL [27] dataset. Each sentence in PARAREL is derived from a triplet (s, r, o) and the object o was replaced with a "[MASK]" token and a paraphrased version is also involved. To apply PARAREL in model editing task, we selected those sentences that end with "[MASK]" token to conform to the format of next-token-prediction 4 . For both datasets, we sampleirrelevant question-answer pair from NQ to evaluate the preservation to out-of-scope editing. We test 10K edit in a batch and denote them as zsRE-10K and PARAREL-10K, respectively.We compare our iReVa with 6 advanced baselines that support batch editing: NO EDITING denotes we do not modify the base model and utilize its original prediction; FT [28] is the simple fine-tuning with a constraint on the key-value adaptor. MEMIT [10] and ROME [9] are two methods employing a casual analysis to detect the most significant hidden states. They view the editing as a minimum optimization and edit the weight directly, which is effective in batch edit; MEND [8] applies rank-one decomposition to divide the model into two rank-one matrices, which is able to carry mass knowledge in the dense metrics; MELO [12] activates specific LoRA block corresponding to specific queries for multiple edits, which support large-scale editing in just one process.We follow the commonly-used evaluation metrics [9, 10] to measure the effect of our editing method.1. Edit Success (ES) measures the models' prediction accuracy on edited datax i ∈ D in by calculat- ing ES = 1 N N i=0 I(y i = f Φ (x i )), which represents whether the new knowledge is successfully injected into the base model.2. Generalization (Paraphrase Success, PS) measures the models' prediction accuracy on paraphrase questions provided by benchmarks. We compute paraphrase success with the same formulation but for x i in paraphrase questions set. Paraphrase success indicates whether the model can recognize similar expressions and provide edited answers.3. Specificity (Neighborhood Success, NS) measures the models' prediction accuracy on irrelevant questions. Different from D out , these questions are only used for preventing data leakage. We compute neighborhood success with the same formulation but for x i in neighborhood questions set.Neighborhood success manifests the capability of solving catastrophic forgetting and preserving irrelevant knowledge stored in model.Score is the average of three aforementioned metrics.Regarding editing datasets, we pre-process the edit input-output pairs following existing studies [8] .If the multiple tokens form a single prediction, we decompose the multiple tokens into multiple data pairs by greedily appending the previous token in the edit output at the end of the edit input 5 .For model selection, we conduct the experiments on GPT2-XL (1.5 Billion parameters) [29] due to its wide application on existing model editing studies. We trained iReVa on a single NVIDIA A800 80G GPU. On two evaluated benchmarks, we set a = 1e -3, b = 1e -3, α = 2e -1, and iReVa is applied in 47-th (48 layers totally) layer inspired by the assertion in [15] . For the margin in activation function, we set θ = 0.75 for zsRE, θ = 0.65 for PARAREL. During training, we conduct experiments on GPT2-XL with setting learning rate as 5e -2, batch size as 1, epoch number as 5.We set the learning rate as 5e -3 for GPT-NEO-2.7B. More implementation details of baselines is displayed in Appendix A.7. We re-implement the comparable baselines using the same configuration reported in existing studies.6 Results and Analyses and PS, but NS will decrease sharply. We also find that the influence of max-pooling is significant, which may attribute to noisy data added by a large amount of active but irrelevant knowledge neurons.= 1 N N i=0 I(f Φ * (x i ) ̸ = f -ki Φ * ) and Con = 1 N N i=0 I(f Φ (x i ) = f -ki Φ * ),Besides, excluding L rec will lead to an observable drop on the three metrics because we discord the effective initialization on K and V. Finally, disabling L irr may induce a marginal improvement in ES and PS, but at the cost of a reduction in NS.  
generalization capabilities of ireva: 
 Layer generalization. To evaluate the effect of iReVa in various layers, we iteratively apply iReVa and the other two baseline editing methods to different layers of GPT2-XL, which consists of 48 layers in total. Figure 6 .5 illustrates the influence of three metrics on different layers with intervals.The tendency shows that the edit in the higher layer results in better editing results. This indicates that LMs' final prediction primarily depends on the information retrieved from higher layers and the knowledge stored in lower layers may be overshadowed. For ROME and MEMIT, apparently, they show distinct generalizations in edit layer. Their ES and PS peak at middle layer like 17 or 22, which proves that the layer generalization is remarkably relevant to the characteristics of different methods. Even though MEMIT achieves good performance in NS when the edit happens in lower layers, overall iReVa outperforms the baselines regarding the comprehensive evaluation metrics.LMs generalization. We also test iReVa on different LLMs as base models, Figure 6 .5 shows iReVa's generality on different backbones. We apply a larger LM GPT-NEO-2.7B [30] and smaller LM GPT2-LARGE [29] to evaluate the effect of iReVa on LMs with different sizes. Both GPT-NEO-2.7B and GPT-LARGE contain two-layer-FFN MLP blocks. IReVa can be deemed as a plug-in module for general LMs, which can be applied to more LMs. From the figure, we observe that iReVa can achieve the best average score on both LMs, which shows its general effect. Edit quantity generalization. We discuss the influence on iReVa's performance with the variation of edit quantity, we simply increase the number of edits in the batch and evaluate ES, PS and NS. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [TODO]Justification: [TODO] Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusions: 
 In this paper, we propose iReVa, a model editing method with traceable knowledge storage, which inserts edit key-value adaptor into the MLP module of a transformer model explicitly. iReVa displays prominent abilities of edit success, generalization and specificity and outperforms baselines with an observable margin. Besides, iReVa first successfully demonstrates its capacity on the knowledge withdrawal. For further research, we will focus on generalize iReVa to more LM architectures.recognition (CVPR'06), volume 2, pages 1735-1742. IEEE, 2006. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: We describe how we initialize k and v in detail. Given the input x i = {w 1 , w 2 , ..., w s }, we first obtain the corresponding embeddings for each token, such that x i = {w 1 , w 2 , ..., w s }. After encoded via l Transformer layers, we obtain a sequence of hidden representations as input {h l 1 , h l 2 , ..., h l s }. In the two-layer-FFN MLP block of l-th layer, after self-attention and layer norm, we have the hidden representation of the last token as:i l s = LAYER_NORM(h l s + SELF_ATTN(h l s )) o l s = V l⊺ g act (K l⊺ i l s ) h l+1 s = SELF_ATTN(i l s + o l s )We extract i l+1 s as the initialization of k0 . Subsequently, {h l+1 1 , h l+1 2 , ..., h l+1 s } are further processed via the higher layers. In the last layer, we make prediction based on the hidden representation in L-th layer, which can be denoted as:P fΦ (y i |x i ) = SOFTMAX(W ⊺ h L s ), where W ∈ R d1×|V | and each column denotes the representation of a token. We extract the column corresponding to the ground truth edit out token y i , that is v0 = W [:,yi] .Recall the knowledge neurons of our key-value adaptor are:o = v ⊺ g act (k ⊺ i) + v⊺ g act ( k⊺ i)Given L, the gradients are computed as:dL d k = g ′ act ( k⊺ i) • v • î⊺ dL do dL dv = g act ( k⊺ i) dL do dL di = [g ′ act (k ⊺ i)v ⊺ k + g ′ act ( k⊺ i)v ⊺ k] dL do .where g ′ act is the derivative of the activation function. We have multiple observations of the gradients:First, we would like the newly inserted neuron to be activated initially, namely g act > 0. Otherwise, the gradients are close to 0 and the neurons are likely to be dead. This is the reason why we initialize the k and v with the consideration of having a high matching value of k ⊺ i. Second, when we update k and v, they are unrelated to k and v, which makes it possible to isolate the irrelevant knowledge.For the knowledge neurons without our key-value adaptor, we have the propagation:o = v ⊺ g act (k ⊺ i).The gradients of i are computed as:dL di = g ′ act (k ⊺ i)v ⊺ k dL do .As we can see, excluding the key-value adaptor in the neuron makes the gradients simply derived from k and v, which maintains the original knowledge in the neurons.The influence of θ is illustrated in A. To enhance iReVa's Specificity, we generate 3 kinds of irrelevant questions q for each (x, y) ∈ D in to minimize K⊺ i • x out , where x out is the representations of q. These questions are listed as follows:a) Randomly generated questions produced by feeding base model with a bos (begin of sentence) token. b) Questions generated by base model with feeding the subject s of the x provided by the benchmark. c) Questions sampled from other examples in training dataset, whose opinion is similar to contrastive learning [31] . During iReVa training, we generate 2 questions in a), 2 questions in b) and 6 questions in c) for each training example.A.5 Pre-processing procedure of zsRE Shown in 2, we split each (x, y) pair into multiple (x ′ , y ′ ) to ensure y ′ is a single-token edit out. This procedure is also applied in the evaluation of zsRE and PARAREL, which measures the (i + 1)-th token of edit-out prediction accuracy given edit-in and i prefixes of edit-out.Algorithm 2 Pre-processing Procedure of PARAREL Init tokens = encode(y); for (b ij , a ij ) ∈ v i do ▷ For specific questions, rephrased versions and answers in v iIf len(b ij ) ≤ 1, then continue;6:Init subject = b ij [0];Init compatible_questions = [];8:for q ijk ∈ b ij [1 :] do 9:subject = lcs(encode(q ijk ), encode(subject));10:If q ijk .endswith("[M ASK]"), then compatible_questions.append(q ijk ); If (subject = "") ∨ (subject = src_question), then continue This section details the pre-process method on close text dataset PARAREL [27] . PARAREL contains 34 types of relations r, with an average of 900 question bags b per relation, totaling 27,738 distinct questions q. And for each question bag, around 9 rephrased versions are recorded with a sole answer a.The entire pre-process algorithm is shown in 3. To make PARAREL applicable for next-tokenprediction task, we reserve the sentences that end with special token "[MASK]". After a round of filtering, we removed question bags b with only 1 valid sentence that ends with "[MASK]" for both Edit Success and Paraphrase Success need to be computed. During this filtering, we collect the subject of question s bag by calculating the longest common sub-array of all q ∈ b tokenized by GPT2Tokenizer [29] simultaneously for specific methods require the subject of a question. The Empirically, we believe PARAREL is harder than zsRE because the average token length of edit target is shorter, thus model can't give more empirical predictions based on given prefix of the target, which is mentioned in A.5. In other word, the account for first-token prediction may influence the difficulty of datasets noticeably.We implement fine tuning on two feed-forward networks(mlp.c_fc, mlp.c_proj) at the layer of 46 with GPT2-XL. Base model is trained for 20 epochs with lr = 1e -4, batch size = 32.We do not load the pre-trained MEND [8] weight, but apply MEND directly. Hyper-parameters of MEND keep consistent with the configuration of MEND's open-source code.ROME [9] and MEMIT [10] 's setups on GPT2-XL also remain identical to the source code. On GPT-NEO-2.7B, we alter the edit layer to 5 for ROME and {3,4,5,6,7,8} for MEMIT.Due to larger edit amount and different backbone for zsRE, we modify several configurations to make MELO [12] comparable to our methods. For MELO's code book, we enlarge the number of blocks (clusters) to 100. Besides, we rewrite MELO's training loss to make it compatible with causal decoder.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: [TODO] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.See Appendix A.1 for detailed description of initialization of k0 and v0 .See the discussion of gradient back-propagation of k and v in Appendix A.2.Here, Dout is generated randomly. See Appendix A.4 for details.Appendix A.6 demonstrates the pre-processing step to PARAREL in detail.The processing procedure is displayed in Appendix A.5 


Decision: The paper presents a model editing method that explicitly targets MLP blocks by retrofits key-value pairs of new piece of knowledge. One of the major limitations of the submitted work is the lack of comparison with obvious baselines that are very similar to the method proposed in the paper. From the rebuttal, it is clear that authors were familiar with those work and they did a good job in explaining the differences. The discussion with the related work on whether an approach can be applied to a certain class of model or not should be done in the paper. Authors should clearly articulate the contribution of their work with respect to other methods in the paper. This is inline with the novelty concern raised by the reviewers and I did not see a response to that weakness.

Since the authors mention that the closest approach to their method, T-Patcher, can not be applied to decoder only model given the current implementation, does iReVa can be applied to encoder only models? If yes, it is natural to think of a comparison given that both approaches are very similar and it is unclear whether the various design decisions taken by iReVa as mentioned in the Global Rebuttal are the reason of better editing or not. If iReVa can not be applied to encoder models, this should be clearly stated. 

The paper also lacks a limitation section which is not compulsory but it is important to clearly state the limitations of the work. The checklist provided by the authors is incomplete and did not provide response to any question which kills the sole purpose of the requirement of submitting the checklist.

Reviews: review: paper_summary: paper_summary: This paper focuses on model editing at a low cost. Evidence suggests that modules carrying knowledge in a Transformer module are primarily the MLP blocks. Therefore, the authors propose a method, namely iReVa, to initialize and retrofit key-value pairs into MLP blocks in a Transformer for explicitly inserting new knowledge. Specifically, they insert new neurons in the MLP blocks for each piece of knowledge. Each neuron is initialized with the embedded key and value derived from the input-output pair, respectively. To prevent dramatic change to the irrelevant knowledge, iReVa further retrofits the key and value by fine-tuning with multiple objectives. Compared to the existing methods such as MEND, ROME, MEMIT, and MELO, iReVa reveals better interpretability and stronger capacity for carrying traceable edits. The experiments on zsRE-10K and PARAREL-10K datasets reveal that iReVa has superior performance regarding edit success, generalization, and specificity. Further edit withdrawal test indicates that iReVa can explicitly manipulate the activation of neurons and easily withdraw the edits.
strength_weakness: strength_weakness: 1.	The discussions on the limitations and broader societal impacts of iReVa are not included in the paper. I have some questions about the application scope of the proposed method. Please see the questions below.

Questions
1.	Could iReVa lead to a dramatically increasing number of parameters? Let’s see if there are millions of knowledge for editing, how can you potentially insert all the knowledge into LMs with iReVa? 
2.	After you change a piece of knowledge, can the reasoning still be conducted for the edited knowledge? For example, if we have edited the president of America, could some reasoning questions like ``Who is the wife of the president of America” also be resolved with the new knowledge?
3.	Typo: ``evident’’ in line 6 should be ``evidence’’. Please check.
questions: questions: Please see weaknesses.
limitations: limitations: No, the author should discuss the limitations of the proposed method such as the application scope, the potential risks, and future improvement to indicate how robust the results are to violate the
assumption. I would like the author to add such information during the rebuttal.
paper_summary: paper_summary: This paper introduces a novel method called iReVa for knowledge editing. iReVa initializes and retrofits key-value pairs into MLP blocks to create a new mapping of knowledge without affecting related information. Compared to existing methods, iReVa offers better interpretability and a stronger ability to make traceable edits.
strength_weakness: strength_weakness: 1. The color in the figure is not obvious to discriminate between the original knowledge neurons and new knowledge neurons.
2. The computation of the proposed method is similar to T-Patcher, I'm curious about the difference between them. The proposed methods are designed to tackle the batch edit, but it seems it still needs to add one neuron for each example.
questions: questions: 1. Line 57-59 focus on self-attention, while the author selects MLP to further investigate, the logic here is a bit confusing. Is there something I missing?
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces iReVa, a novel method for model editing that explicitly initializes and retrofits key-value pairs into MLP blocks of transformer models to perform CRUD (Create, Read, Update, Delete) operations on LMs. iReVa aims to update knowledge in LMs without damaging irrelevant knowledge, offering better interpretability and traceability compared to existing methods. The method is validated through experiments on GPT series models, showing significant improvements in edit success and generalization without affecting specificity.
strength_weakness: strength_weakness: This paper could benefit from a more detailed comparison with other model editing methods, especially those focusing on lifelong learning and continual editing [1][2].

It does not discuss the computational efficiency of iReVa in terms of inference time or memory, which is crucial for real-world applications.

The reliance on the hypothesis that factual knowledge is stored in MLP blocks may be limiting [3], and the authors could explore the broader implications of this assumption.

The method's applicability to other types of tasks, such as erasing hallucinations, is not validated.

There is a noticeable absence of experimental validation on other recent and updated models such as GPT-J (used by ROME etc.), LLaMA.

The technical novelty of iReVa is somewhat limited, as it builds upon existing concepts like MEMIT [4] and key-value memory structures in MLPs [2].

The absence of a strategy for selecting the adaptor layer may hinder the method's rapid migration and application to various language models。

Equation 3 requires clarification, why 'i' and 'o' in Equation 3 are both passed through SELF_ATTEN again?

References

[1] Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors, Hartvigsen et al,
Neurips 2023.

[2] Transformer-Patcher: One Mistake worth One Neuron, Huang et al, ICLR 2023.

[3] What does the Knowledge Neuron Thesis Have to do with Knowledge? Niu et al, ICLR 2024

[4] Mass-Editing Memory in a Transformer, Meng et al, ICLR 2023.
questions: questions: See weaknesses.
limitations: limitations: No
paper_summary: paper_summary: The paper addresses the tracable sequential model editing challenge by plugging in additional model components to a transformer MLP blocks. The proposed approach adds additional model components for each edit, allowing for traceability for each edit.
strength_weakness: strength_weakness: •	The overall approach does not appear to be novel, as it closely resembles T-Patcher. Both iReVa and T-Patcher rely on inserting neurons for editing and using neuron activations to control when to run the patch/adopter. Furthermore, analysis of the editing approach across different layers reveals the same pattern as discussed in the T-Patcher paper which involves adding additional model components in the final layer for optimal results.

•	Experiments with T-Patcher are missing from the comparisons to the existing methods section. Given its similarity, T-Patcher should be included for comparison.

•	Although T-Patcher performs editing in batches, it still uses a single patch of neurons for each edit, making its editing similarly traceable. Thus, the paper's claim of a "stronger capacity for carrying traceable edits" seems unfounded.

•	The Edit Withdrawal Test section is hard to understand. How exactly was the experiment conducted? Were all edits removed or only a limited set? Detailed experimentations for this section are needed as it is the only use case of traceability explored in the paper.

•	Editing techniques that rely on code books with playback vectors e.g. GRACE would allow for edits to be removed. The authors should make it clear that the withdrawal test is not possible for the editing techniques that they have chosen for comparison.
questions: questions: N/A
limitations: limitations: N/A

Comments: title: Reply for Authors' Response
Thanks for your constructive reviews
comment: Thank you for your insightful response! Your concern indeed requires further clarification, as there are some similarities between Method iReVa and Method GRACE when establishing adaptors for multiple edits. Therefore, we will further compare the traceability of iReVa and GRACE, and we hope this will help address your concern.

- Similarity. In general, Method iReVa can be seen as a variant of Method GRACE, where each cluster is limited to only one edit, which is the cluster center itself. Additionally, the activation bias in iReVa can be considered similar to the cluster radius in GRACE. If, in a benchmark, it is rare for multiple edits to share the same edit target, then iReVa and GRACE are essentially similar because GRACE's codebook rarely involves the expand operation.

- Difference. If, in a benchmark, multiple edits share the same edit target (e.g., country name), GRACE‘s codebook will experience a trade-off based on the size of the initiative radius. A radius that is too large can lead to errors and forgetting, while a radius that is too small can result in too many clusters. In contrast, Method iReVa always maintains one edit per neuron, avoiding the issue of edits forgetting. The phenomenon of forgetting often occurs during the split stage in the codebook. When the current edit has a similar input representation x to a certain cluster center C0 but has a different edit target y, GRACE's codebook will perform a split operation, creating a new cluster C1 from the current edit and splitting the radius between the new cluster and C0 based on their distance. In this process, the reduction in C0's radius can cause some edits that originally belonged to C0 to fall outside of C0. If these edits are fundamentally different from C0, the codebook loses these edits. You mentioned in your response that Method GRACE has a low probability of merging two unrelated edits into the same cluster. Considering that different clusters in GRACE's codebook do not overlap with each other, and the representation space for cluster centers is limited, when the number of edits is large (GRACE performs 1K edits whereas iReVa performs 10K edits), the increased number of clusters filling the space may involve more expand or split operations, potentially introducing more errors.

Additionally, regarding the trade-off phenomenon you mentioned between Method iReVa, we discussed this in Section 4.3 of our paper. During training, although the neurons corresponding to previous edits are frozen during backpropagation, they are included in the forward propagation calculations. This mechanism will introduce noise to forward propagation so as to increase the robustness of the editing.

These comparisons will be included in the revised paper after simplification. Thank you again for your valuable feedback.
Thank you for your detailed response. While my concerns about T-Patcher have been addressed, I remain somewhat skeptical about its traceability compared to GRACE.

The GRACE algorithm processes edits individually in a streaming fashion(batch size 1). Each edit typically initiates its own cluster, with the cluster radius being dynamically adjusted based on the labels of subsequent edits that fall within its scope. The default cluster radius is maintained at a relatively small value to minimize the probability of grouping unrelated edits. However, the possibility of unrelated edits being clustered together, while very low, cannot be entirely eliminated. The paper should have a detailed discussion on the traceability of edits in comparison to other works and a comparison between iReVa and T-Patcher as in the global rebuttal. 
The default GRACE approach may impact performance. However, since iReVa does not share neurons across various edits, it could have performance trade-offs in comparison with T-Patcher.

Given the current presentation of the work, I do not feel comfortable raising the score, I will be maintaining my current positive evaluation of your manuscript.
Thanks for your reply. I have read your response carefully. I will keep my initial rating for your manuscript.
Thank you for your reply, my concern has been addressed. I have raised my score.
Thank you for your detailed response, and I will be maintaining my positive evaluation of your manuscript.
Dear Reviewers,

We sincerely thank you for your meticulous and insightful feedback on our manuscript. Your valuable comments have significantly improved the quality of our work, and we greatly appreciate the time and dedication you devoted to reviewing our paper.

We wanted to kindly follow up and see if you’ve had a chance to review our response. We hope it has addressed the concerns you raised in your review. If there are any additional questions or if further clarification is needed, we would be more than happy to assist.

Thank you again for your valuable time and thoughtful consideration.

Best regards,
The Authors