Scalable Model Editing via Customized Expert Networks

Authors: zihan  yao
yu  he
tianyu  qi
ming  li
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 16:26:40

Abstract: Addressing the issues of hallucinations and outdated knowledge in large language models is critical for their reliable application. Model Editing presents a promising avenue for mitigating these challenges in a costeffective manner. However, existing methods often suffer from unsatisfactory generalization and unintended effects on non-edited samples. To overcome these limitations, we introduce a novel approach: Scalable Model Editing via Customized Expert Networks (SCEN), which is a two-stage continuous training paradigm. Specifically, in the first stage, we train lightweight expert networks individually for each piece of knowledge that needs to be updated. Subsequently, we train a corresponding indexing neuron for each expert to control the activation state of that expert. We conducted a series of experiments on the ZsRE and Hallucination benchmarks by tuning the advanced open-source LLM, Llama2, achieving state-of-theart results compared to current mainstream methods. Our code is available at https://github.com/TAL-auroraX/SCEN.

Main paper: 
 introduction: 
 With the introduction of the transformer architecture (Vaswani et al., 2017) , the performance of various tasks in the field of NLP has improved rapidly, however, that has been accompanied by an explosive increase in the number of model parameters (Zhao et al., 2023; Shoeybi et al., 2019) . While we are astounded by the outstanding results produced by large language models (LLMs) (Touvron et al., 2023) , we also face challenges posed by their toxic and biased responses, hallucinations of factual information, and outdated knowledge (Ladhak et al., 2023; Huang et al., 2023a; De Cao et al., 2021) . To address these issues, a widely adopted approach is to fine-tune LLMs for alignment, although this incurs significant computational and time costs. For LLMs deployed in industry, this issue becomes even more critical, particularly within the realm of Education, as toxic content and factual hallucinations can significantly degrade user experience (Liu et al., 2022) . provided an incorrect answer due to outdated knowledge (Q2). This can be corrected using model editing techniques (Q3), while ensuring a certain level of generalizability (Q4). In addition, other non-edited questions remain unaffected (Q5).• We have conducted an in-depth exploration of the mechanisms underlying the storage of factual knowledge within LLMs, as well as the capacity of individual neurons to learn correct samples during sequence editing. This research contribution to enhancing the interpretability of LLMs.  
related work: 
 Probing Knowledge in Pretrained Models. Recent studies have highlighted that large language models (LLMs), such as BERT (Petroni et al., 2019) and T5 (Jiang et al., 2020; Roberts et al., 2020) , effectively store and recall factual knowledge within their parameters, acting as knowledge bases. This capability is leveraged for tasks like closed-book question answering, demonstrating their potential to memorize information from extensive pretraining corpora without further fine-tuning. Probing techniques and analysis have shown that such knowledge is encoded in specific neurons, particularly within the Feed-Forward Network(FFN) layers of transformer models (Meng et al., 2022a) . Researchers have identified these "knowledge neurons" at the top layers of pretrained models and have developed methods to edit factual knowledge by manipulating these characteristic FFN layers (Dai et al., 2021) . This reveals a deeper understanding of the interpretability and structure of transformers, enabling more targeted and computationally efficient fine-tuning.As mentioned in the Introduction, there are three research directions in the mainstream model editing methods, categorized by the location of new knowledge storage. Fine-tuning stands out as a straightforward and intuitive approach, which updates knowledge by updating the original model parameters. Both ROME (Meng et al., 2022a) and MEMIT (Meng et al., 2022b) exhibit enhancements derived from this perspective. This methodology preserves the original parameters and architecture of the model; however, it frequently precipitates catastrophic forgetting. To mitigate this issue, some researchers have recommended strategies that integrate external space into the model. For example, GRACE (Hartvigsen et al., 2024) adopts a novel approach by creating codebooks that store modifiable samples and employs the Euclidean distance metric to navigate the selection of specific parameter trajectories during the inference phase. Alternatively, SERAC (Mitchell et al., 2022) utilizes cache memory units to chronicle information relevant to model editing and executes these modifications by deploying classification models for retrieval tasks. Nonetheless, these strategies necessitate the incorporation of additional storage components, which may be a limitation. Meanwhile, some researches are investigating approaches that involve adapting the model's architecture by integrating additional neurons. An example of this approach is T-Patcher (Huang et al., 2023b) , which introduces new neural units for each token identified as erroneous, aiming to assimilate newly edited knowledge through an end-to-end learning process. While T-Patcher reduces the need for extra memory space and ensure the success rate of editing, it can result in an exponential increase in parameter space and computational overhead. This issue becomes particularly pronounced in the context of LLMs.Continual Learning(CL) in LLMs. CL is an essential aspect of machine learning as it enables models to adapt to new tasks while retaining performance on previous ones. Recent advancements, as delineated by Lin et al. (2022) , introduce an innovative paradigm known as continual model refinement (CMR). This paradigm is designed to efficiently rectify predictive inaccuracies encountered in out-of-distribution data streams, while simultaneously circumventing the peril of catastrophic forgetting. The notion of sequential model editing (SME), as expounded by T-Patcher (Huang et al., 2023b) and further developed in the work of GRACE (Hartvigsen et al., 2024) , can be considered a specialized instantiation of CL. This approach entails the processing of individual data samples in a sequential manner. However, it grapples with the same challenge of catastrophic forgetting that is characteristic of broader CL approaches. 
problem formulation: 
 Define the model f model as the one obtained by training on the data D train , and we denote the subset of D train where f model makes exact predictions as D loc . Suppose we have a model editing task using the dataD edit = [d 1 , d 2 , ..., d n ],where each d i represents a sample containing input and output information (x i ,x r i ,y i ). x i is the input to be edited, and y i is the desired output. x r i is a semantically equivalent rewrite of x i , and the corresponding label for both x i and x r i is y i . We denote the sequential editing process as the time step t ∈ [1, n], where the data required for each time step corresponds to d i in D edit . Meanwhile, we define f t as the model obtained after editing at the time step t. Based on the aforementioned settings, three properties are used to evaluate the effectiveness of the sequential model editing method.Property 1. Reliability: The edited model f t obtained at time step t, with the set of all samples up to time step t denoted as [d 1 , ..., d t ], needs to meet the following expectations:EQUATIONProperty 2. Locality: For non-edited samples D loc , f t needs to ensure that they remain unaffected:EQUATIONProperty 3. Generality: f t needs to ensure that the rewritten sample x r t also receives a correct response after editing.:EQUATIONInspired by the Mixture of Experts (MoE) architecture (Shazeer et al., 2017; Fedus et al., 2022) , we propose to incorporate a series of expert networks for each incoming edit sample in a sequential editing setting. During inference, different requests will be routed to the corresponding experts to ensure optimal performance.All transformer-based language models (LMs) are constructed using stacked transformer blocks. Assuming we are editing the lth layer of the LLM, and that the FFN part of lth layer has two components, W up and W down . We consider W down as a lightweight expert network, with the reason provided in the ablation studies section. We train a corresponding W t down for each sample that needs to be edited. During training, we freeze all the other weights of the f model and activate only W down . As shown in Stage 1 of the left side of Fig. 2 , after each training of W t down is completed, the weights are stored in the local database, and will be used during inference when subsequent activation occurs. Since our expert networks are trained with sample-level granularity, each expert is responsible for handling a single edit sample. This ensures that there is no interference between different samples and prevents the forgetting of multiple edits.After training multiple customized expert networks, it is necessary to activate a specific expert during the inference phase to achieve the desired output. This is not a standard classification task where a sample is assigned to a specific expert. Due to the streaming nature of sequential editing, using a fixed neuron for training will inevitably lead to catastrophic forgetting. Additionally, utilizing a classification model to assign samples to experts would incur additional time costs. Inspired by T-Patcher, we propose a Scalable Indexing Neurons method, which trains a corresponding neuron for each expert using the same edit sample through a specific loss function, and these neurons are incrementally added in sequence as the editing progresses. The newly trained neurons integrated with the existing neurons and utilized during the inference phase. In the following we will provide a detailed description of how the neurons are employed during the training, merging, and inference phases.Training Indexing Neurons. The settings are identical to those used for training a customized expert network with editing at lth layer of the LLM. We introduce a new network structure W neuron which is placed after W up . This network consists of a single output neuron, implying that the parameter size of W neuron is 1 × h, where h represents the dimension of FFN hidden layer. And we employ a sigmoid activation function for the output. The completed calculation is as follows:a t = Sigmoid(W neuron W up (x att t )) (4)where x att t is the representation of the last token in x t obtained after the attention layer of the lth layer.We determine whether a sample needs to be edited based on the activation value a. Samples that require editing, or those that are semantically consistent with these samples, typically exhibit larger activation values, whereas non-edited samples show smaller activation values. To achieve the aforementioned objectives, we design the loss function from three aspects to control the activation value of a . For Reliability and Generality, the effect is ensured by l activate loss, which is calculated as follows:l activate = exp(-a t )(5) where a t represents the activation value that is required at the current time step t to edit the sample.l disactivate loss and l margin loss are used to ensure Locality. We consider all the samples that need to be edited before the current time step t as negative examples a i , i ∈ [1, ..., t -1]. Here, we define the following two loss functions based on the activation values of these negative samples.l disactivate = 1 t -1 t-1 ∑ i=1 exp(a i + α) (6) l margin = 1 t -1 t-1 ∑ i=1 exp(a i -a t + β) (7)where α and β are the corresponding hyperparameters, we aim to minimize the activation values corresponding to a i , maximize the value of a t , and use α and β as the respective penalty terms. In our experiments, α was set to 0.7 and β was set to 0.3. Therefore, the complete loss function for training indexing neurons is:loss = l activate + m(l disactivate + l margin ) (8)m is the hyperparameter as a trade-off between activation and disactivation.Note that during the training of W neuron , only the weights prior to the l-th layer of the LLM are used. Similarly, we only activate W neuron for gradient updates while freezing all other weights.Merge Neurons. We denote the weight of the indexing neuron trained at time step t as W t neuron . Assuming that we have edited t samples, we should be able to get the weights of t indexing neurons [W 1 neuron ,W 2 neuron , ... ,W t neuron ]. During inference, learn to merge these weights together to obtain W merge ∈R t×h . We define it formally as follows:EQUATIONInference Stage. Assume we have a model that has already been edited at lth layer. Now, given a new test sample q , how can we obtain the activation values of the indexing neurons for this sample? When the forward inference reaches the lth layer, we can obtain the answer using the following formula. It should be noted that here we take the last token of the input q as the representation of the sample.EQUATIONwhere there are t activation values in the a q vector, each corresponding to an experts, denote a q i as the activation value at the ith position in a q . Here, we determine whether to activate an expert or use the model's original weights by setting a threshold θ.Following the rule described below, as shown in Eq.( 11). When the activation value is below the threshold θ, the model's original weights W down are used, without activating any experts. Conversely, if the maximum activation value a q exceeds the threshold, the index corresponding to this maximum value will indx to a unique expert, which will be used to replace the original weights, as shown in Eq.( 12). After determining the structure of the FFN l at the lth layer, the inference proceeds forward. The complete inference process is illustrated in the right half of Fig. 2 .EQUATIONExpert(a q ) = arg max i a q i (12)4 Experiments 
experimental setup: 
 Datasets. We utilized two benchmark datasets suitable for sequence editing, ZsRE and Hallucination, to evaluate the effectiveness of our method. ZsRE is a comprehensive question-answering (QA) dataset (Levy et al., 2017) . We randomly selected 20,000 entries from this dataset. Specifically, 10,000 entries,denoted as D train , were used for typical supervised fine-tuning (SFT) based on the LLaMA2 model, while the remaining 10,000 entries, denoted as D test , were reserved for evaluating the performance of the SFT model. Based on the evaluation results, we divided the post-test data into two subsets: D edit and D loc . The subset D edit consists of samples where the SFT model made incorrect predictions. From this subset, we selected the first 200 and first 1000 samples for subsequent editing operations. Conversely, D loc comprises samples where the SFT model made correct predictions. Similarly, we selected the first 1000 samples from this subset to evaluate our method's Localization metric. Detailed information is provided in Appendix A. Hallucination is a dataset designed to evaluate the performance of model editing methods in of mitigating model hallucinations, as described by Manakul et al. (2023) and (Hartvigsen et al., 2024) . Manakul et al. (2023) prompt GPT-3 to generate 238 wikipediastyle biographies for concepts extracted from WikiBio, they then annotate factual accuracy of each sentence, noting which are hallucinations. We followed the processing method described by (Hartvigsen et al., 2024) and created 1392 sequential edits and 592 accurate outputs. Based on the WikiBio dataset, we incorporated 200 samples from OpenWebText and trained Llama 7B and 13B respectively.During the editing phase, we selected the first 200 out of the previous 1392 sequential edits for evaluation, denoted as WikiBio-E. Additionally, we used the results from 516 accurate outputs, referred to as WikiBio-A, combined with 200 samples from OpenWebText as a test set to evaluate the localization.Baselines. We compare the proposed method with several mainstream model editing methods. Fine-tuning based methods (including FT and FT+KL), FT refers to the scenario where the fully connected network within a certain transformer block is trainable,while, FT+KL incorporates the Kullback-Leibler (KL) divergence into the loss function, as described in (Mitchell et al., 2021) , to mitigate catastrophic forgetting. Memory-based methods, such as SERAC (Mitchell et al., 2022) and GRACE (Hartvigsen et al., 2024) , have shown significant promise. GRACE, in particular, is the latest state-of-the-art (SOTA) method based on memory. MEND (Mitchell et al., 2021) aims to learn a HyperNetwork using additional training data to transform the gradients obtained through standard fine-tuning. MEMIT, as proposed by (Meng et al., 2022b) , is a method for directly updating a language model with many memories.In the realm of generative tasks for LLMs, T-Patcher (Huang et al., 2023b ) introduces neurons at the token level. Inspired by this approach, we explore the addition of neurons at the granularity of samples. However, due to the inherent characteristics of T-Patcher, it exhibits substantial training and inference performance constraints. Consequently, we did not experiment with T-Patcher on the Hallucination dataset nor perform a thousand edits on the ZsRE dataset. Detailed parameter settings and experimental details for each method are reported in Appendix C.We used Accuracy and Perplexity to evaluate the ZsRE and Hallucination datasets. In ZsRE, the three metrics Reliability, Generality, and Locality refer respectively to whether all edited samples are successful, whether rewrited samples can answer correctly, and whether non-edited samples can answer correctly. In Hallucination, WikiBio-E, WikiBio-A, and OpenWebText refer to whether the perplexity of the edited samples is reduced and whether the perplexity of the unrelated WikiBio-A and OpenWebText can be maintained at a lower level, respectively.We conducted 200 and 1000 sequential editing experiments on Llama-2 7B and 13B respectively, as shown in Tables 1 and 2 , respectively. For the ZsRE dataset. We focus on two aspects: the trade-off between Reliability and Locality, and the average performance across three metrics. For the Hallucination dataset, we examine whether the ppl of WikiBio-E decreases, and whether the ppl of WikiBio-A and OpenWebText increases due to halluciation effects. The FT+KL method could not be evaluated on the Hallucination dataset due to it's requirement for additional data.The MEMIT method is not a sequential editing method and performs significantly below baseline levels in hallucination scenarios, so it is not used in our comparison.On the ZsRE dataset, both at 200 and 1000 edits for long sequences, Tables 1 and 2 demonstrate that our proposed method(SCEN) substantially outperforms the comparison methods on average across all three metrics. In terms of Generality and Locality, SCEN achieves a better trade-off on both the Llama2 7B and 13B models, and it also demonstrates improved stability. In contrast, the direct fine-tuning(FT) method demonstrated that while Reliability and Generality were maintained, there was a significant drop in Locality. To address this limitation, FT+KL was utilized; however, its impact on improving Locality was minimal. The GRACE achieved the best results on Locality but its Generalization was very insufficient. Similarly, SEARAC exhibited an imbalance between Generality and Locality. T-Patcher achieves results close to SCEN on the Reliability and Generality metrics. However due to the addition of neurons for each token which significantly increases the number of parameters and the fact that T-Patcher is an end-to-end approach, these factors collectively contribute to its unsatisfactory performance on the Locality metric. MEMIT is not a sequential editing approach, and thus it does not achieve superior results across all three metrics. Overall, we propose the dynamic addition of indexing neurons, which can accurately activate experts for long sequence edits without catastrophic forgetting. This method strikes a balance between Generality and Locality without resorting to extremes.On the Hallucination dataset, it can be seen from Table 1 that FT is an effective method that drastically reduces the perplexity of editing samples. Due to its excellent performance on WikiBio-E, FT achieved the best result on the Llama2-7B model, with SCEN ranking second. On the other hand, SCEN achieved the best results on the Llama2-13B model, maintaining a low level of average perplexity across the three test sets. Meanwhile, GRACE was competitive and secured second place. In summary, it can be seen that SCEN also maintained a relatively good stability in the hallucination relief task.Ablation Study. We conducted two ablation experiments in SCEN to investigate the contribution of each component. Specifically, we examined the impact of using different positions of weights as experts for editing and employing different training loss functions for indexing neurons. The complete experimental results are shown in Table .3, where the top three rows illustrate the effect of editing on different positions of weights. W attn indicates that editing is performed in the attention layer, W up indicates that editing is performed in the upper half of the fully-connected layer following the attention layer, and the corresponding W down indicates that editing is performed in the lower half of the fullyconnected layer. For a more concise description of W attn , W up and W down , please refer to One sample in One neuron Two sample in One neuron Four sample in One neuron Fig. 2 . The following three rows in Table .3, please refer to the effect of employing different training loss functions for indexing neurons on the final results. w/o l disactivate &l margin represents the scenario in which we retain only the activation loss, while w/o l margin represents retaining both the activation loss and the disactivate loss. It can be observed that in terms of the positioning of expert weights, W down achieves the optimal performance in editing, whereas W attn exhibits the poorest performance. These findings are consistent with the results reported by ROME (Meng et al., 2022a) . In terms of loss, the main contribution is attributed to l disactivate , while the effect of addition of l margin also shows improvement. It is evident that a single neuron is insufficient to accurately model multiple diverse samples. We also demonstrate the activation of the corresponding indexing neurons for different numbers of samples, as illustrated in Fig. 3 . The horizontal coordinates represent the indexing neurons corresponding to each edited expert, while the vertical coordinates represent the sequentially edited samples. In the left figure, as it is a one-to-one sample, clear highlighting can be observed along the diagonal. The middle and right figures illustrate the two-to-one and Reliability Generality Locality Which layer is better for editing? We examined the performance of the even-numbered layers of Llama2-7B separately, as shown in Fig. 5 . It is evident that in the lower-level transformer blocks of Llama2-7B, editing has minimal effect, thereby preserving the original model results and maintaining a high level of Locality. In contrast, in the higher-level transformer blocks, the impact of editing is pronounced. Starting from the 16th layer onward, Reliability remains at a high level, with minimal fluctuations in Generality and Locality. Consequently, we further validate that the higher-level transformer blocks of LM , based on the transformer architecture, contain some factual knowledge, and editing of these layers has a significant effect. GRACE constructed a dataset via SelfCheckGPT (Manakul et al., 2023) to evaluate the effect of model editing methods to reduce model illusions, and called this dataset Hallucination. Manakul et al. (2023) utilizes GPT3 to extract concepts from WikiBio to generate corresponding biographies. For each sentence generated, they labeled which sentences were factual and which were hallucinatory. Follow the idea of GRACE to edit the highly inaccurate sentences and replace them with real Wikipedia sentences. The highly accurate samples are utilized to judge whether the model still maintains a relatively low level of perplexity on these samples after editing. Through the above processing, a total of 1,392 editing samples and 516 sentences that have been accurate have been generated. Compared with the dataset used in the previous model editing, Hallucination has a longer text length, which is more in line with the data situation in the real scenario. In this paper, we used the first 200 of the 1392 edited samples for our experiments, in the two Llama2-7B/13B models. Partial data are shown in Table 6 . Detailed parameter settings using the comparison method used are described in detail below:For FT+KL, additional data is needed to be used with the KL loss. In the ZsRE data, we should have a vector indexing model(all-mpnet-base-v2) for each edit sample from the 5 most similar samples found in the training data of SFT. The training step for each edit sample is 10 and the learning rate is 0.0001.For the MEND, additional data is needed to train an editor, and to avoid any test set leakage, in the ZsRE data, we use 2000 entries from the data used to train the SFT to train the editor.In the Hallucination dataset, we use the last 400 data out of 1352 entries to train the editor.For the SEARAC method, the method contains a total of three models, of which the classification model we use is distilbert-base-cased, and the threshold we set is 0.6.For the GRACE method, ϵ is set to 3.0, the number of iterations is set to 10, the learning rate is set to 1.0, and the adapter weights are taken to be randomly initialized.For MEMIT, we only experimented on ZsRE, the fact token used makes the last token of the entity, and the positions to be edited are the 20th and 22nd transofmerblock. 
 
 Appendix:  


Decision: The paper introduces SCEN, a two-stage editing method for large language models that uses expert networks and indexing neurons to handle sequential edit requests. This approach enhance model performance by incorporating fine-tuned value parameters within expert networks, showing significant improvements in question-answering and hallucination reduction tasks.
Reviewers appreciate the novelty of the SCEN framework and the strong experimental results it showed. There was complaints about missing comparison to the very relavent prior work, which is addressed during the rebuttal. There are some minor concerns on limited task scope, and clarity issues in the writing.

Reviews: review: paper_summary: paper_summary: This paper proposes a two-stage editing method based on expert networks, called SCEN, in the “sequential” setting that a sequence of edit requests is presented. 

Overall, the proposed two-stage method, the idea of incorporating the FNN value parameters as expert networks with the indexing neurons, is interesting and novel, showing the effectiveness in the experiment settings. A work is a reasonable extension of T-patcher, towards maintaining multiple knowledge neurons and experts. 
 However, the papers are not very convincing on methods and experiment settings as follows: it is not convincing why two-stage methods are required, so comparing to T-patcher, architectural advantages are not clear. Another problem is that the proposed method seems not to be scalable well, as the number of additional indexing and weight parameters is just increasing monotonically for larger scale of edits. A simple extension of T-patcher which is not in two-stage manner needs to be compared. Additionally, the paper needs to be rewritten, particularly because the method of the first stage is not very clear (how to train FNN’s down projection weight). 
Nevertheless, given its positive aspects of the paper, I am currently leaning to the acceptance, while the current decision is likely changed to lower scores.
strength_weakness: strength_weakness: 1.	For each edit request, the proposed editing methods monotonically create a new indexing neuron with a weight vector, and need to keep its own FNN’s value parameters. As a result, if N number of sequential edits is provided, we need to keep N indexing neurons associated with their weight vectors as well as N FNN values parameters separately. It is largely questionable whether increasing the number of parameters is scalable in the practical situation. Even edits where their knowledge and topics are very similar will have their own separate indexing neurons and FNN value parameters, causing the method inefficient.  
2.	In the two-stage method, the actual editing process is performed at the first stage, while an indexing neuron in the second stage just plays of an “indexing role” which turns on or off, depending on the input query. It is not clear why these two stages are necessary. Why the methods are not designed in a unified manner (without two-stages), like T-patcher? 
3.	In the experiments, a simple extension of T-patcher not in the “two”-stage manner needs to be compared. 
4.	An important part is how to actually adjust the parameters of an expert, at the first-stage. But, in Section 3.2, this part is not very clear; how to train W_down^t (down projection weight)? which layers have their own expert parameters?
questions: questions: Please see weaknesses. Below are further comments. 
1)	Why counterfact dataset is not used, while it is used in most existing methods? 
2)	The method is expanded to other multi-hop reasoning setting, like effectively working on Mquake dataset? 
3)	In experiment results, only cases of 200 and 1000 edits are evaluated. How long an editing sequence where the proposed method is scalable well? Assuming that 1M or 100M edits are provided, what are the memory and inference costs in the proposed methods? In such large-scale edit, even applying Eq. (12) that selects the best expert requires non-trivial inference time.
paper_summary: paper_summary: The paper presents SCEN, a two-stage model editing approach for large language models that improves upon outdated knowledge and hallucinations. SCEN stands out for its custom expert networks per knowledge piece and indexing neurons that control expert activation, leading to state-of-the-art results in question-answering and hallucination reduction datasets.
strength_weakness: strength_weakness: * While you have mentioned T-Patcher, it is essential to provide a comprehensive overview of these related works and highlight the differences between your paper and theirs.

* Consider including an experimental discussion on the effects of fine-tuning the attention layer or the $W_{up}$​ weight. This exploration will provide valuable insights into the performance improvements achieved by these modifications.

* In Tables 1 and 2, the model performance based on Llama2-13B as the baseline is generally much worse than that based on Llama2-7B. The authors could explain or reasonably speculate on the reasons behind this.
questions: questions: Please refer to the section titled 'Reasons for Reject' for further details.
paper_summary: paper_summary: The paper proposes a method for continual model editing called SCEN. Like GRACE, it only modifies the l-th layer (a hyperparameter), specifically the down projection matrix, for each edit t which is then stored in a database (an "expert"); after T edits, at test time, the model must decide which expert to use (or none). GRACE does this by codebook maintenance, which requires no further training. SCEN proposes to do this by training an expert selector. There are presumably many ways to parameterize the selector, but the paper loosely follows the training scheme in T-Patcher to train a scalar projection for each edit to activate for that edit (by contrastive loss), then at test time uses the argmax of the projection for the given query if the value is above some threshold. 

Experiments are conducted on ZsRE (QA) and GRACE's version of hallucination task (reducing perplexity on edits while maintaining it on non-edits), using Llama-2. SCEN performs better than GRACE on edit success and generalization, but worse on locality; on average SCEN performs better.
strength_weakness: strength_weakness: - Somewhat limited novelty, which is a general problem with many variations of model editing methods (slight differences in parameterization/activation approach, which are often almost certainly over-optimized for small-scale models and datasets).
- No comparison with T-Patcher, which doesn't make sense to me. It has the same continual model editing setup, and this paper is inspired by it.
- More about T-Patcher: it bakes in the activation for the right expert into training. So even though it doesn't explicitly predict the expert at inference time (i.e., the neurons are not stored in a database but part of the model), it's kind of doing the same thing? T-Patcher is supposed to be also scalable (i.e., the added neurons are a minor portion of the model for thousands of edits). 
- Limited tasks: why only ZsRE and Hallucination, when the main baseline GRACE also includes SCOTUS? 
- Some writing issues (unnatural)
paper_summary: paper_summary: This paper presents an LLM editing method using customized expert networks and indexing neurons.  Experiments have been conducted to demonstrate the effecitveness of the proposed method.
strength_weakness: strength_weakness: The evaluation part of this paper can be further improved.
1) Some ablation studies are expected to analyze the effectiveness of different component of the proposed method, such as the different losses in Eq.(8).
2) How about the space and time efficiency of the proposed method since a separate expert network is necessary for each edited knowledge?

Comments: title: Overall Comment
Reponse on authors' rebuttal
Request for response
Request for response
Request for response
Request for response
comment: Dear Reviewer 67Nb,

Thank you for increasing the score! 

Sorry, our reply may not be clear enough.For some unclear points, please refer to the following explanations:

The expert network trained in the first stage is referred to as $W_{down}$. During training, we freeze the other weights and only train this particular weight. The reason for using indexing neurons is to accurately call the corresponding expert network during inference.

The use of a two-stage process ensures non-interference and offers better performance on the dataset. In [5GUJ], it can be observed that SCEN, by adopting a two-stage process, has fewer erroneously activated neurons.

We will further revise our paper for the camera-ready version, hoping to address all your concerns by then. We are grateful for your recommendation once again.
We sincerely appreciate all reviewers' time and efforts in reviewing our paper. We are glad to find that reviewers generally recognize our strengths:

- The proposed two-stage editing method is **interesting and novel** [67Nb,4Ci8], the proposed expert network and index neurons for implementing selective editing methods are **promising** [amby].

- The method is **simple and a reasonable modification** of the previously proposed model editing methods [5GUJ].

- Experiments on different scales of Llama2 outperform existing model editing methods [67Nb,amby], Achieved the **best balance** among the three indicators of reliability/generalization/locality [5GUJ].

We also greatly appreciate the reviewers pointing out the shortcomings of the paper, for which we have supplemented detailed experiments and discussions. The following content is a summary of our rebuttal:

**(1). Detailed comparison with T-patcher**

- We made comparisons on Llama2-7B and 13B, and outperformed the T-patcher method on both the 7B and 13B models.

|               | Reliability | Generality | Locality |  Avg |
|---------------|:-----------:|:----------:|:--------:|:----:|
|  T-Patcher-7B |     86.0    |    76.3    |   20.5   | 60.9 |
|       SCEN-7B |     100     |    90.0    |   83.3   | 91.1 |
| T-Patcher-13B |     84.5    |    71.2    |    8.5   | 62.9 |
|      SCEN-13B |     99.5    |    80.7    |   83.5   | 87.9 |

- We also detailed the statistics of erroneous neuron activations when continuously editing 200 times with T-Patcher and SCEN. It can be seen that SCEN has fewer erroneously activated neurons compared to T-patcher.

|           | False Activation(%) | Correct Response(%) |
|-----------|---------------------|---------------------|
| T-patcher | 71                  | 48.5                |
| SCEN      | 7.5                 | 93.5                |

- We compared the GPU usage and inference time performance of T-patch and SCEN. SCEN uses less GPU memory during inference, but because it needs to schedule the expert network into GPU memory during inference, this will generate extra time consumption, so it takes longer than T-patcher.

|           | Ex-GPU(M) | In Edits(s) | Out Edits(s) |
|-----------|-----------|-------------|--------------|
| T-patcher | 121.664   | 127.7       | 126.76       |
| SCEN      | 20.996    | 303.5       | 125.32       |

**(2). Ablation Studies**

- In the paper, we use $W_{up}$ as the expert network, and we also compare the situations where other weights are used as the expert network. It can be seen that the effect is best when $W_{down}$ is used as the expert network.

|            | Reliability | Generality | Locality | Avg  |
|------------|-------------|------------|----------|------|
|  $𝑊_{attn}$ | 90.0        | 77.3       | 92.0     | 86.4 |
|  $𝑊_{up}$     | 96.5        | 83.2       | 86.7     | 88.8 |
|  $𝑊_{down}$ | 100         | 90.0       | 83.3     | 91.1 |

- For the index neurons, we demonstrated the impact of different loss composition methods on the final effect.

|                                    | Reliability | Generality | Locality | Avg  |
|------------------------------------|-------------|------------|----------|------|
| w/o $l_{disactivate}$ & $l_{margin}$ | 100         | 93.8       | 73.7     | 89.2 |
| w/o $l_{margin}$                   | 100         | 90.6       | 82.1     | 90.9 |
| SCEN loss (Eq. 8)                  | 100         | 90.0       | 83.3     | 91.1 |

**(3) Issues in the use of the dataset**

Our approach primarily focuses on sequence editing, with a specific emphasis on comparing with previous sequence editing methods. Therefore, in terms of dataset selection, we follow the datasets used in previous works.

Please contact us if we can do something else to help you better understand and recommend our paper.
I appreciate authors' efforts to address my questions. 

While I now better understand the paper, the presentation part -- particularly how the parameters are modified at the first stage --- is still unclear. Also, in the case of expert network (W_down^expert & W_merge) , it is still not very convincing to me why we need to prepare two types of neurons -- neuron and weight neurons, so requiring better explaination and motivation. It seems that the use of only a single type of neuron enables to handlie the required behaviors. Otherwise, a kind of justification and abliation studies may be necessary. 

Overall, I like the basic idea of this paper, as a good extension of T-patcher, with promising performances. 
So, I have increased my recommendation score up to 7.
Dear reviewer 67Nb,

We have posted extra experiment results (e.g., a supplemented baseline and other ablation studies) and clarifications to answer your questions. We wonder if you can let us know whether our responses address your concerns.

Look forward to your reply.

Best regards,

Authors
Dear reviewer amby,

We have posted extra experiment results (e.g., a supplemented baseline and other ablation studies) and clarifications to answer your questions. We wonder if you can let us know whether our responses address your concerns.

Look forward to your reply.

Best regards,

Authors
Dear reviewer 5GUJ,

We have posted extra experiment results (e.g., a supplemented baseline for T-Patcher) and clarifications to answer your questions. We wonder if you can let us know whether our responses address your concerns.

Look forward to your reply.

Best regards,

Authors
Dear reviewer 4Ci8,

We have posted extra experiment results (e.g., a supplemented baseline and other ablation studies) and clarifications to answer your questions. We wonder if you can let us know whether our responses address your concerns.

Look forward to your reply.

Best regards,

Authors