Scalable Model Editing via Customized Expert Networks
Authors: Zihan Yao
Yu He
Tianyu Qi
Ming Li
Venue: COLM (2024)
Keywords: Large Language Model,Model Editing,Continual Learning
TL;DR: We propose a two-stage continual training paradigm for Model Editing, which can achieve state-of-the-art performance through the addition of extra expert networks and neurons in sequence.

Abstract: Addressing the issues of hallucinations and outdated knowledge in large language models is critical for their reliable application. Model Editing presents a promising avenue for mitigating these challenges in a costeffective manner. However, existing methods often suffer from unsatisfactory generalization and unintended effects on non-edited samples. To overcome these limitations, we introduce a novel approach: Scalable Model Editing via Customized Expert Networks (SCEN), which is a two-stage continuous training paradigm. Specifically, in the first stage, we train lightweight expert networks individually for each piece of knowledge that needs to be updated. Subsequently, we train a corresponding indexing neuron for each expert to control the activation state of that expert. We conducted a series of experiments on the ZsRE and Hallucination benchmarks by tuning the advanced open-source LLM, Llama2, achieving state-of-theart results compared to current mainstream methods. Our code is available at https://github.com/TAL-auroraX/SCEN.

Published as a conference paper at COLM 2024
Scalable Model Editing via Customized Expert Networks
Zihan Yao, Yu Tianuu Qi Ming Li TAL Education Beinup,,ejing,CCCC China (yaozihanl, heyu26, qitianyu3, Liming9)etal.com
Abstract
Addressing the issues of hallucinations and outdated knowledge in large language models is critical for their reliable application. Model Editing presents a promising avenue for mitigating these challenges in a cost- effective manner. However, existing methods often suffer from unsatis- factory generalization and unintended effects on non-edited samples. To overcome these limitations, we introduce novel approach: Scalable Model Editing via Customized Expert Networks (SCEN), which is a two-stage continuous training paradigm. Specifically, in the first stage, train lightweight expert networks individually for each piece of knowledge that needs to be updated. Subsequently, we train a corresponding indexing neuron for each expert to control the activation state of that expert. We con- ducted series of experiments on the ZsRE and Hallucination benchmarks by tuning the advanced open-source LLM, Llama2, achieving state-of-thr art results compared to current mainstream methods. Our code is available at https: /gi www.ttt.o.ooo....eooooootttttttt TAL alll-uuoora//CEE cwm/TAL
1 Introduction
With the introduction of the transformer urchtecttur(Vaswwall et al., 2017), the performance of various tasks in the field of NLP has improved rapidly, however, that has been accompa- nied by an explosive increase in the number of model parameters(Zhao et al., 2023; Shoeybi et al., 2019). While we are astounded by the outstanding results produced by large language models (LLMs) (Touvron et al., 2023), we also face challenges posed by their toxic and biased responses, hallucinations of factual information, and outdated knowledge (Ladhak al., 2023; Huang et al., 2023a; De Cao et al., 2021). To address these issues, a widely adopted approach to fine-tune LLMs for alignment, although this incurs significant computational and time costs. For LLMs deployed in industry, this issue becomes even more critical, particularly within the realm of Education, as toxic content and factual hallucinations can significantly degrade user experience(Liu et al., 2022).
Query
Ot: Who prapased the law universal gravllaiio??
LLM
Q1: Who procosed the law unisorsal gaviiatiin? - Newton
G2: Who Precident of the Unived Statee?
GP: Who Previtent of the United Stalee? Trump
Eolt
of United
rewiting
reariting
Oet Who qummerty hells the position of the U.S. presidancy?
Ge: Who curreity holds the positio the U.S. presidency? o Joe Bloen
Inslovant sample
Inolevant sample
What the capital Unied States?
What the capital United States? Washington
Figure 1: This is an example of model editing, where the Large Language Models (LLMs) provided an incorrect answer due to outdated knowledge (Q2). This can be corrected using model editing techniques (Q3), while ensuring certain level of generalizability (Q4). In addition, other non-edited questions remain unaffected (Q5).
*Corresponding Author

---

Published as a conference paper at COLM 2024
To effectively address the aforementioned challenges, researchers have proposed model editing technique(Wang et al., 2023). This approach involves modifying model parameters to rectify the model's outputs, aiming to achieve desired responses for specific questions without compromising the overall accuracy for others(Fig. 1).. Additionally, this approach is both efficient and cost-effective. A successful editing method requires the edited model to meet three criteria(Liu et al., 2022): Reliability (rectify the outputs where the original model failed), Generality (correct response for edits rewriting), and Locality (Non-edited knowledge remains unaffected). Current mainstream methods can be roughly categorized into three types based on the location of the knowledge to be updated(Yao et al., 2023):
Stored in an external space (Resorting to External Knowledge): This method effectively leverages contextual information to enhance the accuracy of model response. However, it may require substantial number of examples to achieve optimal performance. Addition- ally, its dependence on precise retrieval results can introduce noise.
Stored in added parameters (Merging Knowledge into the Model): This method directly modifies the model's parameters, enabling the integration of new knowledge with existing knowledge. However, this may lead to knowledge conflicts, and cause the forgetting of non-edited knowledge.
Stored in the model's own parameters (Editing Intrinsic Knowledge): Although this can achieve precise and enduring modifications to the model's knowledge, it requires a high level of understanding of the model's internal mechanisms and may lead to unforeseen side effects.
Despite some progress, these methods remain unsatisfactory due to their inherent limita- tions. Inspired by T-Patcher(Huang et al., 2023b), which addresses the issue of catastrophic forgetting during sequential editing by adding a series of neurons to the fully connected layer for each ertor-generating token, we similarly store each sample to be edited sequen- tially in distinct experts, and these experts are essentially components of a fully connected neural network. We then control the activation of each expert using a corresponding num- ber of trainable neurons. Therefore, we introduce a novel two-stage continuous training model editing paradígm, SCEN (Scalable Model Editing via Customized Expert Networks). Specifically, in the first stage, we train lightweight expert networks tailored to each sample that needs to be edited. In the subsequent stage, each expert is associated with an indexing neuron. These neurons are dynamically added to the network and trained using a spe- cialized loss function during the sequential editing process. Our approach is meticulously crafted to fulfill three essential criteria for model editing:
Reliability: SCEN ensures reliability by assigning a customized expert to each sample that needs editing, thereby mitigates the risk of interference among different samples. The indexing neurons ensure precise activation of each expert.
Generality: For rewriting samples, SCEN can identify the most suitable expert by indexing neurons, which are trained using a specific margin loss function, thereby enhancing the model's generalizability.
Locality: The architecture of SCEN preserves all the original model weights. As long as the indexing neurons remain inactive, non-edited samples maintain their original outcomes, thereby ensuring effective localization.
In summary, SCEN presents refined approach to model editing, achieving balance between preserving the integrity of original knowledge and incorporating new information, while maintaining the model's robustness and adaptability. The main contributions of our work can be summarized as follows:
We introduce SCEN, novel two-stage continuous training paradigm for model editing. SCEN trains a customized expert network for each sample to be edited, and ensures that each expert is activated only for the current sample through dynamic neuron indexing mechanism. SCEN is adaptable to any language model that is based on the transformer-based architecture.
We conducted experiments on two distinct sizes of large language models, Llama2- IB/1 focusing on two different tasks: question-answering and hallucination mitigation. The experimental results demonstrate that SCEN is an effective ap- proach for model editing, achieving state-of-the-art performance on both tasks.
2

---

Published as a conference paper at COLM 2024
We have conducted an in-depth exploration of the mechanisms underlying the stor- age of factual knowledge within LLMs, as well as the capacity of individual neurons to learn correct samples during sequence editing. This research contribution to enhancing the interpretability of LLMs.
Database
Newton
Joe Biden
Stage-1
Export
Blockn
Blockne
Origin Down
Expert
others
edit query
find expert
Replace
MUP Loyar
W.
eail
Stage-2
Indexing neurons
Who is the President
the United States?
oth ers
up up_proj
Ar Joo Bidon
transferiigg
Attn-Layer
Farror?
Hammittn Farrai
2025
Blocke1
Wi
self-attention
down weight (expert)
Embedding layer
layer
weight
We add
linear layer neuran) each time. finally we combine linear layers (nourons)
proposed the univorsal gravitation
Q2/Who holds the position of the United States President?
nx linear layer (n neurons)
Figure 2: SCEN Overall Architecture The left half of the figure represents the editing stages. Stagel is the process for training the experts and Stage2 is the process corresponding to training the indexing neurons. The right half represents the inference stage, where the corresponding experts are activated by the indexing neurons to complete the subsequent inference.
2
Related Work
Probing Knowledge in Pretrained Models, Recent studies have highlighted that large language models (LLMs), such as BERT (Petroni al., 2019) and T5 (Jiang al., 2020; Roberts et al., 2020), effectively store and recall factual knowledge within their parameters, acting as knowledge bases. This capability leveraged for tasks like closed-book question answering, demonstrating their potential to memorize information from extensive pre- training corpora without further fine-tuning. Probing techniques and analysis have shown that such knowledge is encoded in specific neurons, particularly within the feed-Forward Network(FFN) layers of transformer models(Meng et al., 2022a). Researchers have identified these "knowledge neurons" at the top layers of pretrained models and have developed methods to edit factual knowledge by manipulating these characteristic FFN layers(Dai et al., 2021). This reveals a deeper understanding of the interpretability and structure of transformers, enabling more targeted and computationally efficient fine-tuning.
Model Editing. As mentioned in the Introduction, there are three research directions in the mainstream model editing methods, categorized by the location of new knowledge storage. Fine-tuning stands out straightforward and intuitive approach, which updates knowledge by updating the original model parameters. Both ROME(Meng et al., 2022a) and (MMIT(Meng et al., 2022b) exhibit enhancements derived from this perspective. This methodology preserves the original parameters and architecture of the model; however, it frequently precipitates catastrophic forgetting. To mitigate this issue, some researchers have recommended strategies that integrate external space into the model. For example, GRACE (Hartvigsen et al., 2024) adopts novel approach by creating codebooks that store modifiable samples and employs the Euclidean distance metric to navigate the selection of specific parameter trajectories during the inference phase. Alternatively, SERAC(Mitchell et al., 2022) utilizes cache memory units to chronicle information relevant to model editing and executes
3

---

Published as a conference paper at COLM 2024
these modifications by deploying classification models for retrieval tasks. Nonetheless, these strategies necessitate the incorporation of additional storage components, which may be limitation. Meanwhile, some researches are investigating approaches that involve adapting the model's architecture by integrating additional neurons. An example of this approach is T-Palcher(Huang et al., 2023b), which introduces new neural units for each token identified as erroneous, aiming to assimilate newly edited knowledge through an end-to-end learning process. While 1-Patcher reduces the need for extra memory space and ensure the success rate of editing, it can result in an exponential increase in parameter space and computational overhead. This issue becomes particularly pronounced in the context of LLMs.
Continual Learning(CL) in LLMs. CL. an essential aspect of machine learning as enables models to adapt to new tasks while retaining performance on previous ones. Recent advancements, as delineated by Lin et al. (2022), introduce an innovative paradigm known as continual model refinement (CMR). This paradigm is designed to efficiently rectify pre- dictive inaccuracies encountered in out-of-distribution data streams, while simultaneously circumventing the peril of catastrophic forgetting. The notion of sequential model editing (SME), as expounded by T-Patcher (Huang et al., 2023b) and further developed in the work of GRACE CRACE(Hartvigsen et al., 2024), can be considered specialized instantiation of CL. This approach entails the processing of individual data samples in a sequential manner. However, it grapples with the same challenge of catastrophic forgetting that is characteristic of broader CL approaches.
3 Methodology
3.1 Problem Formulation
Define the model fmodel as the one obtained by training on the data Dirain, and we denote the subset of Drrain where fmade! makes exact predictions as Dipe- Suppose we have model editing task using the data Dati = d2.,dd,..,,d,)) dn]. where each d; represents sample containing input and output information (x)x) x; is the input be edited, and y; is the desired output. x is semantically equivalent rewrite of x;, and the corresponding label for both x; and x is yi. We denote the sequential editing process as the time step E [I,n], where the data required for each time step corresponds to d, in Dedit Meanwhile, we define f; the model obtained after editing at the time step f. Based on the aforementioned settings, three properties are used to evaluate the effectiveness of the sequential model editing method.
Property 1. Reliability: The edited model ft obtained at time step t, with the set of all samples up to time step denoted as [dy..., d:), needs to meet the following expectations:
fxx)) yt, f]
(1)
Property 2. Locality: For non-edited samples Diocr ft needs to ensure that they remain unaffected:
f((x) fmodet (x), E Dioc
(2)
Property 3. Generality: f, needs to ensure that the rewritten sample x also receives correct response after editing.:
f(x+) = f,, E [1,
3.2 Customized Expert Networks
Inspired by the Mixture of Experts (MoE) architecture(Shazeer et al., 2017; Fedus et al., 2022), we propose to incorporate series of expert networks for each incoming edit sample in a sequential editing setting. During inference, different requests will be routed to the corresponding experts to ensure optimal performance.
All transformer-based language models (LMs) are constructed using stacked transformer blocks. Assuming we editing the 1th layer of the LLM,, and that the FFN part of /th layer
4

---

Published as a conference paper at COLM 2024
has two components, Wap and Waoon- We consider Woown as lightweight expert network, with the reason provided in the ablation studies section. We train a corresponding W for each sample that needs to be edited. During training, we freeze all the other weights of the [model and activate only Wdown- As shown Stage 1 of the left side of Fig. after each training of Wt is completed, the weights are stored in the local database, and will be used during inference when subsequent activation occurs. Since our expert networks are trained with sample-level granularity, each expert responsible for handling a single edit sample. This ensures that there is no interference between different samples and prevents the forgetting of multiple edits.
down
doya
3.3 Scalable Indexing Neurons
After training multiple customized expert networks, it is necessary activate a specific expert during the inference phase to achieve the desired output. This is not standard clas- sification task where sample is assigned to specific expert. Due to the streaming nature of sequential editing, using a fixed neuron for training will inevitably lead to catastrophic forgetting. Additionally, utilizing a classification model to assign samples to experts would incur additional time costs. Inspired by T-Patcher, we propose a Scalable Indexing Neurons method, which trains a corresponding neuron for each expert using the same edit sample through specific loss function, and these neurons are incrementally added in sequence as the editing progresses. The newly trained neurons integrated with the existing neurons and utilized during the inference phase. In the following we will provide detailed description of how the neurons are employed during the training, merging, and inference phases.
Training Indexing Neurons. The settings are identical to those used for training a cus- tomized expert network with editing at /th layer of the LLM. We introduce a new network structure Wnewron which is placed after Wup. This network consists of single output neuron, implying that the parameter size of Wnewron is 1 X h, where h represents the dimension of FFN hidden layer. And we employ sigmoid activation function for the output. The completed calculation is as follows:
at (egmooid(neerrrg (xp(x11))
(4)
where x is the representation of the last token in x; obtained after the attention layer of the /th layer.
We determine whether sample needs to be edited based on the activation value Samples that require editing, or those that are semantically consistent with these samples, typically exhibit larger activation values, whereas non-edited samples show smaller activation values. To achieve the aforementioned objectives, we design the loss function from three aspects to control the activation value a. For Reliability and Generality, the effect is ensured by lactivate loss, which is calculated as follows:
lactioute = expp at)
(5)
where at represents the activation value that is required at the current time step t edit the sample.
Idisactioute loss and margin loss are used ensure Locality. We consider all the samples that need to be edited before the current time step as negative examples a; i E [1, I 1]. Here, we define the following two loss functions based on the activation values of these negative samples.
1
ddisactivate
Σ
exp(a; +
(6)
1
margin
Σ
exp(a: (a:
B)
(7)
5

---

Published as a conference paper at COLM 2024
where and f are the corresponding hyperparameters, we aim to minimize the activation values corresponding to a,, maximize the value of at, and use a and & as the respective penalty terms. In experiments,
set to 0.7 and & was set to 0.3. Therefore, the complete loss function for training indexing neurons is:
loss nctivaat + m m(disacctiaate + marrgnn)
(8)
m is the hyperparameter trade-off between activation and disactivation.
Note that during the training of Wueuron, only the weights prior the 1-th layer of the LLM are used. Similarly, we only activate Wnearon for gradient updates while freezing all other weights.
Merge Neurons. We denote the weight of the indexing neuron trained at time step t as Wt Weewoon Assuming that we have edited i samples, we should be able to get the weights of t indexing neurons WW W2 neurony ... .Whewroll- During inference, learn to merge these weights together to obtain Wmerge ER We define formally follows:
Wmerge
[W]] neuron
W2
Weeuron ..y Whereon Wi Wneeoonl
Inference Stage. Assume we have model that has already been edited at /th layer. Now, given a new test sample how can we obtain the activation values of the indexing neurons for this sample? When the forward inference reaches the /th layer, we can obtain the answer using the following formula. It should be noted that here we take the last token of the input q as the representation of the sample.
a = Sigmoid(W (Wmerge Wap (2
(10)
where there are I activation values in the a vector, each corresponding to experts, denote af as the activation value the ith position in a9, Here, we determine whether to activate an expert or use the model's original weights by setting a threshold e.
Following the rule described below, as shown in Eq.(1 When the activation value is below the threshold f, the model's 'ss original weights Wdown are used, without activating any experts. Conversely, the maximum activation value a exceeds the threshold, the index corresponding to this maximum value will indx to unique expert, which will be used to replace the original weights, as shown in (12). After determining the structure of the FFN' at the Ith layer, the inference proceeds forward. The complete inference process is illustrated in the right half of Fig.2.
W Expert(dd) Wup if max(a²) > 0, down Wáomn Wup
FFN
(11)
otherwise,
Experi(a9) arg max
(12)
4 Experiments
4.1 Experimental Setup
Datasets. We utilized two benchmark datasets suitable for sequence editing, ZsRE and Hallucination, to evaluate the effectiveness of our method. ZsRE is a comprehensive crestion-answeringg (QA) dataset(Levy et al., 2017). We randomly selected 20,000 entries from this dataset. Specifically, 10,000 entries denoted as D were used for typical super- vised fine-tuning (SFT) based on the LLaMA2 model, while the remaining 10,000 entries, denoted as Djest were reserved for evaluating the performance of the SFT model. Based on the evaluation results, divided the post-test data into two subsets: Decit and D The

---

Published as a conference paper at COLM 2024
subset Dadit consists of samples where the SFT model made incorrect predictions. From this subset, we selected the first 200 and first 1000 samples for subsequent editing opera- tions. Conversely, Dioc comprises samples where the SFT model made correct predictions. Similarly, we selected the first 1000 samples from this subset to evaluate our method's Localization metric. Detailed information is provided in Appendix A. Hallucination is dataset designed to evaluate the performance of model editing methods in of mitigating model hallucinations, as described by Manakul et al. (2023) and (Hartvigsen et al., 2024). Manakul et al. (2023) prompt GPT-3 to generate 238 wikipediastyle biographies for concepts extracted from WikiBio, they then annotate factual accuracy of each sentence, noting which are hallucinations. We followed the processing method described by (Hartvigsen et al., 2024) and created 1392 sequential edits and 592 accurate outputs. Based on the WikiBio dataset, we incorporated 200 samples from OpenWeb Text and trained Llama 7B and 13B respectively. During the editing phase, we selected the first 200 out of the previous 1392 sequential edits for evaluation, denoted as WikiBio-E. Additionally, we used the results from 516 accurate outputs, referred to as Wi combined with 200 samples from OpenWebTest WebText as a test set to evaluate the localization.
Baselines. We compare the proposed method with several mainstream model editing methods. Fine-tuning based methods (including FT and KT, FT refers to the scenario where the fully connected network within a certain transformer block is trainable, while, FT+KL incorporates the Kullback-Leibler (KL) divergence into the loss function, as described in (Mitchell et al., 2021), to mitigate catastrophic forgetting. Memory-based methods, such as SERAC(Mitchell) et al., 2022) and GRACE (Hartvigsen al., 2024), have shown significant promise. GRACE, in particular, is the latest state-of-the-art (SOTA) method based on memory. MEND (Mitchell et al., 2021) aims learn a HyperNetwork using additional training data to transform the gradients obtained through standard fine-tuning, MEMIT, as proposed by (Meng et al., 2022b), a method for directly updating a language model with many memories. In the realm of generative tasks for LLMs, T-Patcher(HHuane et al., 2023b) introduces neurons at the token level. Inspired by this approach, we explore the addition of neurons at the granularity of samples. However, due to the inherent characteristics of T- Patcher, it exhibits substantial training and inference performance constraints. Consequently, we did not experiment with T-Patcher on the Hallucination dataset nor perform thousand edits on the ZsRE dataset. Detailed parameter settings and experimental details for each method are reported in Appendix C.
Metrics. We used Accuracy and Perplexity to evaluate the ZsRE and Hallucination datasets. In ZsRE, the three metrics Reliability, Generality, and Locality refer respectively to whether all edited samples are successful, whether rewrited samples can answer correctly, and whether non-edited samples can answer correctly. In Hallucination, WikiBio-E, WikiBio-A, and OpenWet Web refer to whether the perplexity of the edited samples is reduced and whether the perplexity of the unrelated WikiBio-A and OpenWebTexx Text can be maintained at a lower level, respectively.
4.2 Experimental Results
We conducted 200 and 1000 sequential editing experiments on Llama-2 7B and 13B respec- tively, as shown in Tables 1 and 2, respectively. For the ZsRE dataset. We focus on two aspects: the trade-off between Reliability and Locality, and the average performance across three metrics. For the Hallucination dataset, we examine whether the ppl of WikiBio-E decreases, and whether the ppl of WikiBio-A and Open Tex increases due to halluciation effects. The FT+KL method could not be evaluated on the Hallucination dataset due to it's requirement for additional data.The MEMIT method is not a sequential editing method and performs significantly below baseline levels in hallucination scenarios, so it is not used in our comparison.
On the ZsRE dataset, both at 200 and 1000 edits for long sequences, Tables and 2 demon- strate that our proposed method(SCEN) substantially outperforms the comparison methods average across all three metrics. In terms of Generality and Locality, SCEN achieves a better trade-off on both the Llama2 7B and 13B models, and it also demonstrates im- proved stability. In contrast, the direct fine-tuning(FT) method demonstrated that while
7

---

Published as a conference paper at COLM 2024
ZsRE(ACC
Hallucination(PPL.
Method
Relibilitty
Cenerality
Locality
Ang.
WikiBio-L
WikiBio-A
OpenWeblext
Aug.
Liama2-7B
93.0
48.6
1.25
2.58
5,31
3.04
FT+KI.
92.0
83.8
55.7
77.2
MEND
1.0
2.83
96.7
33.5
19.3
2.33
4.82
8.82
SERAC
89.0
16.2
81.8
62.3
20.6
2.31
4.79
9.23
MEMIT
24.0
39.9
17.0
27.0
T-Patcher
94.0
87.9
62.9.
81.3
GRACE
94.5
38.2
99.9
77.5
3.67
2.31
4.79
3.59
SCEN
100.0
90.0
83,3
91.1
2.93
2.28
4.82
3,34
Llama2-138
FT
93.5
86.7
48.8
76.3
1.10
2.40
5.94
3.15
T-Patcher
92.5
85.3
55.8.
77.9
GRACE
90.0
40.6
100
76.9
2.21
1.98
4.69
2.96
SCEN
99.5
80.7
83.5
87.9
1.57
1.97
4.70
2.75
Table 1: Comparison of SCEN with existing methods. All experimental results results were obtained after 200 edits.
Llama2-7B (ACC t)
Llama2-13B (ACC
Method
Reliability
Generality
Locality
Aug.
Reliability
Generality
Locality
Ang.
FT
92.1
87.3
38.7
72.7
90.8
83.0
34.9
69.6
GRACE
90.0
33.8
99.9
74.5
92.6
38.2
100.0
76.9
SCEN
96.2
80.2
83.8
86.7
98.0
76.1
70.2
81.4
Table 2: Results were obtained by performing 1000 sequential edits on the ZsRE dataset.
Reliability and Generality were maintained, there was significant drop in Locality. To address this limitation, FT+KL was utilized; however, its impact on improving Locality was minimal. The GRACE achieved the best results on Locality but its Generalization was very insufficient. Similarly, SEARAC exhibited an imbalance between Generality and Locality. T-Patcher achieves results close to SCEN on the Reliability and Generality metrics. However due to the addition of neurons for each token which significantly increases the number of parameters and the fact that T-Patcher is an end-to-end approach, these factors collectively contribute to its unsatisfactory performance is not a sequential editing approach, and thus it does not achieve superior results across all three metrics. Overall, propose the dynamic addition of indexing neurons, which can accurately activate experts for long sequence edits without catastrophic forgetting. This method strikes a balance between Generality and Locality without resorting extremes.
the Locality metric. MEMIT
On the Hallucination dataset, it can be seen from Table that FT is an effective method that drastically reduces the perplexity of editing samples. Due its excellent performance on WikiBio-E, FT achieved the best result on the Llama2-7B model, with SCEN ranking second. On the other hand, SCEN achieved the best results on the Llama2-13B model, maintaining a low level of average perplexity across the three test sets. Meanwhile, GRACE was competitive and secured second place. In summary, it can be seen that SCEN also maintained relatively good stability in the hallucination relief task.
4.3 Ablations and Analysis
Ablation Study. We conducted two ablation experiments in SCEN to investigate the contribution of each component. Specifically, we examined the impact of using different positions of weights experts for editing and employing different training loss functions for indexing neurons. The complete experimental results are shown Table.3, where the top three rows illustrate the effect of editing on different positions of weights. Watt indicates that editing is performed in the attention layer, Wup indicates that editing is performed in the upper half of the fully-connected layer following the attention layer, and the corresponding W. Wooon indicates that editing is performed in the lower half the fully- connected layer. For more concise description of Watnn Wup and W Wdown please refer to
8

---

Published as a conference paper at COLM 2024
1011 15
One sample in One neuron
Two sample in One neuron Four sample in One neuron
Figure 3: The Impact of Varying Sample Sizes on the Activation Patterns of Indexing Neurons.
Fig.2. The following three rows in Table.3, please refer to the effect of employing different training loss functions for indexing neurons on the final results. w w/o ddisactioute& &l margiin represents the scenario in which we retain only the activation loss, while w/o margin represents retaining both the activation loss and the disactivate loss. can be observed that in terms of the positioning of expert weights, Wdown achieves the optimal performance in editing, whereas Wathn exhibits the poorest performance. These findings are consistent with the results reported by ROME(Meng et al., 2022a). In terms of loss, the main contribution is attributed to disactioater while the effect of addition of margin also shows improvement.
Llama2-7B (ZsRE)
Reliability Generality
Locality
Aog.
Wattn
90.0
77.3
92.0
86.4
Wap
96.5
83.2
86.7
88.8
Waoum
100.0
90.0
83.3
w/o ddisactionte margin
100
93.8
73.7
89.2
w/r //o margin
100
90.6
82.1
90.9
SCEN (8))
100
90.0
83.3
Table 3: Ablation Experiments of SCEN on the ZsRE Dataset
How many samples can each expert handle? We consider whether the added parameters can be reduced in terms of optimizing storage space and inference time. We conducted three sets of experiments
which an expert and its corresponding indexing neuron were responsible for handling one sample, two samples, and four samples, respectively. In this way the present, after sequential editing 200 entries, we obtained 200, 100, and 50 experts, respectively. With Table.4 shows that the number of added parameters has decreased, and both Reliability and Generalizability metrics have significantly declined. This decline may be attributed each expert learning from multiple diverse training samples, which consequently leads to a significant reduction in the robustness of indexing neurons.
It is evident that single neuron is insuffi- cient to accurately model multiple diverse samples. We also demonstrate the activa- tion of the corresponding indexing neurons for different numbers of samples, as illus- trated in Fig.3. The horizontal coordinates represent the indexing neurons correspond- ing to each edited expert, while the verti- cal coordinates represent the sequentially edited samples. In the left figure, as it is one-to-one sample, clear highlighting can be observed along the diagonal. The middle and right figures illustrate the two-to-one and
Llama2-7B (ZsRE)
Numbers Reliability Generality Locality Experts
One
100
90.0
83.3
200
Two
86.5
72.7
81.7
100
Four 53.0
43.7
84.7
50
Table 4: Performance of Different Number of Compression Strategies on ZsRE

---

Published as a conference paper at COLM 2024
100
100
30
50
50
25
0.6
0.62
0.64
0.66
0.68
0.7
0.6
0.62
0.64
0.66
0.68
0.7
0.62
0.64
0.66
0.68
0.7
Reliability
Generality
Locality
Figure 4: Impact Value Variations on the Performance of the ZsRE Dataset
four-to-one scenarios, respectively, showing that the activation state progresses in stepwise manner.
Activation parameter analysis. We investigated the effect of the size of the activation threshold value on the final Reliability, Generality and Locality by dividing into 11 hyperparameters ranging from 0.60 to 0.70 in increments of 0.01.The experiments were conducted on the ZsRE dataset, as illustrated in Fig.4. can be observed that lower activation values favor Reliability and Generality, but Locality suffers more. The trend indicates that Reliability and Generality decrease with increasing 6 while Locality increases with increasing 0 Memory-based model editing approaches by the need to balance Generality and Locality. SCEN minimizes the situations where hyperparameters significantly impact the metrics. As shown in Fig.4, our method exhibits relatively small fluctuations across all three metrics within the first half of the thresholds.
frequently challenged
Which layer better for editing? We examined the performance of the even-numbered layers of Llama2-7B separately, as shown in Fig.5. It is evident that in the lower-level transformer blocks of Llamm2-7B, editing has minimal effect, thereby preserving the original model results and maintaining high level of Locality. In contrast, in the higher-level transformer blocks, the impact of editing is pronounced. Starting from the 16th layer onward, Reliability remains at a high level, with minimal fluctuations in Generality and Locality. Consequently, we further validate that the higher-level transformer blocks of LM, based on the transformer architecture, contain some factual knowledge, and editing of these layers has a significant effect.
100
Reliabil ity Generality Locality
60
40
20
24
Figure 5: Results of Editing with SCEN at Even-Numbered Layers of Llama 2-7B
Conclusions
In this paper, we present SCEN, an innovative two-stage continuous training paradigm for model editing. This approach involves training a customized lightweight expert network for each sample to be edited. Additionally, we have developed scalable dynamic neuron indexing mechanism that efficiently activates the corresponding experts. Our experimental results indicate that SCEN consistently surpasses the most latest state-of-the-ate methods in model editing. Furthermore, we delve into the interpretability of LLMs with a focus on transformer-based architectures, substantiating that factual knowledge is predominantly stored in the latter layers of the language model. Looking ahead, we plan to explore advanced weight compression techniques aimed at reducing the number of expert networks and indexing neurons. This effort should lead to faster processing and less storage use, making model editing more practical and efficient.
10

---

Published as a conference paper at COLM 2024
References
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arv preprint arXio:2104.086%6 2021.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. Xiv preprint ar axXt::104.00816, 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion
parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint ar Xiv::2311.05232, 2023a.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher: One mistake worth one neuron. arxiv preprint ar arXie:2301.0978; 2023b.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8: 423-438, 2020.
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKe- own, and Tatsunori B Hashimoto. When do pre-training biases propagate to downstream tasks? a case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3206-3219, 2023.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. Xiv preprint arXic:1706.04115 2017.
Bill Yuchen Lin, Sida Wang, Xi Victoria Lin, Robin Jia, Lin Xiao, Xiang Ren, and Wen-tau Yih. On continual model refinement in out-of-distribution data streams. arXiv preprint XXtx:2205.02014 2022.
Ruibo Liu, Guoqing Zheng, Shashank Gupta, Radhika Gaonkar, Chongyang Gao, Soroush Vosoughi, Milad Shokouhi, and Ahmed Hassan Awadallah. Knowledge infused decoding. arXiv preprint ar XxX::2204.03084 2022.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arviv preprint arXiv:2303.0889) 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. ar Xiv preprint ar xrii:2210000729,, 2022b.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint axXiv:2110.11309, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022.
11

---

Published as a conference paper at COLM 2024
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXio preprint (rXXi:1909.01066, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08110 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. arXiv preprint Xrxo:1701..6533 2017.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. aiv preprint ar Xxie:1909.08053 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al. Knowledge editing for large language models: A survey, ariiv preprint xrii:22310.16218, 2023.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arv preprint arXiv:2305. Kit::2305.1377,, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ar arxiv preprint or Xrxx:22303.1823, 2023.
A ZsRE dataset details
For ZsRE, we first trained the base model using 10k data get an SFT model. The format of the data is shown in Table 5. We added 
 and 
 to the query, which is to simulate the chat model training format of LLM. The format in editing and result evaluation is consistent with the format of the training data.W. additionally sampled 10k samples Drest for constructing edit samples Delit and samples Dlsc for evaluating localization. Dedlt is the sample where the SFT model answered incorrectly in Diest In the experiments of this thesis, we took the first 200 and the first 1,000 samples of Dedit respectively, which were used to evaluate the effect of editing, meanwhile, we used the first 3 rewrites of these samples test set to evaluate the generalizability.
Prompts
Answer
Whose direction is From Dusk till Dawn?

Robert Rodriguez
> September 1985
 World War I


The date of birth for Sinoti Sinoti is what?
 
In which war did Paul Lacombe de La Tour fight?|/INSI
Table Data formats for training SFT models in ZsRE
12

---

Published as a conference paper at COLM 2024
B Hallucination dataset details
GRACE constructed dataset via SelfCheckGPI[Mannkull et al., 2023) to evaluate the effect of model editing methods reduce model illusions, and called this dataset Hallucination. Manakul et al. (2023) utilizes GPT3 to extract concepts from WikiBio to generate correspond- ing biographies. For each sentence generated, they labeled which sentences were factual and which were hallucinatory. Follow the idea of GRACE to edit the highly inaccurate sentences and replace them with real Wikipedia sentences. The highly accurate samples are utilized to judge whether the model still maintains relatively low level of perplexity these samples after editing. Through the above processing, a total of 1,392 editing samples and 516 sentences that have been accurate have been generated. Compared with the dataset used in the previous model editing, Hallucination has a longer text length, which is more in line with the data situation in the real scenario. In this paper, used the first 200 of the 1392 edited samples for our experiments, in the two Llama2-7B B/1BB models. Partial data are shown in Table 6.
WikiBio-W:
This Wikipedia passage about akila dananjaya. Akila Dananjaya (born 2 August 1995) This is Wikipedia passage about wilhelm windelband. Wilhelm Windelband (15 March 1848
WikiBio-A:
This is a Wikipedia passage about rick mahler. Rick Mahler (born Richard Alan Mahler on April
Open SpenWeeTeett
A magazine supplement with an image of Adolf Hitler and the title "The Unreadable Book' is pictured For today's post, I'd like to take look at Californias voter initiative to legalize pot. the measure
Table 6: Data format for continued training in Hallucination
C Experiment details
Detailed parameter settings using the comparison method used are described in detail below:
For FT+KL, additional data is needed to be used with the KL. loss. In the ZsRE data, we should have a vector indexing model(all -mpnet-base-v2) for each edit sample from the 5 most similar samples found in the training data of SFT. The training step for each edit sample 10 and the learning rate is 0.0001.
For the MEND, additional data is needed to train an editor, and to avoid any test set leakage, in the ZsRE data, we use 2000 entries from the data used train the SFT to train the editor. In the Hallucination dataset, we use the last 400 data out of 1352 entries to train the editor.
For the SEARAC method, the method contains a total of three models, of which the classifi- cation model we use is distilbert-base cased, and the threshold we set is 0.6.
For the GRACE method, is set 3.0, the number of iterations is set to 1.0, and the adapter weights are taken to be randomly initialized.
10, the learning rate
For MEMIT, we only experimented on ZsRE, the fact token used makes the last token of the entity, and the positions to be edited are the 20th and 22nd transofmerblock.
13


Reviews:
[{"review_id": "Wg9NnVO1eP", "reviewer": "Reviewer_67Nb", "summary": "This paper proposes a two-stage editing method based on expert networks, called SCEN, in the “sequential” setting that a sequence of edit requests is presented. \n\nOverall, the proposed two-stage method, the idea of incorporating the FNN value parameters as expert networks with the indexing neurons, is interesting and novel, showing the effectiveness in the experiment settings. A work is a reasonable extension of T-patcher, towards maintaining multiple knowledge neurons and experts. \n However, the papers are not very convincing on methods and experiment settings as follows: it is not convincing why two-stage methods are required, so comparing to T-patcher, architectural advantages are not clear. Another problem is that the proposed method seems not to be scalable well, as the number of additional indexing and weight parameters is just increasing monotonically for larger scale of edits. A simple extension of T-patcher which is not in two-stage manner needs to be compared. Additionally, the paper needs to be rewritten, particularly because the method of the first stage is not very clear (how to train FNN’s down projection weight). \nNevertheless, given its positive aspects of the paper, I am currently leaning to the acceptance, while the current decision is likely changed to lower scores.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "# Weaknesses:\n>1.For each edit...\n\nOur original motivation is to ensure that the edited mode work is not influenced by the non-edited data, a fact supported by our experiments. We must acknowledge that while this design aids in sequential editing and expansion, it also consumes more CPU memory space.This opens up paths for future work, like expert weight fusion. Additionally, you can find experiments about GPU usage in Q2 of the rebuttal for reviewer [4Ci8].\n\n>2.In the two-stage...\n\nThe two-stage training is indeed crucial for preserving the original capabilities of the model. However, due to word limitations, you can find specific reasons and some experimental conclusions in the response to Q1 of reviewer [amby].\n\n>3.In the experiments...\n\nwe have now implemented T-Patcher's method on both the llama2 7b and 13B models on the ZsRE datasets for 200 new edits. You can find detailed experimental results in the response to Q2 from reviewer [5GUJ], due to word count limitations.\n\nThe results indicate that our method, SCEN, maintains a good balance between Reliability, Generality, and especially Locality, compared to T-patcher. Additionally, a comparative experiment about neuron activation incorrectly can be found in Q3 from reviewer [5GUJ].\n\n>4.An important part...\n\nIn stage 1 of SCEN, the network parameters are frozen, while only the $W_{down}$ weight matrix of the FFN layer in specific layers (the 20th layer as chosen in this paper) is fine-tuned and stored as an expert. In other words, the stage 1 is the SFT that freezes most of the network parameters.\n\n# Question:\n>1.Why counterfact...\n\n>2.The method is...\n\nOur method primarily focus on sequence editing, so the comparisons are mainly with previous sequence editing approaches. We didn't specifically measure metrics against counterfact datasets for methods like ROME and MEMIT, as they aren't the sequence editing. To make comparisons more straightforward, we usually test using datasets from previous work (such as GRACE, T-patcher). Therefore, metrics from multi-hop datasets might not provide as clear comparisons.\n\n\n>3.In experiment results...\n\nOur method excels at scalable sequence editing. To illustrate, during the inference process, each edit only adds 11K to the parameters. For the Llama2-7B model, even after 10,000 continuous edits, the parameter increase is just 1.57%.\n\nMoreover, when dealing with 100M or more edits, we believe that fine-tuning the model is a more sensible approach than editing the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe proposed two-stage editing incorporating with the expert networks is interesting and novel, while the value parameters of FNN are considered as parameters of each expert. \n2.\tExperiment results show that the proposed methods are promising, comparing to other various prior methods, using the llama2 models.", "reasons_to_reject": "1.\tFor each edit request, the proposed editing methods monotonically create a new indexing neuron with a weight vector, and need to keep its own FNN’s value parameters. As a result, if N number of sequential edits is provided, we need to keep N indexing neurons associated with their weight vectors as well as N FNN values parameters separately. It is largely questionable whether increasing the number of parameters is scalable in the practical situation. Even edits where their knowledge and topics are very similar will have their own separate indexing neurons and FNN value parameters, causing the method inefficient.  \n2.\tIn the two-stage method, the actual editing process is performed at the first stage, while an indexing neuron in the second stage just plays of an “indexing role” which turns on or off, depending on the input query. It is not clear why these two stages are necessary. Why the methods are not designed in a unified manner (without two-stages), like T-patcher? \n3.\tIn the experiments, a simple extension of T-patcher not in the “two”-stage manner needs to be compared. \n4.\tAn important part is how to actually adjust the parameters of an expert, at the first-stage. But, in Section 3.2, this part is not very clear; how to train W_down^t (down projection weight)? which layers have their own expert parameters?", "questions_to_authors": "Please see weaknesses. Below are further comments. \n1)\tWhy counterfact dataset is not used, while it is used in most existing methods? \n2)\tThe method is expanded to other multi-hop reasoning setting, like effectively working on Mquake dataset? \n3)\tIn experiment results, only cases of 200 and 1000 edits are evaluated. How long an editing sequence where the proposed method is scalable well? Assuming that 1M or 100M edits are provided, what are the memory and inference costs in the proposed methods? In such large-scale edit, even applying Eq. (12) that selects the best expert requires non-trivial inference time.", "extra_scores": {}}, {"review_id": "7LMqZJbhyN", "reviewer": "Reviewer_amby", "summary": "The paper presents SCEN, a two-stage model editing approach for large language models that improves upon outdated knowledge and hallucinations. SCEN stands out for its custom expert networks per knowledge piece and indexing neurons that control expert activation, leading to state-of-the-art results in question-answering and hallucination reduction datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1.While you have mentioned T-Patcher...\n\nSCEN and T-patcher are both innovative approaches in the field of sequence model editing, with both introducing the concept of extensibility. But the work mechanisms of these two methods are totally different. \n\nT-patcher integrates newly neurons into model for two purposes: learn the edited data and index to determine whether the neuron is activated. The greatest distinction between SCEN and T-patcher is the two-stage training paradigm proposed by SCEN. Stage 1 focuses on learning to edit data and save weights as an expert in database, the indexing neurons in stage 2 are only used to select the corresponding experts. \n\nThe existence of individual experts can ensure that new knowledge is learned with near-perfect,  while indexing neurons distinguish different experts and ensure that non-edit data is not affected. Additionally indexing neurons are not involved in the decoding process, thereby ensuring they do not disrupt the model's performance.\n\nMore detailed comparative experiments can be found in Q2 and Q3 of the rebuttal for reviewer [5GUJ].\n> 2.Consider including an...\n\nTo tackle this issue, we follow the findings of ROME[1] and employ FFN's $W_{down}$ as the expert. Still, for ensure the rigor of the conclusion, we also carried out additional tests, applying the llama2-7B model to the ZsRE datasets for 200 edits. The results are shown in the table below:\n\n| |Reliability|Generality|Locality|Avg|\n|--|--|--|--|--|\n|$W_{attn}$|90.0|77.3|92.0|86.4|\n|$W_{up}$|96.5|83.2|86.7| 88.8 |\n|$W_{down}$|100|90.0|83.3|91.1|\n\nAccording to the table, we can see that using $W_{down}$ as the expert has the best results, while $W_{attn}$ has the lowest score. This  conclusion is consistent with ROME. \n\n[1]https://arxiv.org/abs/2202.05262\n> 3.In Tables 1 and 2...\n\nRegarding the performance discrepancies observed in Tables 1&2 between Llama2-13B and 7B, we'd like to offer the following clarification:\n\nThe evaluation datasets for Llama2-7B and 13B were indeed different, as we selected questions that each respective model originally failed to answer correctly. Given that Llama2-13B is a more capable model, the questions it initially failed to answer correctly tend to be more challenging. This is reflected in the evaluation data, which, for Llama2-13B, comprises more difficult questions. So this situation occurs.\n\nAdditionally, we focus on compare different methods within the same model using identical data to ensure a fair comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper introduces a novel method for model editing that significantly improves upon existing techniques.\n\n* The two-stage continuous training paradigm is promising, combining the benefits of specialized expert networks with a dynamic neuron indexing mechanism to apply edits selectively.", "reasons_to_reject": "* While you have mentioned T-Patcher, it is essential to provide a comprehensive overview of these related works and highlight the differences between your paper and theirs.\n\n* Consider including an experimental discussion on the effects of fine-tuning the attention layer or the $W_{up}$​ weight. This exploration will provide valuable insights into the performance improvements achieved by these modifications.\n\n* In Tables 1 and 2, the model performance based on Llama2-13B as the baseline is generally much worse than that based on Llama2-7B. The authors could explain or reasonably speculate on the reasons behind this.", "questions_to_authors": "Please refer to the section titled 'Reasons for Reject' for further details.", "extra_scores": {}}, {"review_id": "YIAxlrK5Zz", "reviewer": "Reviewer_5GUJ", "summary": "The paper proposes a method for continual model editing called SCEN. Like GRACE, it only modifies the l-th layer (a hyperparameter), specifically the down projection matrix, for each edit t which is then stored in a database (an \"expert\"); after T edits, at test time, the model must decide which expert to use (or none). GRACE does this by codebook maintenance, which requires no further training. SCEN proposes to do this by training an expert selector. There are presumably many ways to parameterize the selector, but the paper loosely follows the training scheme in T-Patcher to train a scalar projection for each edit to activate for that edit (by contrastive loss), then at test time uses the argmax of the projection for the given query if the value is above some threshold. \n\nExperiments are conducted on ZsRE (QA) and GRACE's version of hallucination task (reducing perplexity on edits while maintaining it on non-edits), using Llama-2. SCEN performs better than GRACE on edit success and generalization, but worse on locality; on average SCEN performs better.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">1.Somewhat limited novelty...\n\nI'd like to highlight the distinct contributions of our two-stage editing approach.\n\nSCEN offers a targeted and efficient way to correct a finite set of erroneous knowledge, which is crucial for preserving the accuracy and relevance of models in a real online environment. It ensures that, as the number of edits increases, the original model's capabilities do not degrade.\n\n>2.No comparison with T-Patcher... \n\nIndeed, it was an oversight on our part not to include a direct comparison with T-Patcher initially. To rectify this, we have now implemented T-Patcher's method on both the llama2 7b and 13B models on the ZsRE datasets for 200 new edits. The table below presents the results:\n\n| |Reliability|Generality|Locality|Avg|\n|--|--|--|--|--|\n|T-Patcher-7B|94.0|87.0|62.9|81.3|\n|SCEN-7B|100|90.0|83.3|91.1|\n|T-Patcher-13B|92.5|85.3|55.8|77.9|\n|SCEN-13B|99.5|80.7|83.5|87.9|\n\nThe results indicate that our method, SCEN, maintains a good balance between Reliability, Generality, and especially Locality, compared to T-patcher. \n\n>3.More about T-Patcher...\n\nOur SCEN method is like T-Patcher because both keep editing knowledge in network weights, but they work in different ways. In SCEN, the experts are stored in memory and only the activated expert is loaded onto the GPU, which optimizes memory usage and ensures precise expert activation. In contrast, T-Patcher's neurons, which are integrated into the model, are always active during inference, leading to increased GPU consumption and potential inaccuracies in neuron activation. You can find the experiment for GPU usage from 4Ci8. And for cases where neurons are activated incorrectly, see the table below:\n| | False Activation(%) |Correct Response(%)|\n|--|--|--|\n| T-patcher | 71 | 48.5|\n| SCEN | 7.5 | 93.5 |\n\nThe results indicate that SCEN has a higher acc rate and a lower rate of false activations for non-edit data.\n\n>4.Limited tasks...\n\nHistorically, models like GRACE were tested on Encoder-based frameworks such as BERT, which are suited for classification tasks like SCOTUS for performance evaluation. However, with the emergence of LLM, the research focus has shifted towards these Decoder-only models. Our approach is primarily based on LLaMA, and therefore does't incorporate traditional classification tasks.\n\n>5.Some writing issues.\n\nWe apologize for any trouble caused by the writing issus and are dedicated to carefully revising and correcting them to ensure a more natural final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A reasonable modification of GRACE (vector codebook -> down projections, Euclidean distance -> trained expert selector). The method is also simple. \n- Better performing than GRACE in the average over reliability/generalization/locality.", "reasons_to_reject": "- Somewhat limited novelty, which is a general problem with many variations of model editing methods (slight differences in parameterization/activation approach, which are often almost certainly over-optimized for small-scale models and datasets).\n- No comparison with T-Patcher, which doesn't make sense to me. It has the same continual model editing setup, and this paper is inspired by it.\n- More about T-Patcher: it bakes in the activation for the right expert into training. So even though it doesn't explicitly predict the expert at inference time (i.e., the neurons are not stored in a database but part of the model), it's kind of doing the same thing? T-Patcher is supposed to be also scalable (i.e., the added neurons are a minor portion of the model for thousands of edits). \n- Limited tasks: why only ZsRE and Hallucination, when the main baseline GRACE also includes SCOTUS? \n- Some writing issues (unnatural)", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mVAbmbV2DX", "reviewer": "Reviewer_4Ci8", "summary": "This paper presents an LLM editing method using customized expert networks and indexing neurons.  Experiments have been conducted to demonstrate the effecitveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable reviews, we hope the following comments could address your concerns\n> 1. Some ablation studies are expected to analyze the effectiveness of different component of the proposed method, such as the different losses in Eq.(8).\n\nIndeed, conducting ablation studies is a crucial step in understanding the impact of each individual component within our proposed method. In light of the first question you posed, we have carried out additional experiments using the llama7B model on the ZsRE datasets for 200 edits. The results of these experiments are presented in the table below:\n| | | ZsRE Llama2-7B| | | |\n|--|--|:--:|--|--|--|\n| | | Reliability|Generality|Locality|Avg|\n| 1 | w/o $l_{disactivate}$ & $l_{margin}$ | 100 | 93.8 | 73.7|89.2|\n| 2 | w/o $l_{margin}$ | 100 | 90.6 | 82.1| 90.9|\n| 3 | SCEN loss (Eq. 8)  | 100 | 90.0 | 83.3 | 91.1 |\n\nWe have drawn the following conclusions: By comparing w/o $l_{margin}$ and SCEN, we observe that the main contribution of our proposed loss comes from $l_{disactivate}$ , while adding $l_{margin}$ still improves the method’s performance. \n> 2. How about the space and time efficiency of the proposed method since a separate expert network is necessary for each edited knowledge?\n\nWe train a customized expert network for each edited knowledge and store the weights in memory. During the inference process, only a single active expert weights is loaded into the GPU's VRAM , ensuring that the VRAM utilization remains controlled. The primary trade-off involves the time required to transfer the weights from memory to VRAM, which takes about 18ms. \n\nCertainly, to provide a more intuitive comparison between  SCEN and T-patcher, which is also designed for editing sequence models, we performed a analysis focused on the differences in VRAM consumption and time overhead. This analysis was based on 1000 editing operations on Llama-7B using the A800（80G）GPU, as illustrated in the table below：\n|  | Ex-GPU(M) | In Edits(s) | Out Edits(s) |\n|--|--|--|--|\n| T-patcher | 121.664 | 127.7 | 126.76 |\n| SCEN  | 20.996| 303.5  | 125.32 |\n\nIn summary, T-patcher is less time-consuming for edited datasets but comes with a higher GPU memory cost. SCEN is more memory-efficient but takes longer to process edited datasets. Both methods have comparable performance for unedited datasets. The conclusion is consistent with the analysis above.\n\nAll of the above modifications will be incorporated into the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using  customized expert networks for model editing seems novel to me.", "reasons_to_reject": "The evaluation part of this paper can be further improved.\n1) Some ablation studies are expected to analyze the effectiveness of different component of the proposed method, such as the different losses in Eq.(8).\n2) How about the space and time efficiency of the proposed method since a separate expert network is necessary for each edited knowledge?", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SCEN, a two-stage editing method for large language models that uses expert networks and indexing neurons to handle sequential edit requests. This approach enhance model performance by incorporating fine-tuned value parameters within expert networks, showing significant improvements in question-answering and hallucination reduction tasks.
Reviewers appreciate the novelty of the SCEN framework and the strong experimental results it showed. There was complaints about missing comparison to the very relavent prior work, which is addressed during the rebuttal. There are some minor concerns on limited task scope, and clarity issues in the writing.
Decision: Accept