Transformer-Patcher: One Mistake Worth One Neuron
Authors: Zeyu Huang
Yikang Shen
Xiaofeng Zhang
Jie Zhou
Wenge Rong
Zhang Xiong
Venue: ICLR 2023 poster (2023)
Keywords: Sequential Model Editing

Abstract: Large Transformer-based Pretrained Language Models (PLMs) dominate almost all Natural Language Processing (NLP) tasks. Nevertheless, they still make mistakes from time to time. For a model deployed in an industrial environment, fixing these mistakes quickly and robustly is vital to improve user experiences. Previous works formalize such problems as Model Editing (ME) and mostly focus on fixing one mistake. However, the one-mistake-fixing scenario is not an accurate abstraction of the real-world challenge. In the deployment of AI services, there are ever-emerging mistakes, and the same mistake may recur if not corrected in time. Thus a preferable solution is to rectify the mistakes as soon as they appear nonstop. Therefore, we extend the existing ME into the Sequential Model Editing (SME) to help develop more practical editing methods. Our study shows that current ME methods either fail to make a sequence of edits or to remember previous edits. We then introduce Transformer-Patcher, a novel model editor that can shift the behavior of transformer-based models by simply adding and training a few neurons in the last Feed-Forward Network layer. Experimental results on both classification and generation tasks show that Transformer-Patcher can successively correct up to thousands of errors (Reliability) and generalize to their equivalent inputs (Generality) while retaining the model’s accuracy on irrelevant inputs (Locality). Our method outperforms previous fine-tuning and HyperNetwork-based methods and achieves state-of-the-art performance for Sequential Model Editing (SME).

Published as a conference paper at ICLR 2023
TRANSFORMER-PATCHER.
ONE MISTAKE WORTH ONE NEURON
Zeyu Huang!.? Yikang Shen', Xiaofeng Zhang'.2, Jic Zhou', Wenge Rong'.,, Zhang Xiong'. State Key Laboratory of Software Development Environment, Beihang University, China French Engineer School, Beihang University, China School of Computer Science and Engineering, Beihang University, China
MMila, University of Montreal, Canada, WeChat AI, Tencent Inc, China
(zeroy, azeroy.huang, huang, yikang shn (eafl/mmalll. com, wi omzhou& wew.hhouuueeeen....om encent. xiiaofeng_,, w. rong, xiongz (ounggzeuuuaa.....m.ao @buaa. edu. cn
com
ABSTRACT
Large Transformer-bait based Pretrained Language Models (PLMs) dominate almost all Natural Language Processing (NLP) tasks. Nevertheless, they still make mis- takes from time to time. For model deployed in industrial environment, fixing these mistakes quickly and robustly is vital to improve user experiences. Previ- ous works formalize such problems as Model Editing (ME) and mostly focus on fixing one mistake. However, the one-mistake-fixinn scenario is not accurate abstraction of the real-world challenge. are ever-emerging mistakes, and the same mistake may recur if not corrected in time. Thus a preferable solution is to rectify the mistakes as soon as they appear nonstop. Therefore, (SME) to help develop more practical editing methods. Our study shows that most current ME methods could yield unsatisfying results in this scenario. We then introduce ior of transformer-brat based models by simply adding and training a few neurons in the last Feed-Forwart Network layer. Experimental results on both classification and generation tasks show that Transformer- Patcher can successively correct up to thousands of errors (Reliability) and generalize to their equivalent inputs (Gen- erality) while retaining the model's accuracy on irrelevant inputs (Locality). Our method outperforms previous fine-tuning and HyperNetworkk-bsce methods and achieves state-of-th-aa performance for Sequential Model Editing (SME). The code is available https: htt///ggtttt//tt// com Zero YuHuang/ Traans/rrassaaai form er-Patches
the deployment of Al services, there
extend the existing ME into Sequential Model Editing
Seansformer-Pacchee a novel model editor that can shift the behav-
INTRODUCTION
Transformer- based models, particularly large Pretrained Language Models (PLMs) (Devlin al., 2019; Brown et al., 2020) have become the backbone model of modern Natural Language Pro- cessing (NLP) and have enabled promising results in various downstream tasks (Lv al., 2019; Budzianowski & Vulic, 2019; Ramnath et al., 2020). However, PLMs still produce undesirable out- puts occasionally (Zhao et al., 2019; Basta al., 2021). The cost of such mistakes is non-negligible. For example, a mistaken automatic translation result could get person arrested (Hem, 2018). 8). One of the most usual expedients was using a manual cache (e.g.. lookup table) to overrule these problem- atic predictions (Sinitsin et al., 2020). Though convenient and straightforward, it lacks robustness and generality because it could be disabled by the slightest change in the input, such as paraphras- ing in natural language. On the other hand, supplemented with problematic examples. While superior in performance, is computationally and temporally expensive to re-train large PLMs with billions
also re-train the model on the original dataset
trillions of parameters.
Previous research formalized such problems as Model Editing (ME) and proposed various methods to intervene model's behavior other examples. Some straightly finetune the model on the example and used a constraint loss to maintain the model's 's overall performance (Zhu al., 2020: Sotoudeh & Thakur, 2021). Some edit
specific example while preventing the model from forgetting

---

Published as a conference paper at ICLR 2023
the model through HyperNetwork, which regards the model and the false predicted example as inputs and produced a weight update for the model's parameters (Cao et al., 2021; Sinitsin et al., 2020: Mitchell et al., 2022a). Despite their impressive progress, they mostly focus one-step edit- ing (fixing one mistake), which is not applicable to practical situations. Because models deployed for real-world applications pop up repeatedly and bother different users. In addition, as illustrated in Figure I, once a wrong an- swer appears in an online question-ansoe answering (QA) model, leaving it unfixed and waiting for future corrections could mislead more people. Therefore, an ideal model editor should provide continuous and promptly fixing of newly emerged mistakes in an effective and efficient manner.
expected to face different errors ceaselessly. And the same error may
User
Correction
The slower the fixing, the more users are bothered
User bothered Error
A QA model online
CO
Wrong Answer
Instant correction improves user experience
Instant
Correction
Figure 1:
Once error occurs in a QA model online, it could bother many users contacting the model if not fixed in time. Instant correction is a superior choice to improve the user experience, motivating to propose Sequential Model Editing problem.
Thus we extend the ME task into the sequential setting and formalize it Sequential Model Edit- ing (SME) task, which requires a model editor to fix a series of mistakes as soon as they appear. The desiderata of a qualified sequential model editor are three properties (Section 3). For each editing, the post-edit model should be of 1) Reliability: make the desirable output given the input: 2) Gener- ality: generalize over other equivalent inputs; 3) Locality: retain its accuracy over irrelevant inputs. We then propose standard SME experiment pipeline that is compatible with different tasks and five evaluation metrics to evaluate the three properties. Experiments show that most existing model editors could fail to generalize to the sequential editing scenario. Fine-tuning-basee methods are vulnerable to forgetting previous edits. HyperNetwork -basd editors are strongly coupled with the initial model that they trained with, thus failing to edit the model after several steps (Section 5).
To handle SME, we introduce Transformer-Patcher. Unlike previous methods, CaassformrrPaatcherr retains all original parameters to prevent harming the model's overall performance. It only adds a handful of trainable neurons (patches) to the last Feed-Forward Network (FFN) layer to revise the model's behavior on the problematic input and achieve low editing cost. Furthermore, train the patch to only respond to specific inputs with the proposed activation loss and memory loss. Experimental results on fact-checking (classification) and question answering (auto-regressive generation) indicated that Transformer- Patcher could rectify a series of mistakes (up to thousands) while almost perfectly retaining the model's overall performance.
The main contributions of this work are twofold: 1) We formally propose a sequential model editing task, as well as its standard experiment pipeline and evaluation metrics. 2) We introduce Transformer- Pransfrrmer-Patcher,, a simple yet effective model editor to revise transformer-baut based PLMs, achiev- ing state-of-the-arr SME performance.
RELATED WORKS
Ford-forward Network Both the Transformer encoder and decoder contain the Feed-Forward Network (FFN). Recent works (Geva et al., 2021; Dai et al., 2022) analogously observed that FFN operates as key-value neural memories (Sukhbuatar et al., 2015). They regarded the input of FFN as query, the first layer as keys, and the second as values. Thus the intermediate hidden dimension of FFN can be interpreted as the number of memories in the layer, and the intermediate hidden state is a vector containing activation values for each memory. Therefore, the final output of FFN can be viewed as the weighted sum of values activated.
Model editors Existing model editors are mainly separated into types: fine-tuning-basse and HyperNetwork-basee Fine-tuning- based editors usually straightly tune the model with an extra loss to eschew over-fitting edit examples. For instance, Zhu et al. (2020) proposed an extra loss
2

---

Published as a conference paper at ICLR 2023
to reduce the distance between pre-edit and post-edit parameters. Mitchell al. (2022a): Meng et al. (2022) equipped fine-tuning with KL-divergence to restrict the post-edit model's output space. For another, HyperNetwork- based editors require additional training phrases. Simitsin et al. (2020)
proposed a Meta Learning-baa maanid (Finn et al., 2017) approach named Editable Training to learn editable parameters for model modification. Cao et al. (2021) proposed KnowledgeFditor (KE) trained with constrained optimization to produce weight updates. Mitchell et al. (2022a) proposed MEND that learns transform the gradient obtained by standard fine-tuning to edit large language models (Raffel et al., 2020). In addition, some works only focus on specific tasks, such as masked language modeling (Dai et al., 2022) and autoregressive language modeling (Meng al., 2022; Geva et al., 2022). They require special input other than edit examples to conduct model editing.
Continual Learning The proposed SME task could be regarded as an emergent variant of Con- tinual Learning (CL) (Mundt al., 2020). And dynamically expandable networks employed for CL as well (Rusu et al., 2016; Li & Hoiem, 2018). 1).. But there are some differences in the setting. In CL, usually, the model continually trained using different datasets and tasks. But SME deals with only one example at once and all examples are from the same task. The difference in setting renders SME unexplored methods. For example, KL divergence loss and L2 normalization are usual methods to address the catastrophic forgetting in CL (De Lange et al., 2022), but previous works (Cao et al., 2021; Mitchell et al., 2022a) and our experiments show that they can hardly maintain models accuracy on irrelevant inputs ME task. And methods that add task specifie parameters for CL usually need extra train- ing (Yoon et al., 2018; Wortsman al., 2020; de Masson d' Autume et al., 2019), thus falling short of SME's application efficiency requirement.
with challenges that may not be properly addressed by general CL
SEQUENTIAL MODEL EDITING PROBLEM
To
Model Editor
Model /2
Model Editur
Model fr
Post-edit
orampo Ve.) (
secies mistakes
The process of sequential model editing task. Given the t-th mistake (Iee Ma.), the editor takes the model ft-1 and (It, Mn,) as input, and outputs the revised model fr.
Figure 2:
Following Mitchell et al. (2022a), model f F can be defined as function f X + Y that maps an input x to its prediction f(x)) Then, given a model f and an edit example pair (Mex Yx.) that f(x) * yr.. model editor ME is to output post-edit model f"
ME FxXXY
F
(f, Vo.)
+ f'
ME(, Fe. Vn..)
Given data stream ((x1, U2 ). ... (xs, (s.))>,)) and an initial model fo, model editor ME needs to conduct edits successively when the model makes undesirable output, as shown in Figure 2.
So
l = O. elif ft-1(xt) Ys else.
ft
fi-
(1)
ME(J+-1, Vz,)
And after every edit in SME the eost model f should satisfy the following three properties:
Property Reliability: the post-edit model should output the desired prediction:
f'(xe)
Vxa
(2)
Property 2 Generality: given edit example Er, = {ιzlızz = ys.} is defined as the set of its equivalent inputs (e.g. rephrased sentences). Then the post-edit model f should satisfy:
Vayg E Brr, 8 (113) yv.
3

---

Published as a conference paper at ICLR 2023
Property 3 Locality: the edit should be implemented locally and precisely, which means the post- edit model should remain accurate on the irrelevant examples set Lz, X ExE,,
yxz Izer (xx = =x,
(4)
In particular, edit should not disrupt the results of past edits in SME setting, which means:
ft(xx) = yzur for where fk-1 (xrk) # yzu
(5)
TRANSFORMER-PATCHHE
Frozen
True
Layer N
Classification:
Elizabeth Truss
Patcher
A Putch
Original Neurons
UK Prime Minister
Fulse (N)
Flizabeth Triss(
Attention Layer
utoregressive
G eeneration:
Who is the UK Prime
Potcher
Transforme mer Layer x (N-1)
Minister?
Rishi Sunak(V)
Input example
Figure 3: Transformer-patee enables efficient correction for classification and generation tasks, rectifies the model's behavior by adding and training several extra neurons in the last FFN layer.
First, we call one misclassification or one wrongly generated token one mistake in the rest of the paper. Aiming at the SME task for transformer-base based models, we propose Transformer-Patcher shown in Figure 3. It freezes all original parameters and adds one neuron (patch) to the last FFN layer for one mistake. And we train the patch to take effect only when encountering correspond- ing mistake. For classification, generation, we count how many tokens are wrongly generated under the teacher-forcing setting and add patch for each of them. This section describes how to add and train patch. Multiple patch editing follows exactly the same principle and is formally described Appendix A.
add only patch rectify the model. For auto-regressive
4.1 WHAT IS A PATCH?
As mentioned in Section FFN operates key-value neuron memories. Its forward computation is a process that retrieves values from matrix V by matching keys in matrix K and the input query q. For a standard FFN, given query q E RRd, its output (FN is:
a = Act(q + by) FFN
(6)
b.
where Act(-) is a non-linear activation function (e.g.. Relu or Gelu), a is the vector of activation values, by. and by are two bias vectors. A patch is extra neuron (an extra key-value pair) added to the last FFN layer. After patching, the new output F FFN, is:
[a apl = Act(qq [K
kpl +
[bx
bp]
(8)
V Up
(F) = [a
ap]
+ b,
(9)
where kp E Rdd is the patch key, vp E Rd is the patch value, b, is a scalar named patch bias, ap
represents the activation value of the patch. With the substitution of equations and 7, equation 9 can be reformulated as:
FFN,(g) FFN + ap Up
(10)
4.2 TRAINING A PATCH FOR EDITING
An ideal edit requires reliability, generality, and locality proposed in Section 3. For reliability, a patch needs be activated according equation 10. Let q. represent the input query of the mistake,
4

---

Published as a conference paper at ICLR 2023
the patch key kp and patch bias by should satisfy:
ap Act kp + bp) + 0
(11)
When Act is ReLU or GeLU, the above condition can be approximated as follows:
qe kp + bp
(12)
To meet the constraint 12, we propose a activation loss la to maximize the activation value:
La exp(- q. kp bp))
(13)
Once patch is activated, according equation 10, it adds bias term Op Dpp to the output of the last layer. Because we are editing the last layer of the model, the output of the model can adjusted to any result without worrying that other components of the model would cancel the editing effect. To obtain the target output, we leverage the task's ' original loss function and rename it as edit loss le- Formally, for an edit example (Nez Me), the patched model's output is Pes le is defined
Le
L(ye, Pe)
(14)
where L(-) is a function of label Be and model output Pa and depends on the specific task.
For locality, the model's behavior should not be shifted on irrelevant examples, thus the patch should not be activated by any irrelevant examples. When using ReLU or GeLU, it can be approximated as that all queries from irrelevant examples qi should have a patch activation value less than or equal to threshold B, i.e., the maximum of them is less than or equal to 3:
Vi E Tx. kp + bp S 3 + max (q: kip + hp) 3
(15)
Thus we propose the memory loss Im to enforce the constraint 15. To imitate the distribution of queries from irrelevant examples, we randomly retain some queries from previously seen examples as memories. Each query is a d-dimensional vector and we can stack them as matrix M E Rd xd where dm is the number of queries saved. Our proposed memory loss (m is the sum of two terms. The first term 4m1 is introduced to make the patch inactivated to all queries in M:
1m S(M kp + by B; k)
(16)
where S(;:k) is a function that receives a vector v and outputs scalar
S(v; k) = (vx[Top
(17)
It first employs element-wise exponential function v and then selects & largest elements com- pute their average as the output. Although constraint 15 is about the maximum, we employ TopK here for more efficient optimization. In case that I, can not absolutely ensure the constraint 15, we propose m2 to distance the activation value of qe and q. That is, the activation value of the mistaken example larger than that of the irrelevant examples by a certain margin 7.
Lm S(M qe) kp + bp 10---:1 k)
(18)
To sum up, the loss Lp for training patch is defined weighted sum of the above losses:
Ip le + al, + mll.. Le + al, + m(mm + lm2)
(19)
where a, m are hyper-parameters. parameters. B is selected as -3 for GeLU and 0 for ReLu, since GeLU(- 3)=z0.004 is small enough and ReLU(0)=0. y is selected as 3 for GeLU and for ReLU.
EXPERIMENTS
5.1
EXPERIMENTAL SETTINGS AND EVALUATION METRICS
We proposed experimental pipeline for SME used for standard datasets with training set Derains validation set Dvat. and test set Dtest- There are two differences between our setting and the previous Model Editing setting. First, employ multi-step editing rather than one-step. Second, previous works usually generate counterfactual edit examples (c.g., replacing the answer to a question with a random one), while employ authentic examples where the model makes mistakes. We first split
5

---

Published as a conference paper at ICLR 2023
the original Dirain into edit set Dedst and new training set D/ train- To evaluate generality, back- translation could be utilized to generate the equivalent set E,,, for edit example Ie E Dedd following previous works (Cao et al., 2021). To evaluate locality, subset D,, randomly sampled from Droin used to see how the post-edit model performs on its training data. Our SME pipeline starts with an initial model Jo trained on train and validated using vat, the model is sequentially edited while encountering mistakes in Dedit- After the ith edit example (x v²), we obtain a post-edit model ft- Supposing that there are T total edits and / represents the indicator function, our proposed metrics are calculated as follows:
1) Success Rate (SR): to evaluate the reliability, we test if the post-edit model outputs the desired prediction. Thus, SR is:
1 SR 7 Σ
- =
(20)
1=
2) Generalization Rate (GR): to evaluate the generality, we test the post-edit model fo on the equivalent set Exy (1 Ne,, of the edit example x' thus GR is:
T N. GR = TN
I(f((=)))=
= =yy)
(21)
3) Edit Retain Rate (ER): to evaluate locality and reliability, we evaluate how many past edits are retained by the final model fr- In real application, a reliable model editor should keep the fixed bugs from recurring again, thus SR alone cannot evaluate reliability, and we define ER by testing the final model on all its past edit examples Epe:
I ER = T [((r(())) = 3./TT
(22)
4) Training Retain Rate (TrainR): to evaluate locality, we compare the performance of the final model of fr and the initial model fo on subsampled test Dir. Thus, the TrainR is defined as:
Train-- E(Eyen, I(f(())=))
(23)
EOSaEE I[fo(x) =)
5) Test Retain Rate (TestR): to evaluate locality, we see the post-edit model still retains the generalization ability unseen data. Then the TestR is defined as:
TestR Elen)EDicn I(T(#)) =
(24)
E(e.)) [(o(z) y)
Datasets and Baselines Both classification and auto-regressive generation tasks are selected for evaluation. Following Cao et al. (2021) and Mitchell et al. (2022a), we employ Fact-Checking (FC) for classification and closed-book Question Answering (QA) for generation. For FC, we apply BERT base model (Devlin et al., 2019) and the FEVER dataset (Thorne et al., 2018). For QA, we apply a BART base model (Lewis et al., 2020) and the Zero- Relation Extraction (zsRE) dataset (Levy et al., 2017). We directly use the equivalent set released by Cao et al. (2021). We use the same data split as Cao al. (2021). Both FC and QA evaluated using accuracy. Our baselines include (1) Fine- Same-TTning-basse
editors: The FT directly fine-tunes the model on the edit example. Following Mitchell et al. (2022a), FT+KL is selected as a baseline. It fine-tunes the model with an extra KL divergence loss 44t. Following Sinitsin et al. (2020) and Zhu et al. (2020), we report fine-tuning-- based baselines by fine-tuning all parameters (Fl(lll) and FT(aII+KLL) or the last layer (last and Fl(ast)+KLL) (2) Two HyperNetwork-
editors: KE (Cao et al., 2021) and MEND (Mitchell et al., 2022a). (3) SERA: a variant of the latest. SOTA memory-basee model editor SERAC (Mitchell et al., 2022b). Other details of our baselines are reported in Appendix B.
Experiment Details Initial models for two tasks are obtained following the same training settings
Cao et al. (2021). For FC, the accuracy of the initial model attains 94,1 1 % By,, 76.9% on Diest- For QA, the accuracy of the initial model attains 56.6% on Dr.. 23. % on Drest- To reduce the experimental uncertainty, randomly split the edit into 20 folders run SME 20

---

Published as a conference paper at ICLR 2023
Table 1: The Success Rate (SR), Generalization Rate (GR), Edit Retain Rate (ER), Training Retain Rate (TrainR), Test Retain Rate (TestR) of Transformer-Patchhee (T-Patcher) and the baselines on FEVER and zsRE dataset. denotes that the SR of the T-patcher on QA is 0.9987. means the method requires extra training phases and training data.
FEVER Fact-Checking
zsRE Question-Answeringg BART-base (139M)
BERT base (110M)
Editor
SR
GR
ER
TrainR
TestR
SR
GR
ER
TrainR
TestR
FT(aatt)
1.00
0.61
0.59
0.893
0,946
1.00
0.58
0.30
0.914
0.924
FT(all)
1.00
0.74
0.83
0.968
0.994
1.00
0.68
0.43
0.865
0.910
FT(latt+KL
1.00
0.53
0.45
0.968
0.998
1.00
0.57
0.28
0.923
0.933
+T
1.00
0.71
0.49
0.998
1.011
1.00
0.68
0.39
0.889
0.925
MEND
0.04
0.03
0.06
0.349
0.652
0.41
0.37
0.00
0.000
0.000
KE
0.14
0.12
0.28
0.486
0.650
0.09
0.08
0.00
0.000
0.000
SERA
1.00
0.89
1.00
0.904
0.916
1.00
0.90
0.98
0.906
0.901
T-Patcher
1.00
0.82
1.00
0.999
1.000
1.00
0.82
0.99
0.997
0.996
The experimental results when utilizing all data in Dedit single run of SME on QA task. The results of the FC task are presented in Table in Appendix C. E represents how many edits have been conducted, N represents how many mistakes have been made by the initial model fo on the entire edit set Dedit-
Table 2:
Editor
SR
GR
ER
TrainR
TestR
N
FT
1.00
0.69
0.14
0.936
0,974
2821
2766
SERA
1.00
0.90
0.97
0.728
0.694
3558
2766
T-Patcher
0.99
0.81
0.97
0.912
0.948
2308
2766
times and report the averaged performance the final result. The initial model fo makes about 63 mistakes an FC folder and about 139 in QA folder on average. For methods requiring memories (fine-tuning with KL and ours), 40,000 memory examples are sampled from Dirain \ Der are employed for both tasks and are updated as editing proceed. The hyperparameters and m are selected as I and 10 respectively for both tasks to make the extra losses and the original task loss in the same order magnitude. Other details can be found in Appendix B.
5.2 EXPERIMENTAL RESULTS
Main results The experiment results are shown in Table 1. Our method achieves strong perfor- mance in all five metries across two tasks. It could make series of model corrections (SR=1) while nearly retaining every past edit (ER=1) and almost perfectly keeping the model's overall per- formance I 11) The fine-tuning-bass
editors could partly preserve the model's behavior and achieve high SR, but it is vuinerable to forgetting previous edits (low ER). Two HyperNetwork--basee editors fail in the SME setting. They have trouble retaining models' over- all performance (low ER, TrainR, TestR) and conducting series of edits (low SR and GR). SERA achieves the highest GR, while can only partially preserve the model's overall performance (TestR, TrainR=0.99) compared to T- T-Patcher. Apart from being effective, well. Using a V100, one edit costs only 7.1s for FC and 18.9s for QA. We could further improve the efficiency to 4.7s and 12.4s by decreasing the number of memory examples 10,000.
method is efficient enough as
Scale up to thousands of edits Table 1 shows that Soinsformer-Patcheer achieves good perfor- mance for about 60 edits on FC and 140 edits on QA, thus we wonder could handle more edits. So we utilize all data in Dedit as single data stream to run SME. As shown in Table 2, Transformer- Patcher could effectively correct up to thousands of mistakes and retain the model's overall performance simultaneously compared with the other two strong baselines. It's interesting to notice that the number of edits E of Transformer-Patte Pattnerrr is less than the number of actual mis- takes N made by the initial model. In other words, method can fix some potential mistakes in the initial model before the error actually happens. On the contrary, the fine-tuning te-iing-n--eeddd method fixes more mistakes than the original model, which means it created more errors during the editing
7

---

Published as a conference paper at ICLR 2023
process. It seems contradictory that method attains fewer E and lower TestR, this may due to the distribution shift between Dedst and Dlest- See more explamation in Appendix C. Furthermore, the post-edit model only gets 1.4% larger for FC and 4.5% larger for QA. We believe this cost is acceptable for automatically correcting the model's mistakes from time to time during deployment. In practice, we suggest using the transformer- paaccer to provide timely response for each mistake online, and after accumulating certain quantities of mistakes, we could fine-tune the original model on all accumulated mistakes, that the patches can be removed. In this way, we could achieve good balance between model size and editing effectiveness.
5.3 ANALYSES
1.0
1.0
T-Patcher
0.6
0.6
MIND
0.4
0.4
T-Paicher
0.0
0
30
60
90
120
150
0.0
93
124
155
(a) Success Rate on FC
(b) Success Rate on QA
Figure 4: Variation of success rate (SR) with the number of edits. Different methods have different
edit times, we plot until they converge.
The collapse of MEND and KE We discuss here why MEND and KE fail in the SME. Figure 4 presents how SR varies with the number edits on both FC and QA. Figure 4 shows that MEND and KE are effective in the first few steps, but shortly after they However, in their original paper (Cao et al., 2021; Mitchell et al., 2022a), they both reported that they achieved high SR when dealing with one-step editing. We find this phenomenon reasonable since both HyperNetwork- with the original parameters. As the editing proceeds, the model becomes more different from the initial one, resulting in their failure. We tried to retrain HyperNets after every edit using the post-edit model, but the cost for re- re-training unacceptable as it costs hours to train HyperNet model editor.
longer able produce valid edits.
editors are trained with the initial model fo and thus strongly coupled
Table The ablation results for two alternatives of memory loss.
FEVER Fact-Checking
zsRE Question-Answeee Answering
Patch
SR
GR
ER
TrainR
TestR
SR
GR
ER
TrainR
TestR
w/o 1m
0.99
0.94
0.61
0.737
0.844
0.99
0.94
0.21
0.069
0.154
KL
1.00
0.76
0.99
0,996
0.998
0.94
0.69
0.49
0.481
0.710
w/o Long
0.95
0.82
0.95
0,994
0.992
0.95
0.82
0.94
0.991
0.984
T-Patcher
1.00
0.82
1.00
0.999
1.000
1.00
0.82
0.99
0.997
0.996
Memory loss To validate the effectiveness of our proposed memory loss, we apply several alter- native patches: (1) T-Patcher w/o 4m, (2) KL Patch, where Im is replaced with the KL divergence loss, (3) T-Patcher w/o lmg- The ablation results in Table 3 show that memory loss is critical. Sim- ply adding patches without memory loss hurts the model's overall performance severely. The KL divergence partially alleviates this problem (higher TrainR, TestR, and ER) but is still unsatisfying on the more complex QA task, which is similar to the Fintuning with KL results in Table 1. By comparing w/o l, and T-Patcher, we observe that the main contribution of our proposed memory loss comes from lm,, while adding l,, still improves the method's performance. Furthermore, to investigate whether our added patches do solely respond to the specific error we visualize the acti- vation values of different patches on their corresponding mistakes in Figure 5 for the QA task. The X-axis represents the mistake (8.2 represents the second mistake of the 8th edit example) and the Y-axis represents the patch. Figure 5a shows that the patch can be activated by multiple irrelevant queries without the constraint of memory loss, leading to low ER, TrainR, and TestR. Figure 5b is lot darker, indicating that the KI. loss tends to deactivate patches to bridge the distribution gap
8

---

Published as a conference paper at ICLR 2023
42
24
(a) T-Patcher w/o km
KI. Patch
(c) T-Patcher
Figure 5: The activation values of three different patches their corresponding mistakes.
0.78
0.8
0.75
0.7
0.6
0.5
Layer
Layer
SR
GR
ER
TrainR
TestR
GR
SR
ER
TrainR
TestR
5000
10000
20000
40000
(a) Patched layer position
(b) Memory size
Figure 6: The ablation studies about patched layer position and the memory size.
before patching and after patching. And figure 5c presents clear diagonal line, which means each patch takes charge of its corresponding mistake. Further analysis of the activation value of different patches is presented in Appendix C.
Patched layer position To validate the benefits of patching the last layer, we focus on the QA task and patch each decoder layer separately. The ablation results are illustrated Figure 6a. First, patching the bottom layer (layer and I) can not make effect edits. This may be because patching the bottom layer severely influences every token the input sequence, making the patch's optimization more difficult. While the patches added to the last layer only influence correspondent mistaken to- kens, Then, compared to the other metrics, what the patching position influenced most is GR, which increases from 0.74 of layer 0.81 of layer 5, proving that patching the top layers may improve the generality. This phenomenon is aligned with previous studies (Jawahar et al., 2019) which found that high-level semantic features are encoded the top layers and superficial information is encoded in lower layers. Besides, patching the last layer could ameliorate the editing efficiency as well. Because computation results of previous layers could be cached and reused while editing.
Memory size In order to verify the robustness of our method, we conduct experiments using different memory sizes (from 5,000 to 40,000) the QA task. As is shown in Figure 6b, our method is not very sensitive to the size of the memory set. Reducing memory examples only causes slight drops in SR, ER, TrainR, and TestR, and slight increase in GR.
CONCLUSION
In this work, we proposed the Sequential Model Editing task, as well as its experiment pipeline and evaluation metrics. We then introduce Transformer-Patcherr, practical method for sequentially editing transformer-bast generation tasks demonstrate its ability to edit the model up thousand times continuously. This method could have positive social impact by fixing serious mistakes in large PLMs, including generating biased predictions and hate speech, benefiting broad spectrum of audiences.
language models. Experiments on both classification and autoregressive

---

Published as a conference paper at ICLR 2023
REFERENCES
Christine Basta, Marta R. Costa-jussà, and Noe Casas. Extensive study on the underlying gender bias in contextualized word embeddings. Neural Computing and Applications, 33(8):3371-3384 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Proceedings of the 2020) Annual Conference Neural Information Processing Systems, 2020.
Pawel Budzianowski and Ivan Vulic. Hello, it's GPT-2 how can I help you? towards the use of pretrained language models for task-oriented dialogue systems. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 15-22, 2019.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, 2021.
Damai Dai, Li Dong, Yaru Hao. Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 8493 8502, 2022.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jía, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(7):3366-333 11113366-336,,00 2022.
Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Proceedings of the 2019 Annual Conference on Neural Information Processing Systems, pp. 13122-13131, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171 4186, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learnin for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1126-1135, 2017.
Mor Geva, Roci Schuster. Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. LM-Debugger: An interactive tool for inspection and intervention in transformer-based language models. CoRR, 0bs/2204.12130. 2022.
Alex Hern Facebook translates "good morning" into "attack them". 2018. URL https: www.w.twggarriaannmmmm wwt. sepoddmorningg attack- arrest
Alestine-isradt -rraiissslelttsslttteeeee
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. What does BERT learn about the structure of lan- guage? Proceedings the 57th Conference of the Association for Computational Linguistics, pp. 3651-3657, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations, 2015.
10

---

Published as a conference paper at ICLR 2023
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning, pp. 333-342, 2017.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sgquenceto--equenee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 7871-7880, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-294) 2018.
Zhengwei Lv, Duoxing Liu, Haifeng Sun, Xiao Liang, Tao Lei, Zhizhong Shi, Feng Zhu, and Lei Yang. AUTOHOME-ORCA at semeval-2019 task 8: Application BERT for fact-checking in community forums. In Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 870-876, 2019.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in GPT. CoRR, abs/2202.05262. 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale. In Proceedings of the 10th International Conference on Learning Representations, 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing at scale. In Proceedings of the 2022 International Conference on Machine Learning, pp. 15817-15831, 2022b.
Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. CoRR, 001/20099.01997 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 11:1140:1140::77 2020.
Sahana Ramnath, Preksha Nema, Deep Sahni, and Mitesh M. Khapra. Towards interpreting BERT for reading comprehension based QA. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 3236-323444 2020.
Andrei A. Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukeuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR. abs/1606.0el 04071, 2016.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In Proceedings of the 8th International Conference Learning Representations, 2020.
Matthew Sotoudeh and Aditya V. Thakur. Provable repair of deep neural networks. Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pp. 588-603,100 2021.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proceedings of the 2015 Annual Conference on Neural Information Processing Systems, pp. 2440-2448, 2015.
James Thoree, Andreas Vlachos, Christos Christodoullpoulls,, and Arpit Mittal. FEVER: A large- scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 819,-819,9 2018.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Proceedings of 2020 Annual Conference on Neural Information Processing Systems, 2020.
11

---

Published as a conference paper at ICLR 2023
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynam- ically expandable networks. Proceedings of the 6th International Conference on Learning Representations, 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language
Technologies, pp. 629 634- 634. 2019.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories transformer models. CoRR, abs/2012.00363, 2020.
12

---

Published as a conference paper at ICLR 2023
A MULTIPLE NEURON PATCHING
In auto-regressive generation tasks, the model may make multiple mistakes in one example. Since FFN is position-wise network, every mistake in the output can be ascribed to one query to the last FFN layer. Therefore, for an example where the model makes n mistakes, each mistake can be ascribed to query q. to the last FFN layer, and we add patches to handle each of them. Specifically, given an input query q, the new output ()))(q) of a FPN with n patches is:
[a apl = Act(q [K
Kp) [bg
+
bpl
(25)
V V
FFN la
apl
+ b.
(26)
where Kp E _Rd is the patch key, Up E iga is the patch value, bp R" is the patch bias, ap Act(q kp + bp) is vector containing activation values of patches. With the substitution of equations and 7, equation 9 can be reformulated as:
(q) FFN (F) + ap vp
if ap = Ō
(27)
else
During calculating the activation loss for multiple patches, we just constraint the patch ke to be acti- vated by its corresponding query qe let qe E IRaxd represent the matrix containing n corresponding queries, then we can obtain A IR' which is defined as a vector containing activation values of each
patch on its corresponding query:
A = dee kip
(28)
can also be formulated as follows:
A = diag ge kp) + bp
(29)
where diag is function to select the diagonal elements from matrix. Then the activation loss for n patches can be calculated as follows:
La S(-A;ka)
(30)
where S is the function defined in Equation 17, hi is hyper-paramest
Memory loss Im for multiple patches remains the sum of two terms I. and 1mz, where Imi identical as Equation 16. As for 222 we restrict that for i-th patch ko all its activation value to query in M should be smaller than that its corresponding query Le. thus lm becomes:
lm2 = S(M kp + hp A- 11--hh k;
(31)
For initialization, every patch ky is initialized as its normalized related query activation value is 1.
of श
so that the initial
B EXPERIMENTAL DETAILS
Data splits We utilize the same data split of training and testing following Cao et al. (2021). For closed-book fact-checking, the binary FEVER dataset originally has 104,966 training instances and 10,444 validation instances. In order to adapt it to the SME task, we keep the original validation set intact and employ it as Weettt and split the original training data into three subsets: new training set Iraii a new validation set D vat and an edit set Dedit in the ratio of 0.8 0.1 : 0.1. As a result, we get 10,496 instances for the edit set. Since the Bert-based classifier attains 88.3% on the edit set, the ideal edit sequence length is 88.36*88.3%/2066
on average.
For closed-book question answering, we employ the zsRE dataset released by Cao al. (2021), which originally has 244,173 examples for training and 27,644 examples for validation. We first filter out examples with only one answer and then employ the same data split process as FEVER in the ratio of 0.9 0.075 0.025. Finally, we get 5,317 edit data and 15,982 for validation, and 24,051 for testing. Since the Bart-based model attains 47.9% on the edit set, the ideal edit sequence length is 3317#47.99/20=111
on average. For both datasets, we randomly sampled subset from Dirain with the size of 10,000 as Dir, and the edit set Dedi is split into n = 20 folders to run SME = 20 times independently. For the model editor requiring memories (fine-tuning with KL and Transformer (ransformer-Patchhr)), we randomly sampled a subset from DV (raiin \ Drr with the size of 40000 and update it as the editing proceeds.
13

---

Published as a conference paper at ICLR 2023
Initial models training Initial models are trained following Cao et al. (2021). For the Fact- Checking task, we fine-tune a BERT base model with an additional linear layer that maps the hidden state of the BOS (beginning of a sentence) token to the probability of the positive label. We maxi- mize the model likelihood and the final model attains an accuracy of 76.9% on Bestt, 94, 1% on Der and 88.3% on Medit- For the QA task, we fine-tune BART base model by maximizing the model likelihood regularized with dropout and label smoothing. The final model attains an accuracy (exact match between model prediction and ground truth) of 23.1% on Deest 56.6% on Dir and 47.9% on Dedit- And these results are comparable with results that the model trained and released by Cao al. (2021) has achieved.
Paansfoomer-Patccher training details For FC, we add one patch for every edit example. For QA, we employ the teacher forcing setting and count how many target tokens are not assigned to the highest likelihood the mistake number. For one edit example, we add up to 5 Patches.FC and QA task share almost the same hyper-parametees. We repeat one edit example times and feed them to PensformmrrPaaccher
batch for training. The initial learning rate is set 0.01. Adam optimizer (Kingma & Ba, 2015) is applied for both tasks. Every patch is initialized with the normalized corresponding query zz.. Such a method makes each patch activated with an initial Tq.1"
activate value 1. The patch value Up E Rnxd is parameterized as element-wise production of two matrices: V E Ruxd and np E Raxd, V/ is initialized with the random number between O and 1, and elements in ny is initialized with an integer 5 to make the patch value dominant over existing values V. The parameter his mentioned in equation 30 is set as 5, and parameter k for memory loss is set as 1000, All hyper-parameters are chosen by running a few examples on the validation set.
Baseline implementation details For KE, we directly utilize the released trained HyperNetwork for conducting SME experiments (Cao et al., 2021).
For MEND, there is no HyperNetwork released and we re-implement the released code with hyper- parameters set as Mitchell et al. (2022a). We employ fine-tuning- suning-based methods following Mitchell et al. (2022a) and Cao et al. (2021).
For all fine-tuning- baaed baselines, we set the learning rate as 1e-5 and utilize Adam's 's optimizer to fine-tune the model until the mistaken example is corrected. For the computation of KL loss for fine-tuning + -Kl-coonsrriinss baselines, we randomly sample a batch of examples in a memory set with the size of 512.
For SERAC (Mitchell et al., 2022b), we implement one variant of it: SERA. The SERAC main- tains cache of all edit examples. Given an input, it first employs a scope classifier estimate if the input is relevant to (falls in the scope of) any cached edit examples. If so, it then employs a counterfactual model (needs to have the identical output space as the original model) to produce the output relying on the most relevant cached example. Otherwise, it returns the output of the original model. In our proposed SME experiment setting, the in-scope examples have the same label as the edit example, thus the function of the counterfactual model is to reproduce the answer of the relevant example. During the implementation of QA, we choose the Bart-base as the counterfactual model, but we find is not trivial for the Bart model to reproduce the answer (the original paper use T5 for generation tasks), thus it is more practical to directly return the label of the cached edit example. We refer this direct-return method SERA and include it as our baseline for both Fact- Bhet-Checking and question-Answeree tasks. All other implementation details about SERA are the same as the original paper (Mitchell et al., 2022b).
Environment details
For all methods, we run SME experiment n=20 times on n different edit folders simultaneously using NVIDIA Tesla V000 GPUs And cost around I hour for running Tmaasormer--atchhe on FEVER and around 3 hours on zsRE.
C EXTRA EXPERIMENT RESULTS
Variation of locality with the number of edits The metric ER, TestR, and TrainR reflect the locality of the final model, but how models behave in the middle is still unclear to us. Thus and investigate how their locality varies
choose KE, MEND, KLL, and Transformer-Patee with the number of edits on the QA task. The results are shown in Figure 7. As editing continues,
14

---

Published as a conference paper at ICLR 2023
Table Mean and deviation of absolute patches activation values on three different kinds of exam- ples
FEVER Fact-Checking
zsRE Question-Answering
Patch
Edit
Past-edit
Random
Edit
Past-edit
Random
w/o mm KL T-Patcher
34.3:19.3
15.7=8.1
0.5+3.0
11.3221.3
1.23#1.64
0.141003
9.15+12.7
0.0110,11
0.05:10.2 0.05
1.12
0,03:1006
0.12:10.1
10.25++2.3
0.00=0.0
6.78+
0.00=0.00
0.10+0.1
Table 5: The standard deviation of Edit Retain Rate (ER), Training Retain Rate (TrainR), Test Retain Rate (TestR) of Transformer-Patchh (T-Patcher) and fine-tuning based baselines on FEVER and zsRE dataset.
FEVER Tact-Checking
zsRE Question-Answering
BERT-base (110M)
BART-base (139M)
Editor
ER
TrainR
TestR
ER
TrainR
TestR
FT(ast)
0.05589
0.06242
0.03322
0.03981
0.00920
0,01860
FT(all)
0.07008
0.03368
0,02178
0.05168
0.02322
0.01781
FT(last)KKL
0.05929
0.02516
0.01635
0.03173
0.01293
0.01697
FTI
0.06248
0.00677
0.01116
0.06433
0.01659
0.01953
T-Patcher
0.00000
0.00045
0.00048
0.00916
0.00101
0.00115
w/o Im
0.10332
0.21872
0.14569
0.23259
0.05063
0.15795
KL
0.00078
0.00536
0.00248
0.07124
0.08237
0.02469
more and more damage has been done to the model by other baselines, except Transformer-Patcher
1.0
1.0
1.0
KE
FTBN
0.4
Purcer
0.4
Puacher
0.2
0.2
0.2
0.0
0.0
20
11.6
145
29
87
11.6
145
29
58
116
145
(a) Edit Retain Rate
(b) Test Retain Rate
(c) Train Retain Rate
Figure Variation of ER, TestR, and TrainR with the number of edits on QA task.
Standard deviation of experiment results Since some values Table 1 and Table are very close, we report the standard deviation in Table 5. Note that the SR and the GR are calculated using all different folders at the same time, the standard deviation is therefore O. According Table 5, Transformer-Pee Patcher achieves the smallest deviation on ER, Traink, and TestR.
Statistics of activation values of different patches In order to study the activation situation of patches on different examples. we present the mean and deviation of absolute patches activation values on three different mistakes: 1) Edit: the mistake for which the patch is added; 2) Past-edit: mistakes from previous edit examples; 3) Random: mistake examples randomly sampled from Drest- As BERT and BART utilize GeLU, both positive and negative activation values could activate the patch. We employ absolute value to measure to what extent the patch is activated. The results are shown in Table 4. First, the T-Patcher w/o Im attains the highest value for Edit queries, indicating the effectiveness of our activation loss. Then our memory loss can effectively push the activation values of Past-edit and Random queries to 0, thus disabling the patch irrelevant examples. The
15

---

Published as a conference paper at ICLR 2023
Table 6: The Success Rate (SR), Generalization Rate (GR), Edit Retain Rate (ER), Training Retain Rate (TrainR), Test Retain Rate (TestR) of TransformmrrPPtcher (T-Patcher) with fixed memory set.
FEVER Fact- Chec-Che BERT-base (110M)
zsRE Question-Answering BART-base (139M)
Editor
SR
GR
ER
TrainR
TestR
SR
GR
ER
TrainR
TestR
T-Patcher
1.00
0.82
0.999
1.000
1.000
1.00
0.82
0.97
0.999
0.997
Table 7: The experimental results when utilizing all data in Dedit single run of SME. E repre- sents how many edits have been conducted. N represents how many mistakes have been made by the initial model fo on the entire edit set Dedit-
Taks
SR
GR
ER
TrainR
TestR
E
N
Editor
FEVER
1.00
0.82
1.00
0.999
1.000
998
1231
T-Patcher
zRRE
0.99
0.81
0.97
0.912
0.948
2308
2766
FEVER
1.00
0.54
0.16
0.998
1.002
1250
1231
FT
zRRE
1.00
0.69
0.14
0.936
0.974
2821
2766
FEVER
1.00
0.89
1.00
0.717
0.709
1588
1231
SERA
ZsRE
1.00
0.90
0.97
0.728
0.694
3558
2766
KL Patch has the lowest activation value of Edit query on both tasks, which explains the lower SR of QA in Table 3.
Editing results of Transformer-Patthh with fixed memory set The experimental results 1 are obtained using a memory set that is updated with the editing proceeds. Thus in Table 6 we present the editing results of Transsorrerr-Paccher using fixed memory set. We only observe a slight decline in ER and a slight rise in TrainR. The results further show the robustness of our method. Besides, we have to highlight that our method allows us save more previous edits memory and leverage more memories in the training process. Because we do not need to save original raw data but only corresponding input queries (several constant vectors that do not require gradients). On the contrary, KL requires feeding mini-batch of raw data into the pre-edit model and post-edit model separately, thus the GPU memory becomes restriction of the number of memories utilized in one batch. But Transformer- Canscer could apply hundreds of thousands of memory vectors in one batch and cost minimal GPU memory and computation resources.
Contradictory of lower E and lower TestR in Table 2 It seems inconsistent that Transformer- Patcher has achieved fewer E and lower TestR than FT method. Because one would expect the model to reduce future errors and behave better on the test set by fixing errors. The phenomenon may be because of the data distribution gap between the edit set and the test set. Thus the improve- ment of "reducing future errors" can not directly lead to higher TestR. For FEVER, the accuracy of the initial model attains 88.3% on the edit set and 76.9% on the test set. For zsRE, the accuracy of the initial model attains 47.9% on the edit set and 23. 1. % on the test set. A distinet gap between the edit set and test set is observed. Thus we should comprehensively consider all metrics to evaluate methods. Another reasonable explanation is that our modified model may slightly overfit the edit example. But fitting more to edit examples may be desired feature in actual applications because we expect the model to be closer to the real data met during deployment.
ACKNOWLEDGMENTS
This work was partially supported by the State Key Laboratory of Software Development Environ- ment of China under Grant SDE
16


Reviews:
[{"review_id": "iaWlDQVHCi", "reviewer": "Reviewer_n4JS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author first proposed the Sequential Model Editing task which requires fixing a series of mistakes for a pre-trained Transformer model by a model editor. And then the author introduced Transformer-Patcher which can be an effective model editor to fix a series of mistakes without serious performance drop. ", "summary_of_the_review": "For some technical details in the model, I didn't do much checking. I prefer to accept this paper and also need to look at the feedback of other reviewers before final deciding.\n", "strength_and_weaknesses": "(+) The paper is clear and easy to follow.\n\n(+) For the Model Editing (ME) problem proposed by Mitchell et al., 2022a, this paper proposes a sequential model editing task (SME) with more practical significance and application scenarios as mentioned in introduction part.\n\n(+) And then the authors give a solution to the SME problem and conduct experiments on both classification and autoregressive generation tasks to prove its ability to edit the model up to thousand times continuously and outperform other methods.\n\n(+) The code is attached as supplementary material and I think should be released if this paper is accepted. \n\n(-) Although the authors claim to support multiple edit operations, they did not perform a stress testing on the model, similar to testing the effect of the model in extreme cases, such as thousands or millions of edit operations. The effect of the final model may deviate from what is described in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OnD0kEZpQx", "reviewer": "Reviewer_YxK1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce the task of Sequential Model Editing (SME) where a model needs to adapt to and fix its mistakes from a stream of incoming input-output examples of a certain machine learning task. The setting extends Model Editing (ME), where people focus on editing the model to fix a single occurrence of an observed mistake. The authors define the desired properties of SME, namely reliability, generality, and locality. The authors then propose Transformer-Patcher for SME, which adds a single neuron to the last fully-connected layer of a Transformer; along with specially designed losses and training schemes for Transformer-Patcher. The authors measure 5 metrics that cover all the three desired aspects. Experiments show that 1) HyperNetwork methods fail MSE, 2) the proposed Transformer-Patcher clearly outperforms the fine-tuning and HyperNetwork baselines, often by a big margin, 3) Transformer-Patcher is effective up to thousands of edits. The authors further provide analysis on the metrics progression of the tested methods, and ablation study of the losses.", "summary_of_the_review": "This is a strong work that extends an existing task (Model Editing) to a more realistic and challenging setting (Sequential Model Editing), where previous successful methods fail. The authors clearly state the desiderata, method, and metrics, along with the intuitions behind them. Experimental results show strong support that the proposed Transformer-Patcher is effective by a clear margin. Further analysis confirms the effectiveness of the design. Model Editing is of interest as the language models grow ever large and are increasingly adopted in real applications. I think this is a work that would interest audiences from the application of large language models, model editing, model adaptation, continuous learning, adversarials and robustness, model security and more.", "strength_and_weaknesses": "### Strengths\n\n1. The proposed task of Sequential Model Editing (SME) is novel, interesting and of practical value. With the increasing adoption of large language models, training, and even fine-tuning, can become computationally expensive or inefficient. We've seen rising interest in Model Editing where people try to find efficient ways to fix undesired model behavior. However, existing works focus on single mistakes from an input-output example. In real ML applications, a model needs to take user requests constantly, any downtime or unfixed mistakes could harm user experience. The SME setting manifests the need for a model-editing method that is timely (quick to apply), efficient (cost/computation effective) and effective (fixes the mistake) even after a series of accumulating edits (not just a single edit).\n2. The SME desiderata (Sec 3), namely reliability, generality, and locality, are clearly stated and formulated. It gives a clear base for what the authors are looking for.\n3. The design of Transformer-Patcher (Sec 4) is simple and of intuitive sense.\n4. The design of the objective losses (Sec 4.2) are clearly explained and are of intuitive sense. In Sec 5.3, the ablation study suggests the usefulness of the losses. I do think the exact formula for l_{m1} (Eq (17), see Q1) seems a bit overcomplicated, but this is relatively minor.\n5. The experimental support is strong that SME is an important and challenging setting and that Transformer-Patcher clearly outperforms competitive baselines. This strength can be further break down to\n   1. The 5 metrics (Sec 5.1) are carefully chosen to reflect different aspects namely reliability, generality, and locality, and are clearly described.\n   2. Competitive baselines are chosen (Sec 5.1), including several variants of fine-tuning and recently proposed HyperNetworks KE (Cao et al., 2021) and MEND (Mitchell et al., 2022a).\n   2. Results (Table 1 and 2) show almost unanimous improvements over baselines across all metrics and both tasks, often by a big margin.\n   3. Main results (Table 1) and metric progression over number of edits (Figure 4 and 5) show that HyperNetworks completely fail SME, despite being successful in single step ME.\n   4. Ablation study (Table 3, Figure 6) shows that their proposed losses are effective in preventing regression across edits.\n7. The paper is clearly written and easy to follow. Some details and phrasing can be made more clear, but they do not harm the main strengths.\n8. Experimental details are provided in the appendix. Code is provided in the supplementary material. Both are good for reproducibility,\n\n### Weaknesses\n\nNothing major. See my Questions and LPs. I think clarifying them can further improve the paper's strength and help me gain more confidence in recommending the work.\n\n### Questions for the authors\n1. P5, Eq (17), \"S(v; k) = Avg[TopK(exp(v); k)]\". Why do we need TopK? If we care about maximum (\"Although constraint 15 is about the maximum...\"), why not simply take the maximum, or use a surrogate like softmax?\n2. Table 1 and 2. Why are TrainR and TextR generally < 1? If we are fixing errors (SR↑, GR↑) and not forgetting previous fixes (ER↑), then we should see the final models be more accurate than the initial models (TrainR>1, TestR↑>1)? The authors argue at one point (P8, L6) that there can be distribution shit between the train and test set. However, we are also seeing TrainR < 1 for all methods and datasets. Are we missing something here?\n\n### Localized points (LPs)\n1. P3, Eq (4), Locality, \"$\\forall x_j \\in I_{x_e}, f′(x_j) = y_{x_j}$\". $\\forall x_j \\in I_{x_e}, f′(x_j) = f(y_{x_j})$ makes sense as \"locality\" to me. The former means that the edited model $f'$ should give correct predictions to irrelevant examples, which is the learning task itself. The latter means that the edited model $f'$ should preserve the same prediction as the unedited model $f$ to irrelevant examples, which is more aligned with \"locality\" -- we do not want to change the model behavior at irrelevant examples, because we believe the model is mostly correct but cannot be sure without testing it against the ground-truth data.\n2. P7, L3 after the figures, \"The ideal edit length for each run is about 63 for\nFC and 139 for zsRE.\" What is \"edit length\" and what does \"ideal'' mean here? Is edit length the size of the D_edit or the number of edits the proposed method performed? Is it a hyperparameter that the authors decided based on experimental observations? Does \"ideal\" mean that the proposed method performs worse after 63/139 mistakes or edits? It seems like this claim belongs to results, not the settings.\n2. P7, Experiment Details. It's worth mentioning the size of the test datasets. How many examples are for the test for each task? What is the total number of time steps T for each of the 20-fold runs?\n3. P7, Table 1, \"* denotes that SR of T-patcher on QA is not 1 but very close to 1.\" Does it mean other SRs in the column are actually exact 1? I suppose that the authors mean that the T-patcher SR is >0.995 so it rounds to 1.00 if we only keep two decimal points, but I cannot be sure. If that is indeed the case, I suggest improving the phrasing, e.g. sampling write out the exact number like \"* T-patcher SR on QA = 0.9965\".\n4. P7, bottom, \"Table 1 shows that Transformer-Patcher achieves good performance for about one hundred edits, ...\" Exactly *how many* edits? And again, does \"edit\" here mean the size of the test set D_edit, or the number of times the proposed method applied a patch?\n\n### Typography/Minor points\n1. Abstract, \"... either fail to make a sequence of edits or to remember previous edits\" -> \"... either fail to make a sequence of edits or fail to remember previous edits\".\n2. P2, top, \"Therefore, an ideal model editor should conduct ...\" Perhaps a better phrasing can be \"Therefore, an ideal model editor should enable _countinuous_ fixing of newly emerged mistakes in a both effective and efficient manner.\"\n3. P2, middle, \"To handle SME.We introduce ...\" -> \"To handle SME, we introduce ...\"\n4. P3, Figure 2. Unify the notations of $y_1$ or $y_{x_1}$.\n5. P3, Sec 3. I suggest use $y_e$, $y_j$, $y_1$, etc instead of $y_{x_e}$,  $y_{x_j}$, $y_{x_1}$, etc to reduce clutter.\n6. P3, Property 1-3. Start with either capitalized or uncapitalized letters.\n7. P3, Eq (5), \"$f_t(x_k) = y_{x_k} , \\{k | f_{k-1}...$\" -> \"$f_t(x_k) = y_{x_k}$ , for $k$ where $f_{k-1} ...$\".\n8. P4, Figure 3. Maybe revise the UK Prime Minister example...\n9. P6, Eq (21). Is it $T \\sum N_t$ or $T N_t$? It makes sense to me if you first average over N_t samples at time t and then average the average over T time steps.\n10. P6, Eq (23), \"we compare the performance the final model of f_T and the initial model f_0 on sub-training test D_tr.\" -> \"we compare the performance of the final model of f_T and the initial model f_0 on the subsampled training set D_tr\".\n11. P6, Eq (20-22). I recommend moving the denominators before the \\sum's. It would look more intuitive to me.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0U4mdVfrzLr", "reviewer": "Reviewer_AwJg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses the model editing problem where the edits are applied sequentially. The paper proposes to add one neuron per edit to the FFN layers of the last transformer module for language models. The hidden dimension of the FFN module is therefore increased by one after one edit is applied. The weights associated with the added neuron are optimized by Adam with the edit and some previous data stored in memory. The experiments evaluate the proposed method with MEND, KE, and some variants of fine-tuning under the sequential model editing setting. The results of two model editing datasets (FEVER, zsRE) show that the proposed method achieves a high edit success rate without affecting non-edit cases.", "summary_of_the_review": "The paper provides a novel method for a less explored problem setting in model editing. The experiment results look promising. However, as mentioned above, the execution has many parts that need significant improvements to support the claims better. The mixed pros and cons lead to a borderline score.\n\n\n[Update]\n\n======== After rebuttal ========\n\nI appreciated the author's detailed responses. Most of my concerns are addressed, although some still need to be fully satisfactory. I have exchanged opinions with other reviewers and agree that the current form's pros slightly out-weight the cons. The score was updated to reflect the overall evaluation. Below are some points that I hope the paper could further improve:\n1. Add stronger support for generalizability.\n2. Add a section to discuss the limitations. Large editing complexity could be one. Generalizability is another arguable one. I believe there are others, and I hope the author discloses all for the community to benefit from your findings.\n3. A better comparison with prior work. The proposed SERA (simplified from SERAC) does not provide a convincing excuse to make the simplification of the SOTA method. Interestingly, SERA has better GR (generalizability) than the proposed method, making the standing of this work less solid. Anyway, SERA is still a nice baseline to add; therefore, I do not consider this a deal breaker.", "strength_and_weaknesses": "[Strengths]\n1. The proposed method looks novel and can scale to thousands of edits. The result looks very good.\n2. The sequential edit setting is not much explored by previous works.\n3. The exp setting in Table 2 is meaningful in that it \"employs authentic examples where the model makes mistakes\".\n\n[Weaknesses]\n1. The method looks slow and requires significant computation (ex: 23.8 seconds per edit for QA).\n2. The paper should have compared the closest work SERAC [1]. SERAC can do sequential model editing and make no changes to the original base model. \n3. The clarity has a large room for improvement. Please see the next section for details.\n4. The claim in the last paragraph of section 4 is not supported. The benefit of editing the last block can be supported by applying the same method to other layers. Besides, more aggressive paraphrasing of facts/questions could be applied to prove better generalizability or create more challenging cases like in [1].\n5. The paper needs a detailed description of how it manages the memory to store previous examples. Does the method sensitive to the amount of stored data? What is the value of d_m used in the experiments?\n6. The ablation in Table 3 can be more fine-grained. For example, ablating L_m1 and L_m2 separately.\n7. To show SME is a setting that ME methods may fail, Table 1 could add the batched results of MEND/KE.\n\n[1] https://arxiv.org/abs/2206.06520", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-iCDxkQaec", "reviewer": "Reviewer_5q14", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new problem, named Sequential Model Editing, that requires a model editor to make a sequence of error corrections while keeping all the previous edits, preserving model performance, as well as generalizing to equivalent inputs. Instead of modifying the model parameters, a set of new parameters (termed neurons in the paper) are added to the last layer of transformers. Because of the existence of the ReLU/GeLU activation functions, the neurons will not be activated for irrelevant inputs. Compared to previous models, Transformer Patcher is shown to be more effective on those metrics.", "summary_of_the_review": "The paper proposes a new model editing problem: Sequential Model Editing, with a method named Transformer Patcher. The method is shown to be effective on the problem, but whether this is a useful task and whether baselines are properly configured need to be discussed.", "strength_and_weaknesses": "Strengths:\n\n1. The paper presents a new problem to the community: How can a model editor fix a sequence of edits without forgetting? The problem is new and most of the previous methods can fail on this task.\n2. Additionally, the paper proposes a few metrics to evaluate the capability of model editors in different dimensions: Reliability, generality, and locality. 5 metrics are proposed to evaluate those properties.\n3. Compared to baseline models, Transformer Patcher shows effectiveness on all of these metrics, especially for GR and ER.\n4. The proposed method can be lightweight for minor error correction since it avoids fine-tunings. It can also be scaled up to thousands of edits.\n\nWeaknesses:\n\n1. The problem proposed in the paper, i.e. sequential model editing, assumes the model would correct the errors one by one, while the problem can be re-formulated so that the model only performs one edit on a batch of errors. It might be practically useful for industrial applications where information may gradually accumulate and the product needs to be up-to-date, but it might not be the typical case.\n2. The problem settings put other models in an unfavorable light. They're not designed to be operated multiple times, and running so many rounds of training to correct certain errors could lead to catastrophic forgetting (Shown in fig 5). In another word, they're not proper baselines to compare with.\n3. Although the method can be scaled up to more edits, the cost is also significant. Since each edit will introduce additional parameters, the memory and computational overhead could be non-negligible, while the basic fine-tuning method does not introduce any additional parameters.\n\nQuestions:\n\nNeurons are designed to be active only for certain errors, but dot-product seems not to be complex enough to recognize the patterns of errors. Thus, I'm interested in how often are the neurons activated. E.g. can they be activated with paraphrases? How often are they wrongly activated for irrelevant examples?\n\n-----\n\nAfter rebuttal: I decided to raise my score from 5 to 6 as I'm leaning to accept the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster