SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On zsRE and SCOTUS, GRACE's averaged TRR and ERR outperforms its closest competitors by  $19\%$  and  $9\%$ , respectively. Comparing the average performance across all methods, GRACE's improvement is  $86\%$  and  $129\%$  due to other methods' inability to balance ERR and TRR. For instance, while Defer achieves the highest TRR on SCOTUS it trades off ERR. In contrast, FT+Retrain achieves the highest ERR on SCOTUS, but trades off TRR. On both datasets, MEND and Defer both struggle to balance TRR and ERR without access to privileged data.
GRACE also outperforms the comparisons on the Hallucination task. We observe that the finetuning methods easily overfit to new edits, leading to competitive ERR but poor TRR. However, their good ERR comes at the expense of ARR in these methods. We also find that ROME and Memory both perform competitively with GRACE, indicating the value of parameter efficiency in lifelong model editing. Finally, we report the average wall-clock time it takes to perform one edit, finding that GRACE makes edits twice as fast as finetuning while competing with other Adaptors.
Excitingly, GRACE can achieve high-quality edits with small codebooks. For example, GRACE uses few keys to edit T5 on zsRE: 1000 edits are made using only 137 keys. Such compression is only feasible by managing the  $\epsilon$ -balls effectively over time. This is also parameter-efficient: with details in Section 3.2.3, the zsRE codebook contains 210,569 scalar values (.35% of T5's parameters) and
(a) Training Retention.
(b) Edit Success.
Figure 3: ES and TRR while editing GPT2-XL on Hallucination. Lower values are better because TRR and ERR measure perplexity. GRACE outperforms the comparisons by making successful edits while maintaining the model's training knowledge. All metrics are shown in Figure 13.
only 70,281 are learnable. The number of keys is also lower-bounded by the number of unique edit labels and is related to the complexity of the inputs. On SCOTUS, GRACE also achieves a relatively small codebook, showing a balance between input complexity and generalizability of each edit. Since SCOTUS is a document classification task with 11 classes, there could be as few as 11 keys, but given each document contains many sentences, there is a lower chance that the same keys can be used for many documents. That GRACE's keys represent 1.51 edits on average indicates significant generalization. As expected, the lower-bound on codebook size is evident in the Hallucination task, where GRACE uses a new key for almost every edit. This is because edit labels are unique sentences. Despite a large codebook, such success at selectively inserting tokens that generate full sentences poses an exciting direction for controllable and editable language modeling.
In Figure 3, we show a more-detailed comparison of all methods for the Hallucination editing task over time, focusing on TRR and ES (see Figure 13 for all metrics). As expected, finetuning methods excel at ES, but perform poorly at TRR. On the flip side, ROME and Memory have reasonably-low ES and TRR scores, though both suffer as edits progress. While ROME is competitive early in editing, it especially suffers on ERR (Table 2) and GRACE's TRR remains  $2\mathrm{x}$  better after making all edits.
3.2.2 Model Analysis: Memorization vs. Generalization
Next, we study GRACE's memorization vs. generalization performance by editing T5 on zsRE on extremely long sequences of edits while varying  $\epsilon_{\mathrm{init}}$  and the edited layer. We split each zsRE question's set of rephrasings into two sets: edits and holdouts. Then, we pass edits through GRACE until 3,000 edits are made. This is extremely large: even 10 edits catastrophically decays performance [12] and [16] performs roughly half as many edits. After each edit, we measure TRR, ERR, F1 on the entire Holdout set, and record the number of keys. Since we evaluate GRACE on the entire Holdout set after each edit, the results start low since has seen no rephrasings of held out edits. Therefore, later measures of Holdout performance are more representative than earlier. Figure 4 shows our results for  $\epsilon_{\mathrm{init}} = 0.1$  and  $\epsilon_{\mathrm{init}} = 3.0$ , while the rest are in Appendix H. We derive the following findings.
Layer and  $\epsilon_{\mathrm{init}}$  choice balance memorization and generalization. We first find that different blocks lead to different editing performance. Notably, editing Blocks two and four achieve high TRR, high ERR, and strong generalization for both choices of  $\epsilon_{\mathrm{init}}$ . On the contrary, for Block 6 we see that since each  $\epsilon_{\mathrm{init}}$  is small, they do not generalize at all and also lead to poor ERR. As expected, when  $\epsilon_{\mathrm{init}}$  is small, TRR performance is optimal, since most edits require a new key. While creating a large codebook can be feasible, the trade-off in Holdout becomes clear, as detailed in the next finding. The relationship between layer choice, performance, and the size of the codebook likely stems from a layer's representational capacity: Layers that map semantically-equivalent inputs near one another will be easier to edit with GRACE.
GRACE edits generalize to unseen inputs. Steadily increasing Holdout shows that that GRACE edits can generalize to previously-unseen holdout edits. Excitingly, interior layers appear to generalize better than early and late layers, which is backed up by their use of fewer keys. Larger  $\epsilon_{\mathrm{init}}$  values also lead to better generalization, as expected, implying that the semantically-similar inputs indeed land in the same deferral radii. The later layer, Block 6, appears to
GRACE codebooks stay small and stabilize over time. Finally, the number of keys over time steadily flattens, indicating that the  $\epsilon$  values indeed adapt to the data distribution. As we increase  $\epsilon_{\mathrm{init}}$ ,
the resultant codebooks get smaller, indicating control over codebook size. Small codebooks are also important for parameter efficiency and in generalization.
Figure 4: Impact of  $\epsilon_{\mathrm{init}}$  and block choice for GRACE editing T5 on zsRE for 3000 sequential edits. Other  $\epsilon_{\mathrm{init}}$  values are in Appendix H. Along with TRR and ERR, we also measure F1 on a "Holdout" edit set containing unseen rephrasings of all edits. We find that blocks 0 and 6 use more keys and achieve higher TRR, but can lead to lower ERR and generalize worse, given lower holdout values.
(a)  $\epsilon_{\mathrm{init}} = 0.1$
(b)  $\epsilon_{\mathrm{init}} = 3.0$
3.2.3 Parameter Efficiency
GRACE's memory requirements are small and straightforward: A new edit requires  $|h^{l-1}| + |h^l| + 1$  parameters, where  $|h^{l-1}|$  is the dimension of the key,  $|h^l|$  is the dimension of the value, and  $\epsilon_{\mathrm{init}}$  is a scalar. Further, the key's  $|h^{l-1}|$  parameters are not trained. As detailed in Appendix E, this is comparable to the alternatives when performing thousands of edits, so performance gain is not from parameter count. This intuition is reinforced by the fact that at inference time, if the Adaptor is activated, predictions are only altered using the  $|h^{l-1}| + |h^l| + 1$  parameters of the chosen entry.
3.2.4 Interpreting GRACE Codebooks
A key benefit of GRACE is that learned codebooks can be detached from the model and inspected. This way, edits can easily be undone without impacting the model and the codebooks can be inspected throughout editing. To demonstrate this advantage, we inspect how keys and their  $\epsilon$  change throughout editing Block 4 of the T5 QA model. In this experiment, we investigate how well GRACE edits generalize to unseen edits. At each edit in a sequence of 1,000 inputs, we pass the entire holdout set of edits through the newly-edited model. For each holdout instance, we record in which key's  $\epsilon$ -ball it lands, if any. This way, we track whether a newly-added key generalizes to multiple holdouts successfully. We also track what proportion of the holdouts land inside any key to evaluate generalization over time. In Figure 5, we summarize the results from one such experiment, with more details in Appendix D. We find that the number of holdouts per key stabilizes over time. Interestingly,  $\epsilon_{\mathrm{init}}$  values capture too many holdouts per question, which explains the trade-off between generalization and TRR. In the Appendix, we also show that the number of holdouts per key are stable, while others are highly-variable. This implies that our codebook maintenance strategy can have big ripple effects according to the key. Further, some regions of the latent space are likely better than others for performing GRACE edits.
3.2.5 Inference Time
To better understand GRACE's limitations, we compare the inference time of the T5-small QA model before and after editing. We compute the time it takes to run one instance through the model for each of 5,000 edits. We edit T5's block 4 and use a small  $\epsilon_{\mathrm{init}}$  of 0.1 to encourage a quickly-growing
Figure 5: Interpreting GRACE keys throughout editing. Larger  $\epsilon_{\mathrm{init}}$  achieve good generalization. The grey line is the true holdouts per edit.
Figure 6: Comparing inference time of GRACE-edited and unedited T5 QA model as the size of the codebook increases.
codebook. To evaluate the variance in training time, we replicate this experiment ten times, each leading to a codebook of roughly 4500 elements. We then average over 20-timestep windows and show standard deviation across all replications.
We find that inference with a GRACE-edited model was only  $1.32\mathrm{x}$  slower than an unedited model on average, as shown in Figure 6. This is a one-time cost that remains fixed even as the codebook grows. Inference time is unchanged because search between one query and a full set of keys can be vectorized. Until the codebook outgrows available memory, this cost remains fixed. That GRACE causes any slowdown is a limitation and an avenue for future work.
Model editing. Model editing is a new and active research area where the goal is to make targeted changes to a pre-trained model's behavior. Most methods propose variants of regularized-finetuning via auxiliary data, like training instances from the pre-trained model's training data or by using semantically-equivalent versions of new edits [37]. Access to such data is non-trivial, especially as training data and paradigms are becoming proprietary, and collecting semantically-equivalent inputs is often unlikely. Recent works have retained these requirements, while extending to pretrain hypernetworks that predict edits [8, 30, 31], often decomposing weight updates into low-rank components [28, 29]. Recent approaches all study transformer architectures due to their popularity and high training cost [48, 51]. To ensure targeted edits, recent methods like MEND [30] and ROME [28] also draw inspiration from parameter-efficient finetuning [15]. But such methods are known to often require more finetuning steps and are prone to overfit more than regular finetuning [39, 50]. Further, recent works show that edited models are deeply fragile [4] and methods for picking which parameters to update are surprisingly unreliable [13].
Unfortunately, nearly all model editing works consider only static edits, making only one edit to a model. While some recent works like MEMIT [29] indeed perform multiple edits, they do so simultaneously, not over time during deployment. Other works demonstrate that editing performance decays quickly when making multiple edits [30], though recent works like SERAC [31] and MEMIT [29] show burgeoning results in this direction. However, these exciting works still use large amounts of privileged information. Most similar to our work, two recent papers discuss sequential editing. First, [12] shows that after editing the same model 10 times, editing performance drops dramatically. Second, a concurrent paper [16] recently proposed a sequential editing setup. However, their method and implementation is architecture-specific and relies on large sources of unrelated inputs.
Continual Learning. Continual learning methods are a reasonable approach to lifelong model editing. Most-similar to our setup, recent works have investigated continual finetuning, where large language models are refined over time as new instances arrive. For example, [25] perform a large benchmark of continual finetuning. They find that regularizing finetuning with continual learning methods like Elastic Weight Consolidation [19], Experience Replay [36], and Maximally Interfered Replay [1], quickly decays performance on prior tasks, though it helps to remember some previous inputs. This implies that editing, as opposed to regular continual finetuning, is particularly challenging, since edits are unlikely to be uniformly distributed [14]. One promising avenue for continual learning is key-value methods, stemming from computer vision [26, 34, 42] For example, recent works have demonstrated continual prompt-learning for NLP [44, 45] for applications like text retrieval [47]. Recent works have shown that discrete key-value methods in particular perform well with shifting distributions [41], with recent works extending to question answering [7]. By caching values, these approaches keep inputs in-distribution for downstream encoders while opening doors to longer-term
memory, resources permitting. We corroborate these advantages in our experiments, where we demonstrate GRACE's robustness to shifting inputs and labels over long sequences of edits.
5 Limitations and Ethical Considerations
Lifelong model editing is new and challenging, so our method GRACE has limitations. Understandably, adding similarity search to the layers of a model will slow down inference. Still, while we do not emphasize inference time, accelerating GRACE is natural future step that has been successful in similar methods [15, 33]. Future works may also scale GRACE up to multi-layer edits, a setting unconsidered in this work. While GRACE has already scaled up continual editing to  $\sim 5\mathrm{k}$  edits, real world scenarios might entail approaches that work at an ever larger editing scale, which presents a promising direction of future work. Another limitation of GRACE-style edits is implication: Behaviors are edited in isolation. For example, if we edit a model's knowledge of the latest pandemic in the U.S. from Swine Flu to COVID, its knowledge of the second-to-last pandemic will not be updated.
While editing models can improve their behavior, it can surely be used for harm. For example, a bad actor might edit a LLM to increase hate. This limitation is true for all model editors, GRACE included. However, most model editors today directly update the model's weights. This makes it hard to trace back what edits have been made to a model and understand their impact. Transparent editors like GRACE are a promising direction for overcoming this problem. For example, GRACE's codebook can be directly inspected to see what predictions stem from each value and examine properties of the latent space covered by the  $\epsilon$ -balls.
Pre-trained models continue to grow and are being applied to a diverse set of downstream tasks. However, they still misbehave in unpredictable ways when deployed. Correcting such behavior is a challenging problem, especially when only some of the model's behavior is problematic. While regularized finetuning or retraining on better data can mitigate this problem somewhat, big models remain too expensive to train for most researchers. We instead build on the model editing literature, and study Lifelong Model Editing, an important but understudied problem. With a focus on transformer models for natural language processing, we edit models thousands of times in a row as edits stream during simulated deployments. Our edits only use singular, authentic errors, in contrast to recent works which make synthetic edits and require large amounts of exogenous data like sets of training edits, semantically-equivalent examples, or pre-training data. We then present GRACE, a plug-in Adaptor for a model's chosen layer that leaves the trained weights untouched. GRACE Adaptors (1) retain the functionality of the original model, while (2) successfully editing model predictions without forgetting previous inputs. Using three real-world datasets, we edit T5, BERT, and GPT models thousands of times and find that GRACE significantly outperforms seven state-of-the-art alternatives. We further investigate GRACE's capacity to make extremely long sequences of edits and show that GRACE can generalize its edits to unseen inputs, avoiding sheer memorization.
We are grateful to all the support received while conducting this research. This project was supported by Quanta, and Marzyeh Ghassemi was supported by the Herman L. F. von Helmholtz Career Development Professorship and the CIFAR Azrieli Global Scholar award. Yoon Kim was supported by MachineLearningApplications@CSAIL.
[1] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. ArXiv preprint, abs/1908.04742, 2019.
[2] Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. In Empirical Methods in Natural Language Processing, 2022.
[38] Harold Spaeth, Lee Epstein, Andrew Segal, Ted Ruger, Sarah Benesh, Jeffrey Segal, and Andrew Martin. Supreme court database, 2020. URL http://supremecourtdatabase.org/.
[39] Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3949-3969, 2022.
[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[41] Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint arXiv:2207.11240, 2022.
[42] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[43] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[44] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. arXiv preprint arXiv:2204.04799, 2022.
[45] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[46] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014.
[47] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
[48] Yang Xu, Yutai Hou, and Wanxiang Che. Language anisotropic cross-lingual model editing. arXiv preprint arXiv:2205.12677, 2022.
[49] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[50] Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint arXiv:2208.10160, 2022.
[51] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
Training Details. All methods are optimized using Adam [18]. Since edits are singular and sequential in our setup, the batch size is always 1. We trained all methods using various GPUs including 48GB NVIDIA RTX A6000s, 40GB NVIDIA A100s, and 80GB NVIDIA A100s. Timing experiments are reported from experiments using a 48GB NVIDIA RTX A6000 GPU. GRACE does not depend on the scale of the model, but the scale of the model depends on available compute resources. To avoid sharding, we use models that fit on one GPU, thought GRACE principles apply beyond this setup. For Adaptor-based editors, we use 100 iterations of gradient descent per input. For experiments using SCOTUS for classification and zsRE for Question Answering, we include early stopping once the model's output is corrected.
Hyperparameters. For our Finetuning, Finetuning with EWC, Finetuning with periodic retraining, Memory, Defer, and MEND comparisons, we consider learning rates of 1.0,  $1e^{-1}$ ,  $1e^{-2}$ ,  $1e^{-3}$ ,  $1e^{-4}$ , and  $1e^{-5}$ . We find that Finetuning, Memory, and MEND worked best with  $1e^{-2}$ . For Adaptor methods GRACE and Defer, we found that a large learning rate of 1.0 was required to update the model's behavior since their only control on the model's output is by replacing tokens at one layer.
Choosing which layer to edit is another hyperparameter for all editors. In all of our comparisons between editors, each editor edits the same layer. For T5, this is last dense layer of the last encoder block (encoder.block[7].layer[1].DenseReluDense.wo), for BERT it is the second-to-last layer (bert encoder layer [10].output.dense), and for GPT2-XL we edit the fully-connected component of the thirty-sixth layer (transformer.h[35].mlp.c_fc). Layer 35 is not random: layers later than 35 led to less-consistent edit success. This is supported by recent work showing the impact of choosing the right layers to finetune [23]. We study these layer choices below, but note that choosing this layer is a hyperparameter in practice: for the purposes of comparison, we ensure editors are compared when editing the same layers.
GRACE has one unique hyperparameter,  $\epsilon_{\mathrm{init}}$ , which is the size of the  $\epsilon$ -ball initialized around new codebook entries. In our main results, we set  $\epsilon_{\mathrm{init}} = 0.5$  for zsRE (leads to 137 keys/1000 edits),  $\epsilon_{\mathrm{init}} = 1.0$  for SCOTUS (leads to 252 keys/375 edits), and  $\epsilon_{\mathrm{init}} = 1.0$  for Hallucination (leads to 1341 keys/1392 edits). We study the impacts of choosing  $\epsilon_{\mathrm{init}}$  further in Appendices F and H.
Runtime. Our main results report runtime for each method on the Hallucination dataset editing GPT2-XL. This time recorded is only the average time required to make one edit. This ignores all preprocessing, precomputation, or pretraining.
B Additional Dataset and Model Descriptions
As summarized in Table 1, we provide detailed summaries of the datasets and models we use for editing in our experiments.
B.1 Editing T5 on zsRE Question Answering
zsRE Dataset. zsRE [24] is a context-free question-answering dataset that has been extensively studied in the model editing literature [8, 16, 30, 31]. The same as previous works, we extract our set set from the same validation dataset. For our main experiments, our edit set is the first 200 question-answer pairs in the zsRE validation set that have at least 5 rephrasings each. We then use the first 5 rephrasings for each, leading to 1000 possible edits. To ablate design choices in GRACE, we use these same edits in Appendix F.