WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Authors: Lukas Thede
Karsten Roth
Matthias Bethge
Zeynep Akata
Thomas Hartvigsen
Venue: ICML 2025 poster (2025)
Keywords: LLM
Knowledge Editing
Lifelong Learning
Primary area: deep_learning->large_language_models
TL;DR: We propose a lifelong, large-scale and real-world knowledge edit benchmark, using which we provide a comprehensive study on best practices for real-world knowledge editing at scale.

Abstract: Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into lifelong knowledge editing to real-world edits at practically relevant scale. We first introduce \texttt{WikiBigEdit}; a large-scale benchmark of real-world Wikidata edits, built to automatically extend lifelong for future-proof benchmarking. In its first instance, it includes over 500K question-answer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use \texttt{WikiBigEdit} to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their capabilities to generic modification techniques such as retrieval augmentation and continual finetuning to acquire a complete picture of the practical extent of current lifelong knowledge editing.

EikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Lukas Thede 123 Karsten Roth 123
Matthias Bethge
Zeynep Akata 234 Thomas Hartvigsen
Abstract
However, frequent full model retraining is prohibitively expensive. To address this, lifelong knowledge editing tech- niques (Hartvigsen et al., 2023; Meng et al., 2023; Wang et al., 2024a) have recently emerged. In controlled bench- mark studies, they have shown success incorporating new facts along three key metrics: (1) generalization beyond merely memorizing question- annwerr pairs, (2) locality of edits to maintain pretraining knowledge, and (3) retention of all previously seen edits.
Keeping large language models factually up-to- date is crucial for deployment, yet costly retrain- ing remains a challenge. Knowledge editing of- fers a promising alternative, but methods are only tested small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into life- long knowledge editing to real-world edits at a practically relevant scale. We first introduce WikiBi wikiBigEdit t;; large-scale benchmark of real- world Wikidata edits, built to automatically ex- tend lifelong for future-proof benchmarking. In its first instance, it includes over 500K question- answer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use Wiki i Bi BEdit to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their ca- pabilities generic modification techniques such retrieval augmentation and continual finetun- ing to acquire complete picture of the practical extent of current lifelong knowledge editing.'
Unfortunately, existing benchmarks are small synthetic, and precede the knowledge cutoff of modern LLMs. While more recent efforts Jang et al. (2022); Khodja et al. (2024) avoid synthetic edits by using wikidata, they remain small- scale (around 20K), and have yet to examine the difficulty of edits, limiting our understanding of really lifelong knowl- edge editing capabilities at practically-nen needed scales. LLMs are nowadays trained on trillions of text tokens (e.g. Llama- 3 (et al., 2024a) on 1ST), and studying lifelong factuality updates demands large volumes of recent real-world knowl- edge edits.
To better understand lifelong knowledge editing at a scale and scenario of practical importance, we introduce WikiBigEdit a large-scale, lifelong benchmark (Prabhu et al., 2024; Ghosh et al., 2024) with fully automated dataset extraction pipeline that continuously expands with new real-world factual edits to ensure future-proof factual- ity probing. At its inception, WikiBiggiie 500K high-quality question-answer answer pairs (around million tokens; magnitudes larger than standard edit benchmarks) spanning eight time-nttrraal. over five months (February July 2024) reflecting a typical interval between subsequent version releases of large-scale LLMs² to better emulate the deployment time required to cover with factual updates.
1. Introduction
Keeping large language models factually up-to-date is a critical challenge their practical deployment (Hartvigsen al., 2023; Meng et al., 2023; 2022; Wang et al., 2024a; Zhang et al., 2024) due to given knowledge cutoffs and limited tail-end concept coverage (Cheng et al., 2024; Zheng et al., 2024; Zhang et al., 2024; Udandarao et al., 2024) incurring inaccuracies. Finding ways to inject or update new and revised facts is crucial; Particularly for factuality- critical applications like medicine (Bao et al., 2023; Jeong et al., 2024), law (Colombo al., 2024; Lai al., 2024) or education (Team et al., 2024c; Wang al., 2024b).
covers over
Derived from periodic changes to Wikidata knowl- edge graphs (Jang et al., 2022; Khodja et al., 2024), wi wikiiight gEdit t covers large range of factual edits and refinements. Moreover, Wik WikiBigEdit introduces com- prehensive evaluation axes going beyond standard knowl-
'Tübingen AI Center, University of Tübingen HHelmholtz Mu- nich 'Munich Center for Machine Learning (MCML) 'Technical University, Munich University of Virginia. Correspondence to: Lukas, Thede <lukas. thede @edeeuun (wwn-uunnuuuiiuuu tubbinge.dbe...
*CounterFact (Meng al., 2023) 20K, ZsRE (Levy al., 2017) IK. GeIPCheckGPT (Manakul al., 2023) at 600. 'E.g. Llama (Touvron et al., 2023) to LLama 2 (et al., 2023). DeepSeck-V (et al., 2024b) to V2 (et al., 2024c) and DeepSeck- V3 (et al., 2024d).
Proceedings of the ndnd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s). Code available at

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
edge editing performance measures, incorporating local- ity checks, multi-hop reasoning across incorporated edits as well as standard and complex generalization tests. We t to explore the properties of real-world knowledge edits and conduct an extensive study of knowl- edge editing techniques for Wiki WikiiiggEit--ce t-sale updates, focusing on their ability to integrate numerous sequential edits while maintaining previous knowledge and factual accuracy beyond simple memorization. Additionally, we compare these methods against other generic modification techniques such as retrieval augmentation (Vu et al., 2023; Prabhu et al., 2023a; Gui et al., 2024) and continual fine- tuning (Ibrahim et al., 2024; Roth et al., 2024). Our study reveals limited transfer of knowledge editing techniques to practically scalable lifelong updates and highlights that even when evaluated around knowledge editing desiderata, stan- dard approaches such as retrieval augmentation or continual finetuning with model merging can perform better.
Beachmark
Date
Data Source
Task
Lifeoog
Mhop
Google guodie
Upen factial QA
Tissa QN
650K
Vizioss
Tikis soucces (web)
Trista QA
MS MARCO
1M4
2016
Bing Soathhparii
Scarch quario
X
Hatpet QA
112K
2018
Wiliisdia (suraie))
Mullishop QA
X
use
PENIR
EEVER
60X
155K
2023
2018
Hypeflizked Wikipodia
Homan-write clams
Milp fact verid.
Faet werthattnn
X
X
ELLS
270K
2019
Redat (ELI sileeddi
Loug foems QA
X
WIL:OA
3.5K
2015
Wikipslia
Fazied QA
X
Dare
200K
Varions
Vaned web sources
temponn UA
X
SmramingUA
150K
2017--3131
WMT new inicles
Eaemi-hoood Q4
X
ActivaaA
530K
1965 2008
Hitoriall news anchines
Fact based QA
Hallo
50K
2123-3134
Kand Wikipadka
Facl verifzation
CLARK
1.KK
2021-3024
Wilipsdia
Kundaagge QA
Pop2A
2022
Wilipadia, Wissban
hott-hoed
Womp
7K
20221
Wikipadia, Wikiben
Tempural QA
WilFFoDDiT
2021
2013
Wilidan
Foctual
gidt
562K
2024
Wikidatt
Pact-husrd QA
Table I. Comparison of QA and edit benchmarks in terms of
size, date, question types, sources, and lifelong applicability; high- lighting the unique placement of our Wikii, Bi gEdit t. benchmark.
new facts without retraining. ZsRE (Levy et al., 2017) and COUNTERFACT (Meng et al.. 2023) focus on syn- thetic updates, while AToKE (Yin et al., 2024) and Chro- noEdit (Ge et al., 2024) evaluate temporal consistency but remain limited in scale and scope. ELKEN (Peng al., 2024) and EVEDIT (Liu et al., 2024b) generate edits based on real-world events. TemporalWiki (Jang et al., 2022) and WikiFactDiff' (Khodja et al., 2024) are more closely aligned with WikiBigEdit. Both evaluate temporal up- date consistency using consecutive Wikidata snapshots. Wi kiki Widditt provides over a magnitude more samples (Table 2), future-proofing and comprehensive evaluation along all editing desiderata.
Overall, we make the following contributions to the study of real-world lifelong knowledge editing as scale:
We present wi ki Riggitt: t:: large-scale benchmark of 500k high-quality question-answe pairs spanning eight timesteps over six months, derived from real- world factual changes in Wikidata to both analyze and simulate real-world knowledge edit tasks at a practi- cally relevant scale.
A fully automated extraction pipeline which continu- ously extracts new, suitable factual edits from Wikidata knowledge graphs, enabling the benchmark evolve with new knowledge updates continuously.
Knowledge Editing, Approaches for integrating new knowledge into LLMs include local modification methods such as ROME (Meng et al., 2023) and MEMIT (Meng et al., 2022), global optimization techniques like MEND (Mitchell al., 2021), and external memorization strategies such as SERAC (Mitchell et al., 2022), GRACE (Hartvigsen et al., 2023), RECIPE (Chen et al., 2024), and WISE (Wang et al., 2024a). While these techniques support efficient model updates, they often suffer from catastrophic forget- ting and knowledge degradation when applied to large-scale or sequential edits (Hsueh et al., 2024; Gupta et al., 2024b; Yang et al., 2024b). Our benchmark offers a practical frame- work to examine these limitations under realistic, evolving knowledge conditions.
Using WikiBigEdit, we thoroughly analyze the ca- pability of existing lifelong knowledge editing methods to conduct lifelong edits at scale; contrasted against re- trieval augmentation and continual finetuning to under- stand limits in relation to other established approaches.
2. Related Works
(Lifelong) Knowledge Editing Benchmarks. Traditional QA datasets like NQ (Kwiatkowski et al., 2019), Trivi- aQA (Joshi et al., 2017), and MS MARCO (Bajaj et al., 2018) target open-domain factual QA, HotpotQA (Yang et al., 2018) and EX-FEVER (Ma et al., 2024) assess multi- hop reasoning. Fan al. (2019); Li al. (2024); Cheng et al. (2024); Mallen et al. (2023) focus on long-form ex- planatory questions and time-sensitive QA. StreamingQA (Liška et al., 2022), ArchivalQA (Wang et al., 2021) or HelloFresh (Franzmeyer et al., 2024) address evolving or time-sensitive, event-driven data, while TiC-LM (Li et al., 2025) explores continual pretraining of LLMs. Knowl- edge editing benchmarks assess the ability to integrate
Retrieval Augmented Generation (RAG). By combining a retrieval module with a generative language model, this class of methods addresses issues such as outdated knowledge and untraceable generations (Lewis et al., 2020; Guu et al., 2020; Karpukhin et al., 2020). RAG approaches have shown strong performance on mhowledge-intensing tasks, even at smaller model scales (Borgeaud et al., 2022; Izacard et al., 2022), and enable attribution by linking generated outputs
*While WikiFactDiff contains over 450K triple-text pairs, only 20K subset (WFDaep) commonly used for editing due to its defined evaluation protocol.
2

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
to retrieved sources (Rashkin al., 2022; Honovich al., 2022). Moreover, RAG supports dynamic knowledge up- dates, mitigating context-memory conflicts when parametric information becomes stale (Vu et al., 2023). However, RAG faces limitations in multi-hop reasoning and compositional retrieval. More advanced variants- such as HippoRAG (Gutiérrez et al., 2025) or agent-based RAG systems- aim to address these limitations by improving reasoning over retrieved information.
Snapshot (Old)
Changed Factual Triplets
Unchanged Factual Triplets
Subject
Ralotion
Object
Subject
Relation
Object
Compare
Socetiie
Snapshot
Happur
part
(New)
LFc4
nanstactuce:
Apple
Hascstica knthas
MoChacaand
YFilter Factual Triplets
Changed
Mhop
Locality Unchanngd
GPT-4o-mini (A. for Update and Localit)
Updatt
Rephrose
De
Personas
Mhop
Locality
Continual Finetuming and Model Merging. Continual fine-tuning has emerged as a reliable approach for general large-scale model updates (Garg et al., 2024; Prabhu et al., 2023b; Ibrahim et al., 2024). Roth et al. (2024) show that parameter-efficien finetuning via e-g. LoRA (Hu et al., 2021) or GaLore (Zhao et al., 2024), can be very effective, leveraging regularization provided by limiting trainable pa- rameter counts (Thede et al., 2024). Parameter-selective techniques, including BitFit (Zaken et al., 2022) and LNFit (Min et al., 2023), further reduce computational overhead by targeting specific parameter subsets. Additionally, regu- larization strategies from continual learning, such as EWC (Kirkpatrick al., 2017) and SI (Zenke et al., 2017), have shown notable success in mitigating forgetting during the continual adaptation of foundation models. Model merg- ing ( (Wortsman et al., 2022; Ilharco et al., 2022; Ramé et al., 2024) has shown promising gains coupled with finetuning for continual tasks (Stojanovski et al., 2022; Marouf et al., 2024; Kozal et al., 2024; Marczak et al., 2024; Dziadzio et al., 2024). We investigate and contrast these approaches against knowledge editing at scale.
Filter QA Pairs
Update
+ Rephrase
Personas wets ooee
Mhop
E Localitt
Q onuite nomeed ofher?
Q w-tterwerr
the Weremmnndkiioee fex ra origana
mrsnalle of the anrty ofher a nemweeo
was Loccate A: 2oczato
poweer nomeed?
Training
Evoluation
Figure 1. Automated Dataset Generation Pipeline. (1) Com- paring two Wikidata knowledge graph snapshots; extract sets of triplets, (2) filter
unchanged and changed cubject-relation-ole for quality, (3) generate locality probes, (4) produce multi-hop hop rea- soning qintuples, (5) convert into question-answa pairs using e.g. GPT-3.5, (6) introduce rephrased and persona-based evaluations to further test generalization, conduct (7) final quality assurance. Put together, this can continuously extend wi WikiBig gedit with additional time-specific high-quality knowledge edit data.
of exact question-anae -nnswrr pairings.
3.1. Lifelong Automated Extraction of Factual Edits
As a proxy for world knowledge, we leverage Wikidata knowledge graphs, following recent works (Jang et al., 2022: Khodja et al., 2024). By performing pairwise comparisons of snapshots taken at different points in time, we identify changes that approximate real-world knowledge evolution. We thus automatically generate Wi MikiBigEdit using our proposed seven-step pipeline, as follows:
3. WikiBigEdit
Wi WikiBig gEdit t addresses three key aspects: (1) it extracts a large number of realistic knowledge edits, (2) incorporates extensive evaluations to assess generalization, reasoning, and locality, and (3) operates fully automatically. The com- plete pipeline is shown in Figure 1.
(1) Periodic Snapshot Acquiiiion. The automated pipeline is initiated through periodic downloads of most recent snap- shots of the Wikidata knowledge graph, and paired with the last downloaded snapshot. Pairwise comparison of these snapshots allows to extract two distinct sets of subject- relation-object triplets T = (subj, rel, obj), S = {T:): Suatie representing unchanged facts and Schanged capturing updated or newly added information (c.f. top of Figure 1). While Schanged will be used to create the knowledge edits to be incorporated into base model, Sixed is primarily utilized for evaluations. Each triplet describes a given relation (such between subject (e.g.
Problem Definition. The task of large-scale, lifelong knowledge editing requires updating pre-trained LLM f with a stream of factual updates [ui U2,........ MT], where each update = (q,, at) consists of a question qe and its factual answer a,. These updates are grouped into B se- quential batches [U,, U2..... Un], where each batch can encompass anything from single edit to multiple. For each batch update 6 (also denoted as timestep in this work), the model fb-1, trained on updates from prior batches U2b, is further updated with the current batch U2 to produce y². The model must meet three key objectives: (I) integrate updates from U., (2) retain knowledge from UzA and (3) ensure that the semantic understanding required to correctly solve pre- viously seen edits generalizes beyond simple memorization
as "capital", "manufacturer"; "Norway") and object (s.a. "Oslo"). A detailed overview is provided in the supplementary.
(2) Initial Filtering. To ensure high-quality triplets, we apply several filtering steps to both Sqaati and Schanged: (i) We exclude triplets with eircular dependencies, (ii) where the
3

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
subject or object entities consist of non-Roman characters, (iii) single characters, (iv) overly long phrases (e-g., more than five words) and (v) where single subject-relation combination maps to multiple objects to avoid ambiguity.
Timestop
Date Range
Samples
Unsolved
TI
2024/02/01 - 2024/02/20
26,922
80%
T2
2024/02/20- - 2024/03/01
29,835
84%
2024/03/11 2024/03/20
54,504
85%
T4
2024/03/20
2024/04/01
43,443
85%
T5
2024/04/01
2024/05/01
121,116
81%
(3) Generation of Locality Probes. The filtered Schanged defines the knowledge updates. To evaluate whether knowl- edge edits remain local, i.c. do not influence (related) exist- ing knowledge, we derive a locality set Siecaity from Sstatie and Schanged; where each factual triplet in the changed set is paired with a triplet semantically similar in subject and relation but with distinct object. For each factual update, this gives a corresponding question whose answer should remain unaffected to probe the locality of knowledge edits.
T6
2024/05/01
- 2024/16001
101.728
82%
2024/06001
2024/06/20
69,403
82%
T8
2024/06/20 2024/07/01
55,431
82%
Total
502,382
82%
Table 2. Description of WikiBigEdit timesteps. Sample counts reflect number of question-answer pairs associated with a given time-interval, alongside information highlighting the per- centage of unsolved edits across studied L.MMs; highlighting the large number of required real-world factual updates across LLMs.
(4) Inclusion of Multi-Hop Reasoning Tuples. To enable reasoning evaluation, we further extract multi-hop (mhop) tuples Smoop from Schanged. In particular, we identify pairs of factual triplets (Ti, Tj) where T, serves as the subject of the second triplet, T for example T, = (podcast, named after, iPod) and Tj- = (iPod, manufacturer, Apple). The triplets Ti and T are then concatenated to form two- hop factual qintuples Tij. e.g. (podcast, named after, iPod, manufacturer, Apple). By restricting the extraction of Smhop to Schanged, we ensure fair evaluation, as the model will have been updated with both individual facts prior to the reasoning task. We find that for a filtered batch Schanged: we can recombine around 3.9% update triplets into Smhop-
ject, object and relation remain correctly encoded in both.
(7) Final Filtering Stage. A final filtering process ensures the quality of the generated question-answer pairs for both training and evaluation sets by verifying that the gener- ated questions contain the respective subject and relation while not giving away the answer and answers incorporate the right object. After this stage, the final SQA S SQA changed locality
SQA SQA mhop' Stephraeed an Spe personas are incorporated as new knowl- edge edit batch into WikiBigEdit.
3.2. Analysis and Exploration
In this section, we first summarize various benchmark char- acteristics before leveraging the first instantiation of our large-scale Wi WikiBig BEdit t to study the structure and prop- erties of real-world knowledge edits derived from Wikidata knowledge graphs. Table 2 provides an overview of the eight time-intervals that constitute WikiBigEdit The factual updates are extracted on a roughly bi-monthly fre- quency from Wikidata snapshots spanning February to July 2024; with the availability of Wikidata snapshots determin- ing size and extend of factual updates associated with each timestep.
(5) Generation of Question-Answer Edit Pairs. For Schanged, Sweality and Smiop: we generate corresponding question-answer pairs using GPT-4o-mini (Brown et al., 2020) (GPT-3.5 for update and locality) through few-shot prompting (for all prompt templates, please see the sup- plementary). For Smhop, we generate questions that omit the middle entity of the two-hop factual quintuple. Using this, we generate the corresponding question-anse annswrr-- sets SQA SQA SQA SOA changes locality mhap' Stocality and Snopp are used for eval- uation, while changed constitutes the respective fact-based training data.
To ensure high data quality, we filter out approximately 76% of initial changes, removing cyelic or overly long entities (1 (11%), non-entity-to-entity relations (50%), and nondeter- ministic triplets (15%). The filtered updates range from 27k to 69k per step, with monthly factual changes exceed- ing 100k. This variability reflects the dynamic nature of knowledge updates and practical constraints when aiming to ensure that models remain up-to-date. Overall, this version contains 502.382 question-ansee answer pairs across 6.9M tokens.
(6) Generation of Personalized and Rephrased Question- Answer Edit Pairs for Generalization Studies. While mhoop allow us to probe locality and reasoning Sephrase and SQA personas containing rephrased variants of changed to check generaliza- tion of edits beyond memorization. In particular, utilize GPTT-4omiini to provide additional question-ansee -nswer varia- changed by enforcing the language model to either directly rephrase
locality and SQA
generalization of edits, we also provide SQA
SOA
tions of SQA
SOA SOA changed to produce rephrased or directly generate SQA Spesonus by mimicking various difffernnt "personas" (e.g. "detective", "pirate", "philosopher" defined by a de- SQA tailed description, see supplementary). For both rephrased personu host-hoc filtering is applied ensure that sub-
Impact of Language Model Choice QA Genera- tion The quality of generated question-answer pairs in WikiBi LikiBiggt Bigiitt depends in part on the underlying language model used during generation. Before we conduct further experiments, we begin by examining the sensitivity of the generation process to different L.MSs by comparing out-
and SDA

---

Understanding the Limits of Lifclong Knowledge Editing in LLMs
Nunter of Suuplss
Num. Samples by Timestamp
Averag Accusnacy
Avg. Accuncy by Timestamp
Pres Trrad
Nunter Quppiins
d) Subject Count
c) Object Count
Sabjoet Views
1503
el Obfect Views
Tonal
1038-
1992
Miaral/75
2000
596-
593
Geraza-18 -18
pi0 lok 1o0 Iow aoa 2066
2004
100
Nee p0i0 201s nox Dian
XG
16
20
16
15
10
20
Fast Tamestupp
<) Tempral Resolution of Extracted Edits
b)) Model Performance by Objectt Count
1) Model Performance by Objject Views
Mas
Stare of Sangics
Desod
Meat Accuncy
Moas Aucuney
Spectfic
T?
73
21
14
16
2
15
12
14
Specific
Extracted
Figure 2. Analysis and Exploration of Real-World Factual Updates in WikiBigEdit. (a) Distribution of ttmesstamps associated with factoids where temporal information was available. For real-world updates, facts cover both historical, but especially more recent context. (b) Average accuracy of five LLMs evaluated across facts broken down by timestamps. We find that contrasted against facts clearly preceding knowledge cutoffs (pre-2023), there is a notable drop in average capabilities, highlighting the need to both address edits on pretraining context, but especially the inclusion of recent facts or corrections. (c) Distribution edits extracted for each timestep and their corresponding dates. While the majority of edits align with their respective timestep, edits in larger parts also incorporate changes of information associated with primarily "older" time intervals. (d-g) Real-world edits cover both highly specific facts (measured on subject and object counts in pretraining corpora) well as general, common knowledge modifications. (h-i) Especially highly specific edits as defined by pretraining concept frequency does performance go down.
Update Triplet
Tzo County Reglonal Airport stote of use in use
Understanding the temporal resolution of real-world fac- tual updates. To assess the temporal resolution of knowl- edge edits in our benchmark, we analyze timestamps for factual triplets where date information is available (10,409 -enswern- pairs). As shown in Figure (a), updates strongly focus on recent events, though many also modify facts predating the edit dates. Figure (c) further reveals that extracted updates are not strictly confined to their respective timesteps, underscoring the dual challenge of integrating both new knowledge and modifying existing facts from pretraining corpora. When stratified by knowledge cutoff (pre-2023 vs. 2023/2024), all tested LLMs perform worse on post-cutoff facts (Cheng et al., 2024) (b), highlighting the critical need for effective integration of new knowledge.
GPT-3.5
What is the current state of use for Lea County Regional Airpott? state of Lea County Regional Airport? What is the current state of use of Lea County Regional Airport?
GPT-4o-mini
What is
GPT-4o
Mhop Tuple
2004 VS75 discoverer- Marc Duic gender male
GPT-3.5
Who the discoverer inventor 2004 VS75?
GPT-44-mini
What is the sex or gender of the discoverer of 2004 VS75?
GPT-4o
What is gender of the discoverer or inventor 2004 VS75?
Table 3. Comparison of generated questions across different lan- guage models for simple factual update (top) and a multi-hop reasoning example (bottom). While all models perform well on the single-hop case, GPT-3.5 struggles with compositional reasoning, often producing incomplete multi-hop questions. GPT-4o-mini and GPT-4o generate more complete and semantically accurate queries.
Specificity of factual updates. We assess the specificity of knowledge edits by distinguishing well-known from highly specialized facts, using two measures: (1) entity frequency from the Infinigram API (Liu et al., 2024a), and (2) Wikipedia view counts (Jan 2023-June 2024) (Mallen et al., 2023). As shown in Figure 2 (d-g), WikiBigEdit includes both general (e.g., Which continent Krasnodar a part of?) and highly specific questions (c.g.. Who was the doctoral advisor of Omar Khayyám?). To evaluate how specificity affects performance, we test five LLMs (Llama- 2-7b, Llama-3-8b, Mistral-7b, Gemma-7b, xGen-7b) and correlate their accuracy. As shown in (h), lower entity fre- quency correlates with reduced performance, consistent with Udandarao et al. (2024); Cheng et al. (2024), while view counts (i) show less consistent trends. Specificity appears primarily driven by the object rather than the subject. See supplementary for details.
puts from GPT-3.5-turbo, GPT-4o-mini, and GPT-4o. For both the Update and Locality sets, which are directly de- rived from structured subject-relation-ojjeer triplets, model choice has negligible impact. Even the less capable GPT- 3.5-turbo consistently produced well-formed, semantically correct questions. As shown in Table 3, the outputs across all models for such simple factual prompts are stylistically com- parable and accurate. In contrast, the Multi-hop and Persona sets were more sensitive to model choice, as they require compositional reasoning and stylistic transformation. For instance, GPT-3.5-turbo often struggled to combine facts effectively in multi-hop settings, producing incomplete or ambiguous questions. GPT-4o-mini and GPT-4o, in com- parison, reliably produced complete and coherent outputs. These observations confirm our selection for GPT-4o-mini to generate Multi-hop and Persona sets due to its strong balance between generation quality and computational effi- ciency.
5

---

Understanding the Limits of Lifclong Knowledge Editing in LLMs
AikiBigEdit - Analysis Summary
Rephrase
When evaluating LLMs on a large number of real-world factual updates, performance is at best mediocre, but especially low for highly specific and recent facts. For LLMS to serve accurate and timely knowledge bases, the ability to edit knowl- edge across historic, recent specific context at scale is thus crucial.
Personas
Mothods
100%
75%
Pre-Edit
50%
LORA
2.6
0%
Update
LoRA Merge
MEMT
WISE
RAG
Mhop
4. Understanding the Limits of Lifelong Knowledge Editing in LLMs
Locality
Leveraging WikiBigEdit, we evaluate the ability of life- long knowledge editing techniques to incorporate large- scale factual updates. Section 4. presents a comprehen- sive application of various lifelong editing approaches and compares them to alternative methods like continual fine- tuning and retrieval augmentation. Section 4.2 analyzes the specific limitations of knowledge editing techniques, with Section 4.3 and Section 4.4 providing detailed evaluations of retrieval augmentation and continual finetuning.
Figure 3. Large-Scale Lifelong Knowledge Editing on WikiBigEdit. Comparison of knowledge editing techniques and other standards for model modification (retrieval-augmmetted generation, RAG, and continual finetuning using LoRA + merging). Results reveal that RAG vastly outperforms specialized knowledge editing techniques (at higher inference cost). At equivalent inference, simple continual finetuning consistently improves on advanced editing techniques at scale.
4.1. Lifelong Knowledge Editing on WikiBigEdit
Experimental Details. All experiments are performed on a compute cluster equipped with Nvidia A A100 and H100 GPUs, leveraging PyTorch (Paszke et al., 2019) and building on the EasyEdit codebase (Zhang et al., 2024). All eval- uations are conducted on five state-of-the-art LLMss Llama- 2-7b (et al., 2023) Llama-3-8b (et al., 2024a), Mistral- 7b (Jiang et al., 2023), Gemma-7b (Team et al., 2024a) and xGen-7b (Nijkamp et al., 2023). To representatively study knowledge editing, we leverage the following approaches: ROME (Meng et al., 2023), R-ROME (Gupta et al., 2024a), MEMIT (Meng et al., 2022), and WISE (Wang et al., 2024a). For our eetrieval-augmeette store update question -answrr -nswer pairs in memory alongside as- sociated key embeddings generated by embedding questions with the cal-m--etab---ee embedding model (Song et al., 2020), the best general eeteence-mmeedding model provided in the SBERT repository (Reimers& & Gurevych, 2019). Dur- ing inference, the top-2 most similar entries are retrieved from memory efficiently using the Annoy solver (Bern- hardsson, 2018). This setup ensures efficient retrieval and supports multi-hop questions with minimal computational overhead. For our continual finetuning experiments, we explore variations of low-rank adaptation using LoRA (Hu et al., 2021) alongside simple interpolation-base bbased model- merging (Wortsman et al., 2022; Stojanovski et al., 2022: Marouf et al., 2024) which has shown to perform competi- tively even when compared to more involved merging tech- niques (Roth et al., 2024; Dziadzio et al., 2024). Adapters are trained for 10 epochs on each timestep. Additional de- tails are provided in the corresponding subsections and the supplementary.
To begin, Figure 3 presents a high-level comparison of the average accuracy across all eight timesteps [U1.... Ug] for all five LLMs listed above. After incorporating factual changes of a timestep, evaluation is conducted across all five evaluation axes spanned by respective evaluation data SQA SOA cQA SQA rephrased personas localityy mmhop' and a performance evalua- tion on the actual edit data SQA and timesteps are averaged and visualized.
updats Results across all models
RAG (green line) proves to be the most effective approach, significantly outperforming knowledge editing techniques across evaluation axes, nearly tripling accuracy on training edits. This result is expected, as memorizing previously seen edits makes solving the task straightforward. Mem- orization also boosts performance on generalization and locality metrics, commonly used in knowledge editing lit- erature. However, compositional reasoning on multihop questions sees only slight improvements, as expected.
generation experiments, we
RAG naturally trades performance increases for longer inference times. However, with effective approximate solvers and high-quality sentence embeddings, we find that inference time at most doubles after processing all of Wiki LikiBigEdis (c.f. Figure 16). Even so, knowledge edit- ing techniques like WISE and MEMIT, specifically designed to factually update LLMs, cannot match simple low-rank adapter-based finetuning (LoRA-FT) at scale. Moreover, with interpolation based model merging (LeRe-Merge), commonly used in training and finetuning LLMs (Yang et al., 2024a; Roth et al., 2024; Team et al., 2024a;b; Dzi- adzio et al., 2024), consistent gains are observed across
6

---

Understanding the Limits of Lifclong Knowledge Editing in LLMs
fits. While WISE avoids catastrophic failure and integrates new facts over longer update sequences, its performance steadily declines within the first 10K updates, ultimately converging to pre-update knowledge levels. This suggests that by avoiding direct model modifications, WISE struggles to retain updates over time, making it unsuitable for large- scale sequential integration, especially compared to retrieval augmentation and simple continual fine-tuning, which incur additional inference cost.
1.9
LORA (Avg)
0.5
Edit Accuracy
RAG (Avg.)
MEMIT
R-ROME
0.4
ROME
WISE
0.2
Pre-Edi (Avg.)
2000
4000
6000
8000
10000
Number of Edits
Figure 4. Performance Decay of Knowledge Editing Techniques. Update accuracy over first 10K edits for knowledge editing methods compared to simple baselines that show higher perfor- mance (LoRA, RAG). Local modification methods collapse early (MEMIT, ROME, R-ROME), and even those specifically designed for large volumes of edits (WISE) converge to pre-edit accuracy.
Key Takeaway
Knowledge editing is reliant on local weight mod- ification and collapses quickly. Even methods de- signed for lifelong editing converge to pre-update performance early, So large-scale lifelong editing requires significantly improving existing methods.
all evaluation axes. These results strongly suggest that at scale, standard knowledge editing techniques fall short even when compared to straightforward continual finetun- ing approaches commonly employed for general continual pretraining (Ibrahim et al., 2024; Roth et al., 2024).
4.3. Retrieval Augmentation for WikiBigEdit
As shown in Figure 3, retrieval-augmentte generation (RAG) demonstrates significant improvements over di- rect knowledge editing techniques. While expected for performance on training edits, the increased perfor- mance across all evaluation axes, including generality, is somewhat surprising, as prior retrieval approaches like GRACE (Hartvigsen et al., 2023) have shown limited gener- alization capabilities (Wang et al., 2024a). However, on real- istic edits and at scale, we find that efficiently implemented retrieval- surreval--ugmmetattion outperforms any editing method (see Section 4 and supplementary for details).
Key Takeaway
Updated via WikiBigEdit, RAG significantly outperforms all approaches at double the inference time; though expectedly struggles with multi-hop reasoning. Given a fixed inference budget, we find that at scale, knowledge editing approaches evalu- ated across all standard editing evaluation axes are actually matched or outperformed by simple contin- ual finetuning, especially using merging.
Looking the detailed results of the RAG baseline on Wi k Bi gEdit t in Figure 5, the improvements across all evaluation axes (except multi-hop reasoning) are significant across 500K+ updates. The top row shows the average ac- curacy on the current timestep, with bold lines representing average performance across five models (light background lines). The bottom row illustrates average performance across all timesteps after training on the current timestep's 's edits. As noted previously, RAG achieved near-perfect accuracy on Supaate by simply retrieving the exact stored questions from memory; exhibiting no sign of "forgetting" on trained edits (c.f. bottom row, leftmost). Additionally, edits remain highly localized since no weight updates are performed. Interestingly, while RAG also maintains high av- erage performance on generalization-bat based evaluation axes ("rephrasing" and "personas"), achieving average accura- cies of 78.64% and 66.16% respectively, RAG still exhibits symptoms akin to catastrophic forgetting encountered in continual learning and knowledge editing (Kirkpatrick et al., 2017: Zenke et al., 2017; Wang et al., 2024a). Here, perfor- mance on older timestep data decays with every new large update batch (bottom, second and third), as the retrieval
The following sections provide a more detailed breakdown of specific failure scenarios and drivers.
4.2. The Limits of Knowledge Editing at Scale
This section provides an in-depth analysis of knowledge editing applied to long and realistic update sequences, with results shown in Figure 4. Using Llama-2-7b as the under- lying LLM, we present the first 10K update steps from the initial timestep in Wik milkiBigh WikiBigEdit for clarity. A complete version with all models is available in the supplementary.
The accuracy of ROME, R-ROME, and MEMIT - local knowledge editing techniques that locally modify specific model weights- - rapidly degrades within the first few hun- dred updates, leading to model collapse. This fallure on large-scale, sequential knowledge editing aligns with recent findings (Gupta et al., 2024b; Yang et al., 2024b) and high- lights difficulties for lifelong knowledge editing at scale. Even without direct model edits, methods like WISE - de- signed for larger update volumes by storing adapted weight matrices for retrieval during inference- show limited bene-
7

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Update
Kephanse
Personcs
Milop
Localiy
Pr--tdh
0.5
1.5
03-
Mi
1LAMAN2-1
Gesau
0.4
644
LLAMN.36
--.
X0a>18
169
204
120
Auc. Dillooeeee
Tixxstop::
Trrsoep
Trrseep
Tireeee
Tinetep
Trweor TV
Tiwwrr
13 210
75
71
20
Staber Baita 03)
Niabler of Edit (53
Namber Edt [K]
Nantert of Eaito
Nantier Rall
Figure 5. Retrieval-Augmeented Generation for Lifelong Knowledge Editing. (Top) Mean batch accuracy for evaluations: update, rephrase, personas, multi-hop and locality across all models. (Bottom) Accuracy changes for preceding timesteps after training on a given timestep, highlighting forgetting symptoms on generalization tasks, and marginal henefits at best for reasoning over edits.
memory becomes more and more obfuscated with context irrelevant for data from particular timesteps.
Retrieval Acewacy
Anowe Ave. if i'Coneeyy Retrieved
1.3
1.3
0.5
Finally, while RAG outperforms knowledge editing ap- proaches on standard measures (locality, generalization, and overall edit performance), its multi-hop reasoning accuracy remains near pre-update levels (second from right in Fig- ure 5). Figure 6 reveals the bottleneck: retrieval accuracy is low likely due to the absence of the middle subject in the multi-hop question- with correct retrieval rates below 50% for the first hop and even lower for the second. Only 10% of evaluation samples retrieve both facts correctly. Even when both hops are retrieved, answer accuracy remains near single-hop levels (i.e. solving multi-hop questions using only information from just the first hop). as models struggle to combine retrieved information without dedicated training for reasoning tasks. While advanced methods like Hip- AG (Gutiérrez et al., 2025) may help, they typically require longer contexts or higher compute; we leave their integration to future work.
Come
0.4
044
0.9
0.
100
200
315
500
100
Hop Kesicon
Rayz Ros
Figure 6. Multi-hop RAG. (Left) Retrieval accuracy for individ- ual and both hops, highlighting significant challenges in retrieving both relevant facts simultancously. (Right) Answer accuracy con- ditioned on correct retrieval, showing that even when both hops are retrieved, models struggle to combine the information.
vantages extend to specialized knowledge editing for LLMs. As shown in Figure 3, simple non-specialized finetuning over all Wiki WiBiiiiiit updates outperforms specialized knowledge editing techniques across all standard evaluation axes, with gains further amplified by interpolation based model merging. These findings emphasize the need to re- visit design choices in knowledge editing, evaluate on larger, realistic benchmarks, and improve baseline comparisons.
Key Takeaway
Simple RAG outperforms much more involved knowledge editing techniques across all editing eval- uation axes at larger scale and real-world edits, How- ever, performance on evaluation over older edits still decays, and reasoning over edited facts shows marginal benefits at best.
Finally, we provide detailed perspective on lifelong knowl- edge editing using continual finetuning in Figure 7 (rank-4 LoRA; additional results in the supplementary"). As shown in the top row, performance remains competitive in early timesteps but steadily declines as updates accumulate, even- tually converging to or even falling below pre-update ac- curacy, particularly for locality and multi-hop reasoning. This aligns with expectations, catastrophic forgetting per- sists even in low-rank adaptation (Thede et al., 2024; Roth et al., 2024). However, for the first 100K updates, contin- ual finetuning shows notable success in integrating edits.
4.4. Continual Finetuning
To conclude, investigate continual finetuning using low- rank adaptation (LoRA) as simple yet effective method to incorporate large volumes of new facts. While recent works (e.g., Thede et al. (2024); Roth et al. (2024)) have demonstrated the benefits of continual low-rank finetuning for general continual learning tasks, we show that these ad-
'High-rank configurations exhibit significant instability, includ- ing catastrophic failure Mistral and poor performance across other models.
8

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Update
Rephaose
Personcs
Milop
Locality
0.8
0.0
Pre-dr
(AAAA Meppo
Pose BBt
(140A-713
8.4
8.4
1.4
Model Avg
Mical 11
21AMA-2-71
Commu-71
L1AMA3-1B
XGas-88
0.8
180
200
300
-01
500
200
189
204
4%0
4.6
100
30
300
400
300
Tinearp
Tioeag
Acc. www.Dieeesee
-12
-10
-15
-40
Tixesty T3
Timeapp
Tixesg
Timeap
-12
-6-
25-
-2-
Tineag T4
Timwag TV
-11
20
20
Pau-a: (LRRADD
Pua-6ii (LattA-Monp))
14
25
Staber Bait 03)
Ntabler of Edita (53
Nanber Edit [K]
Nanter of Eaito
Nantier Tatl
Figure 7. Continual Finetuning for Knowledge Editing, (Top) Mean batch accuracy for update, rephrase, personas, mhop, and locality sets for continual LoRA finetuning (LoRA-FT) with model merging (LogR-Merge). (Bottom) Accuracy differences for individual timesteps. LoRA-FT degrades over time, while merging ensures consistent editability across all k i Biggdit.
These results highlight its viability for mid-term knowledge integration but also its limitations at scale.
5. Conclusion
This paper explores lifelong knowledge editing for large language models at practical scale. We introduce Wi WikiBigEdi quisition pipeline that extracts real-world factual updates from Wikidata, enabling lifelong expansion. In its first version, WikiBigEdit, contains over 500K QA pairs for large-scale lifelong knowledge editing, complemented by comprehensive evaluation suite covering standard knowl- edge editing criteria, reasoning, and generalization.
Interestingly however, as we couple continual LoRA fine- tuning with simple model merging (after training of each timestep, current adapter weights are simply merged into preceding adapter weights using an interpolation weight of 0.25), achieve significantly improved editing per- formance across Wi Biiiti While LoRA-Merge ini- tially shows lower performance on shorter update sequences, it demonstrates superior stability as updates accumulate; and begins to notably outperform simple continual finetuning (and thereby also any specialized knowledge editing tech- nique) after approximately 100K edits on all evaluation sets, maintaining consistent performance without signifi- cant degrudation. Importantly, LoRA-Merge shows at most marginal evidence of catastrophic forgetting, and in parts even exhibits positive backward transfer (where training on a current timestep actually improves performance on all ed- its seen previously)- - see Figure 7 bottom row. These results reinforce the effectiveness of model merging for general con- tinual finetuning and pretraining tasks (Stojanovski al., 2022; Thede et al., 2024; Roth et al., 2024), highlighting its promise for large-scale knowledge editing.
benchmark with an automated data ac-
Leveraging Wik ki Bi BigEdit, we conduct an extensive ex- perimental study in which we identify key limitations of existing knowledge editing approaches such as ROME, MEMIT, and WISE, which struggle with scalability and stability in large-scale update settings. Our results also demonstrate the efficacy of retrieval-augee aarievae--egeeeeeedd generation (RAG) when no restriction inference time is given and highlight continual finetuning coupled with model merging as a competitive alternative at an equivalent inference cost.
We believe that Wi k ii gEdit provides an important ref- erence to understand the transfer of knowledge editing meth- ods to real-world scale and facts, as well as suggestions to- wards more complete baseline studies during development of such specialized techniques.
Key Takeaway
A Acknowledgements
Generic adapter-based finetuning matches special- ized knowledge editing for mid-level knowledge editing; though exhibits catastrophic forgetting at scale. However, equipped with simple model merg- ing, we find consistent inclusion of new facts across all of WikiBigEdii
LT and KR thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. KR also thanks the European Laboratory for Learning and Intel- ligent Systems (ELLIS) PhD program for support. We are grateful for support by the Carl Zeiss Foundation, project "Certification and Foundations of Safe Machine Learning Systems in Healthcare". This work was partially funded by the ERC (853489- DEXIM) and the Alfried Krupp von Bohlen und Halbach Foundation, which we thank for their generous support.
This suggests better base- line comparisons and studies at scale for the develop- ment of specialized knowledge editing techniques.

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Impact Statement
Chen, Q.. Zhang, T., He, X., Li, D., Wang, C., Huang. L., and Xue, H. Lifelong knowledge editing for Ilms with retrieval-augmenned continuous prompt learning, 2024. URL httpps:// arxiv. oro/oogg//.....ooooo............ htt///bbb/2200... 20002 03279.
Maintaining factual accuracy in large language models (LLMs) critical, as these models are increasingly relied upon in domains where misinformation or outdated knowl- edge can have serious societal consequences. By focusing on the lifelong integration of factual knowledge, this pa- per directly addresses the challenge of ensuring that LLMs remain up-to-date, reliable, and accurate over time. The pro- posed WikiBigg Bidit benchmark provides a valuable tool designing and evaluating methods that enhance factual- ity, offering a scalable and realistic framework grounded in real-world knowledge evolution. By advancing techniques for continual factual updates, this work contributes reduc- ing the risks of outdated or incorrect information, promoting trust and accountability in the use of LLMs for applications in education, healthcare, jourmalism, and beyond.
Cheng. J., Marone, M., Weller, O., Lawrie, D., Khashabi, D., and Durme, B. V. Dated data: Tracing knowl- edge cutoffs in large language models. In First Con- ference on Language Modeling, 2024. URL https: / //openreview net
jy 6.
Colombo, P., Pires, T. P., Boudiaf, M., Culver, D., Melo, R., Corro, C., Martins, A. F., Esposito, F., Raposo, V. L., Morgado, S., et al. Saullm-7b: pioncering large lan- guage model for law. arXiv preprint arXiv:2403.0388. 2024.
Dziadzio, S., Udandarao, V., Roth, K., Prabhu, A., Akata, Z. Albanie, S., and Bethge, M. How merge your multimodal models over time? arXiv preprint arXiv:2412.06712. 2024.
References
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary. S., and Wang. T. Ms marco: A human generated machine reading comprehension dataset, 2018. URL https: //arxiv...vvvv// org/abs/ 611 611.611111 09268.
et al., A. G. The Ilama herd of models, 2024a. URL https: httpss://axxvvvorre at///bbs//22 /207 21783.
et al., D.-A. Deepseek llm: Scaling open-source lan- guage models with longtermism, 2024b. URL https:
/ a/aaiiiv.. wrt.ass//rga//////.e///.....asa http:///b////.a...e// 22401.02954.
Bao, Z., Chen, W., Xiao, S., Ren, K., Wu, J., Zhong, C., Peng, J., Huang, X., and Wei, Disc-medllm: Bridging general large language models and real-world medical consultation. arXiv preprint arXiv:2308. 14346, 2023.
et al., D.-A. Deepseek-v2. strong, economical, and effi- cient mixturee-of-xxpertss language model, 2024c. URL 04434..
https: /ttpps//arzii..orr/aba..
technical et al., D.-A. Deepseek-v3 technical report, 2024d. URL 19437.
Bernhardsson, E. Annoy: Approximate Nearest Neighbors in +++/Python, 2018. URL https://ppt httpppppppooo/o project/annoy/ / Python package version 1.13.0.
et al., H. T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL btttss//arxiv..om
2307.
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J.-B., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang. S., Maggiore, L.,, Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., and Sifre, L. Improving language models by retrieving from trillions of tokens, 2022. URL https: arxiv. org/ abs 2212 04426.
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, and Auli, M. Eli5: Long form question answering, 2019. URL https::p//// www.xii..rg/aabb... 0911009997799...... 09190.
Franzmeyer, T., Shtedritski, A., Albanie, S., Torr, P., Hen- riques, F,, and Foerster, J. N. HelloFresh: LLM evalua- tions on streams of Real-Wor World human editorial actions across X community notes and wikipedia edits. June 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Garg, S., Farajtabar, M., Pouransari, H., Vemulapalli, R., Mchta, S., Tuzel, O., Shankar, V., and Faghri, F. Tic-clip: Continual training of clip models, 2024. URL https: //arxiv. org/ abs 221...222...2.2
Askell, A., Agarwal, S., Herbert-Vosse A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCan- dlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Ge, X., Mousavi, A., Grave, E., Joulin, A., Qian, K., Han, B., Areftyan, M., and Li, Y. Time sensitive knowledge editing through efficient finetuning. arXiv [cs.CL], June 2024.
10

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Ghosh, A., Dziadzio, S., Prabhu, A., Udandarao, V., Al- banie, S., and Bethge, M. Onebench test them all: Sample-level benchmarking over open-ended capabilities. arXiv preprint arXiv:2412.06745. 2024.
Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L. Patching open-vocabulary models by interpolating weights, 2022. URL BtLhtts:/// arxiv..iiivv///... org/ abs 2200. 05592.
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi- Yu, J.. Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot learning with retrieval augmented language models, 2022. URL https:// arxiv. org/ abs///bb//// abwaaaas/ss0/ 2200 03299.
Gui, 7,, Sun, S., Li, R., Yuan, J., An, 7.,, Roth, K., Prabhu, A., and Torr, P. knn-clip: Retrieval enables training-free segmentation on continually expanding large vocabular- ies. arXiv preprint arXiv:2404.09444 2024.
Gupta, A., Baskaran, S., and Anumanchipalli, G. Rebuild- ing ROME: Resolving model collapse during sequential model editing. arXiv [cs. [cs.CL], March 2024a.
Jang, J., Ye, S., Lee, C., Yang. S., Shin, J., Han, Kim, G., and Seo, M. TemporalWiki: A lifelong benchmark for training and evaluating Ever-Evolving language models. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Pp. 0237-6250 Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics.
Gupta, A., Rao, A., and Anumanchipalli, G. Model editing at scale leads to gradual and catastrophic for- getting, 2024b. URL https://arxis: org/ abs/ 2401. 07453.
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., and Su, Y. Hipporag: Neurobiologically inspired long-term mem- ory for large language models, 2025. URL https: //arxiv.org abs/ 24052 14831.
Jeong, D. P., Garg. S., Lipton, Z. C., and Oberst, M. Med- ical adaptation of large language and vision-language models: Are we making progress? arXiv preprint arXiv:2411.04116 2024.
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.- W. Realm: Retrieval-augmenntd language model pre- training, 2020. URL 2002. 08909.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L.., Lavril, T., Wang. T., Lacroix, T. and Sayed, W. E. Mistral 7b. CoRR, abs/23 10.06825, 2023. doi: 10445550AARXII.2310.6662
abs/
Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, November 2023.
URL https: 2310 221....82..0..0.......
/ //doi. http:/11.4888850 400444550////
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https:
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kuk- liansy, D., Cohen, V., Scialom, T.. Szpektor, I,, Hassidim, A., and Matias, Y. True: Re-evaluating factual consis- tency evaluation, 2022. URL https:tt////.. httttttps///aiitii http:://xxxvvvoooeee abs/ 2204.2204.. 04991.
//arxiv. org abs http//ass11755 1t1//17050000 03551.
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and tau Yih, W. Dense passage retrieval for open-domain question answering, 2020. URL https: //arxiv. wwwpxiv.org//ee
www..orgggabaa.. abs 2200400044440 04906.
Hsueh, C.-H., Huang, P. K.-M., Lin, T.-.., Liao, C.WW,, Fang, H.-.., Huang, C.-W., and Chen, Y.-N. Editing the mind of giants: An in-depth exploration of pitfalls of knowledge editing in large language models. arXiv [cs. [cs.CL], June 2024.
Khodja, H. A., Béchet, F., Brabant, Q., Nasr, A., and
Lecorvé, G. Wikifactdiff: A large, realistic, and tem- porally adaptable dataset for atomic factual knowledge update in causal language models, 2024. URL https: //arxiv. org abs /2//2/400 14...113666
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: /arxiv. org/abs/ 22w.. 09685.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J.. Ramalho, T., Grabska-Barwinska. Barwinska. A., Hassabis. D.. Clopath, C., Kumaran, D., and Hadsell, R. Overcom- ing catastrophic forgetting in neural networks. Pro- ceedings of the National Academy of Sciences, 114 (13):3521-3522 March 2017. ISSN 1091-6490. doi: 10.1073/pnas. 18351114 URL http:/ //xx dxidoi.
Ibrahim, A., Thérien, B., Gupta, K., Richter, M. L., An- thony. Q.. Lesort, T., Belilovsky, E., and Rish, I. Simple and scalable strategies to continually pre-train large lan- guage models, 2024. URL https: /aaaxiivvvoo....a org/ abs/2403. 08763.
1053//0.1033/poo (6118351144

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Kozal, J., Wasilewski, J., Krawezyk, B., and Wožniak, M. Continual learning with weight interpolation, 2024. URL https: /tw.xiiivvv..//// org/ wtw./bs//////oo/o/e//o///aaaaeaa abs/2404 4404. 04002.
Liu, J., Min, S., Zetrlemoyer, L., Choi, Y., and Hajishirzi, H. Infini-gram: Scaling unbounded n-gram language models to trillion tokens, 2024a. URL https: ptttp://arce //arxiv. org
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, L., Kel- сey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Liu, J. Yu, P., Zhang, Y., Li, S., Zhang, 7., and Ji, H. EVEDIT: Event-based knowledge editing with deductive editing boundaries. February 2024b.
Ma, H., Xu, W., Wei, Y., Chen, L.,, Wang, L., Liu, Q. Wu, S., and Wang, L. Ex-fever: A dataset for multi- hop explainable fact verification, 2024. URL https: / a/aaiiiv..oo wwt.:/s..ggbbab..t..o...... 23100. 09754.
Lai, J., Gan, W., Wu, J., Qi, Z., and Philip, S. Y. Large language models in law: A survey. Al Open, 2024.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Inves- tigating effectiveness of parametric and non- non-parametric memories, 2023. URL https: http:s///axxvvvo/// / 2212 2212.1051..
Levy, O., Seo, M., Choi, E., and Zettlemoyer, L, Zero- shot relation extraction via reading comprehension. In Levy, R. and Specia, L. (eds.), Proceedings of the 21st Conference Computtational Natural Language Learn- ing (CoNL.. 2017), pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguis- URL https://
Manakul, P., Liusie, A., and Gales, M. SelfCheckGPT: Zero-resource black-box hallucination detection for gen- erative large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/1120233 emnlp-main.557. URL https: /t//aclanntin http:::://aaaaaholooyyy.. org/2023, emp p- mnmmain.... .5.51/
tics. doi: 10. 10.18653/ selanthology.ooze (ttgg//17-1103
Lewis, P., Perez, E., Piktus, A., Petroni, F,, Karpukhin, V., Goyal, N., Küttler, H., Lewis, M.. Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmenned genera- tion for nowleddg-intenstee
Marezak, D., Twardowski, B., Trzciński, T., and Cygert, S. Magmax: Leveraging model merging for seamless con- tinual learning, 2024. URL https:://///e /tapss//axxvvvvooo abs/2407.06322
nlp tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9459-9474 Curran Associates, Inc., 2020. URL https: htttps://prcceeddnggs nee.eeee cc Seaaperr_fillessppppe////0 3b192220000577701122222228222820 pdf.
Marouf, I. E., Roy, S., Tartaglione, E., and Lathuilière, S. Weighted ensemble models are strong continual learn- ers, 2024. URL wtt.tttts///aatiiiie http///xxvv..r/abbs..ss/e ab3/2312.. 08977.
Paper.
Li, B. Z., Liu, E., Ross, A., Zeitoun, A., Neubig, G., and Andreas, J. Language modeling with editable external knowledge, 2024. URL
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. arXiv [cs. CLJ, October 2022.
org/ abs/
2406. 11830.
Li, J., Armandpour, M., Mirzadeh, I., Mehta, S., Shankar, V., Vemulapalli, R., Bengio. S., Tuzel, O., Farajtabar, M., Pouransari, H., and Faghri, Tic-Im: A web-scale benchmark for time-continual 11m pretraining, 2025. URL https: w.a.iiiv.. org/ halgg//b///e 2504 02107.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locat- ing and editing factual associations in gpt, 2023. URL https: http://axvvvvooo .ww.ivoorr//bbae
Min, T. D., Mancini, M., Alahari, K., Alameda-Pinede X., and Ricci, E. On the effectiveness of layernorm tuning for continual learning in vision transformers, 2023. URL. https: / /aaaaaiivvv///.////aa////aaa/.va wwtp////rrgaa/b////a/a/oa/
Liška, A., Kočiský, T., Gribovskaya, E., Terzi, T., Sezener, E., Agrawal, D., de Masson d' Autume, C., Scholtes, T., Zaheer, M., Young. S., GillenannMMeMahon E., Austin, S., Blunsom, P., and Lazaridou, A. StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models. May 2022.
09610.
Mitchell, E., Lin, C.,, Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. arXiv [cs.LG], October 2021.
12

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. Memory-hased model editing scale, 2022. URL https: /aaxiv. org/ 2abs/2205 06520.
Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. Mpnet: Masked and permuted pre-training for language under- standing, Advances in neural information processing systems, 33: 33:16857-16867, 2020.
Nijkamp, E.,, Xie, T., Hayashi, H., Pang. B., Xia, C., Xing, C., Vig, J., Yavuz, S., Laban, P., Krause, B., et al. Xgen- 7b technical report. arXiv preprint urXiv:2309.03450, 2023.
Stojanovski. Z., Roth, K., and Akata, Momentum-basee weight interpolation of strong zero-shot models for con- tinual learning, 2022. URL abs / 2211221111031 03186.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.., Desmaison, A., Kopf, A., Yang, E., Detoto Z.,, Raison, M. Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, seeh-pefforaece deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, I.., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024a.
Team, G., Riviere, M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.00118 2024b.
Peng. H., Wang. X., Li, C., Zeng. K., Duo, J., Cao, Y., Hou, L., and Li, Event-level knowledge editing. February 2024.
Prabhu, A., Cai, Z., Dokania, P., Torr, P., Koltun, V., and Sener, O. Online continual learning without the storage constraint. arXiv preprint MxXiv:2355009253 2023a.
Team, L., Modi, A., Veerubhotla, A. S., Rysbek, A., Huber, A., Wiltshire, B., Veprek, B., Gillick, D., Kasenberg, D., Ahmed, D., et al. Learnlm: Improving gemini for learning. arXiv preprint arXiv:2412.16422 2024c.
Prabhu, A., Hammoud, H. A. A. K., Dokania, P., Torr, P. H. S.. Lim, S.-N., Ghanem, B., and Bibi, A. Com- putationally budgeted continual learning: What does org/ abs/
Thede, L., Roth, K., Hénaff, O. J., Bethge, M., and Akata, Z. Reflecting the state of rehearsal -feeee continual learning with pretrained models, 2024. URL https: org/ abs / 2406. 09384.
matter?, 2023b. URL https: 2303. 11165.
Prabhu, A., Udandarao, V., Torr, P., Bethge, M., Bibi, A., and Albanie, S. Lifelong benchmarks: Efficient model evaluation in an era of rapid progress. arXiv preprint 19X14:2402.19472, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave. E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https: http///axxvvv///ee/ee Abs/2302.139)
Ramé, A.,
Vieillard, N., Hussenot, L.,, Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. Warm: On the benefits of weight averaged reward models, 2024. URL https: //arxiv.orr org/ abs/ / 2401 12187.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Is multihop qa in dire condition? measuring and reducing disconnected reasoning, 2020. https: / www.xi..org/ab///00000 http://bbb/2000000788
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., and Re- itter, D. Measuring attribution in natural language gen- cration models, 2022. URL https: http://xxvvvrrro/ abs/2112.128700
Udandarao, V., Prabhu, A., Ghosh, A., Sharma, Y., Torr, P., Bibi, A., Albanie, S., and Bethge, M. No "zero- shot" without exponential data: Pretraining concept frequency determines multimodal model performance. The Thirty-eighth Annual Conference on Newral In- formation Processing Systems, 2024. URL https: / //openreview.o net/ www.moioumimmoreeee dit.mmiiii-VVGixxiii
Reimers, N. and Gurevych, I. Sentence-bert, Sentence em- beddings using siamese bert-networks. Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computational Linguistics, 11 2019, URL https: /ttp:://axxxvvoor/// abs/1908. 10084.
Roth, K., Udandarao,
V., Dziadzio, S., Prabhu, A., Cherti, M., Vinyals, O., Hénaff, O., Albanie, S., Bethge, M., and Akata, Z. A practitioner's 's guide to continual multimodal pretraining. arXiv preprint 2408. /4471, 2024.
Vu, T., Iyyer, M., Wang. X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. Freshllms: Refreshing large language models with search engine augmentation, arXiv preprint arXiv:2310.032 2023.
13

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Wang, J., Jatowt, A., and Yoshikawa, M. ArchivalQA: A large-scale benchmark dataset for open domain question answering over historical news collections. September 2021.
Zheng, D., Lapata, M., and Pan, J. Z. Large language models reliable knowledge bases? arXiv preprint wrx7:2240.113578 2024.
Wang, P., Li, Z., Zhang, N., Xu, Z., Yao. Y., Jiang, Y., Xie, P., Huang, F., and Chen, H. WISE: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv (cs. CLs, May 2024a.
Wang, S., Xu, T., Li, H., Zhang, C., Liang, J., Tang, J., Yu, P. S., and Wen, Q. Large language models for education: survey and outlook. arXiv preprint orXiv:2403.18100 2024b.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Korbliihh, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models, 2022. http::::: /ttp:s/////xvvvve/// org// abs/ 01903.
Yang. E., Shen, L., Guo, G., Wang. X., Cao, X., Zhang, J., and Tao, D. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint 20w:2408.076666 2024a.
Yang. W., Sun, F., Ma, X., Liu, X., Yin, D., and Cheng, X. The butterfly effect of model editing: Few edits can trigger large language models collapse. arXiv [cs.Al], February 2024b.
Yang, Z. Qi, Zhang, S., Bengio, Y., Cohen, W. W. Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering, 2018. URL https://arxiv. orw.lvoorrr/aoo /bs/ 1809. 09600.
Yin, X., Jiang, J., Yang, L., and Wan, X. History matters: Temporal knowledge editing in large language model. AAAL 38(17):19413-1444 March 2024.
Zaken, B., Ravfogel, S., and Goldberg, Y. Bitfit:
Simple parfiiter-effficeent fine-tuning for transformer- based masked language-modeks 2022. URL https: //arxiiv org/ abs /abb//1165 10199.
Zenke, F., Poole, B., and Ganguli, S. Continual learning
through synaptic intelligence, 2017. URL https:// arxiv. owg/v.org///oo abs 1 7003.0.00 04200.
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., et al. A comprehensive study of knowledge editing for large language models. arXiv preprint wxXi::241.002266 2024.
Zhao, J.. Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficier IIm training by gradient low-rank projection, 2024. URL ht https: //tax/iv.rggaa///////////ooota// orgg/bs///233 03507.
//arxiv.
14

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
A. Benchmark Generation
This section provides additional details about the benchmark generation process, focusing on the templates used to create -nestioo-answer question answer pairs from the extracted factual updates. Each subset of the benchmark update, locality, rephrase, personas, and multi-hop-- - is generated using carefully designed prompts tailored to their specific evaluation objectives. Below, we describe the prompt templates and methodologies for generating these subsets.
A.1. Update and Locality Set
The update set serves as the foundation of our benchmark, containing questions derived directly from the extracted factual updates. The locality set, the other hand, evaluates the robustness of knowledge updates by ensuring that unrelated but similar facts remain unaffected.
We employ the following prompt template to generate both the update and locality sets. The prompt includes general task description, detailed instructions, and few-shot examples to guide the model generating high-quality question-answer pairs:
15

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Prompt Template Update/Locality
Given a fact triplet where the answer the object of the triplct. Ensure that the question explicitly references the relation and structured to lead uniquely to the chosen answer, which must be the exact text of the object. Use the description of the relation to improve the specificity and clarity of the question. Make the question specific enough ensure that there is only possible correct answer, eliminating any ambiguity.
format (subject, relation, object) and a description
relation, generate question-answer pair
Instructions for Generation:
1. Construct the Question:
Ensure the question is directly tied to the relation and points exclusively to the selected answer.
Include additional context from the subject to make the question more specific and unambiguous.
Use the description
relation enhance quality of question. Use only exact text of object in answer.
2. Format: Provide the question-answwe -answer pair in the following format:
Q:
A:
Examples:
Fact Triplet: Tumberry Lighthouse, color, white
Relation Description: color the subject.
Q: What is the color of Tumberry Lighthouse? A: white
Fact Triplet: Folsom Library, part of, Rensselaer Libraries
Relation Description: part of this subject; inverse property of "part of" (P361). "has parts the class" (P2670).
Q: What institution does Folsom Library belong to?
A: Rensselaer Libraries
Fact Triplet:
Volt Europa, subsidiary, Volt Netherlands
Relation Description: A company controlled by another company.
Q: What is the name of the subsidiary of Volt Europa in the Netherlands?
A: Volt Netherlands
Fact Triplet: Zonia Baber, educated at, Chicago State University
Relation Description: The institution where the subject received their education.
Q: At which university did Zonia Baber receive her education?
A: Chicago State University
Task:
According to the instructions and examples, provide QA pair for the following fact triplet:
Fact Triplet: 0. 0). 0
Relation Description: 0
A.2. Rephrase Set
The rephrase set is designed to test the generalization of updated knowledge by reformulating the questions from the update set. The reformulated questions retain the same factual context but vary in phrasing, introducing linguistic diversity to the benchmark.
The following prompt used generate the rephrased questions:
16

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Prompt Template Rephrase
You are given a question-answer pair. Reformulate the question such its content remains identical the given answer still the only accurate answer.
Instructions for Generation:
1. Question Reformulation:
Restructure the question while maintaining the same content and context.
Ensure that the answer provided in the original question remains the only correct answer.
2. Response Format: Provide the reformulated question in the following format:
Reformulated Question:
Examples: Original Question: What is the capital the United States?
Answer: Washington, D.C.
Reformulated Question: What city serves the capital of the United States?
Original Question: Who painted the Mona Lisa?
Answer: Leonardo da Vinci
Reformulated Question: By whom was the Mona Lisa painted?
Original Question: When was Declaration Independence signed?
Answer: July 4, 1776
Reformulated Question: On what date was the Declaration of Independence signed?
Task: Original Question: O Answer:
A.3. Persona Set
The persona adds layer of complexity by requiring the model generate questions in the style of distinet personas. Each question from the update set is rewritten based on the characteristics of randomly selected persona.
Persona
Description
Detective
You are a world-weary detective narrating your investigations. Your tone gritty, mysterious, and evocative of noir fiction. Reformulate the following questions as if you're puzzling out clues in a case. Keep the core meaning intact but phrase it in your distinct detective style.
Casual
You are a casual, friendly conversationalist with a relaxed and approachable tone. You're 're curious and engaging, as if you't 'r chatting with a friend over coffee or in a group discussion. Reformulate the following questions sound natural and conversational while keeping the core meaning intact.
Pirate
You are swashbuckling pirate with a flair for colorful language and sea-themed metaphors. Your tone is bold, rough, and full of pirate lingo, as if you't 're recounting tales from the high seas. Reformulate the following questions to sound like they 're being asked by true pirate. Keep the core meaning intact but add a piratical twist.
Philosopher
You are an ancient philosopher, deeply contemplative and wise, always pondering the greater truths of existence. Your tone is reflective, profound, and formal, as if you are crafting dialogue or treatise on the nature of knowledge and events. Reformulate the following questions as they are inquiries befitting philosophical discourse. Keep the core meaning intact but phrase it in your distinct, thoughtful style.
17

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Caveman
You caveman, speaking in simple and primitive language, with short sentences and limited vocabulary. Your tone is direct, straightforward, and reflects the early stages of human communication. Reformulate the following questions in way that matches your basic and minimalistic speech style, while keeping the core meaning intact.
Table 4: Descriptions of the personas used for rephrasing questions with varied styles.
The generation process uses the same structure the rephrase prompt, with the addition of persona-specifie descriptions. Table A.3 provides overview of the personas used, including detective, casual speaker, pirate, philosopher, and stone-age character. For each question, one persona is chosen, and its description and few-shot examples are inserted into the rephrase prompt to guide the generation process.
A.4. Mhop Set
The multi-hop set evaluates the reasoning capabilities of the model by requiring it to combine information from two factual triplets. The questions are designed to omit the middle entity, prompting the model to infer the correct answer by reasoning over both facts.
The following prompt used formulate the multi-hop questions from the paired factual triplets:
Prompt Template Mhop
Given multi-hop fact tuple in the format (entity-O, relation-1, [MASKED-ENTITY--]] relation-2, rellaionn,,,, entity-2) and description of each relation, generate multi-hop question-answer pair where the answer "entity-2." Ensure that the question respects the directional nature of each relation, especially relation-2, so that it logically reflects how the masked entity- relates to entity-2. The question should be structured uniquely lead to entity-2 as the answer.
Instructions for Generation:
1. Construct the Question:
Design the question to identify entity-2 2 by logically combining relation-1 and relation-2.
Respect the directionality of each relation as described: if relation-2 describes relation where WAAKED-ENTTTY'S is the subject and entity-2 is the object (c.g., "is member of"), ensure this direction is reflected in the question.
Avoid reversing the direction of relation-2, and confirm the natural flow between [MASKED MASKED-ENTITYY.. and entity-2.
Include context from entity-O 0 to make the question clear and specific.
Do not reference
in the answer: only entity-2 should be the answer.
2. Format: Provide output in the following format:
Q: [Constructed Question]
A: [entity-2]
Examples:
Fact Tuple: Yasis Malih, place birth, [MASKED-ENTTTY--] member of, Organization of World Heritage Cities Relation Descriptions: Place of birth of this person.. Membership of this place. Q: Which organization is the place of birth of Yasis Malih member of? A: Organization of World Heritage Cities
Fact Tuple: Intelligent Land Investments, instance of, MASKED-ENTITYS Instance type of this entity., Owner of this type. Q: Who owns the instance of Intelligent Land Investments? Entrepreneur highest point, Kékes Relation Descriptions: Country of citizenship of this person., Highest point of this place. Q: What is the highest point of the country of citizenship of Josef Novak?
owned by. entrepreneur Relation Deseriptions:
Fact Tuple: Josef Novak, country of citizenshipp
A: Kékes
Fact Tuple: Pholidaster squamatus, taxon author, [MASKED-ENTITY-I]] field of work, starfish Relation Descriptions: Taxon author of this species., Field work of this person. Q: What is field of work the taxon author Pholidaster squamatus? Startish
Task:
According these instructions and examples, provide multi-hop QA pair for the following fact tuple: Fact Tuple: (). (). (MASKED-ENTITYANE (). Relation Descriptions: ().
18

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
B. Benchmark Description
This section provides additional insights into the generated benchmark, including its update topics, question types, temporal aspects, and specificity analysis. These analyses aim to highlight the benchmark's 's diversity, temporal resolution, and suitability for evaluating knowledge integration methods.
B.1. Update Topics
To better understand the focus of the updates in each timestep, we analyze the subjects, relations, and objects of the extracted factual triplets. Figure 8 presents word clouds for each timestep, revealing varying topical focuses across the benchmark. For example, in Timestep 7 of updates related to this event. This diversity underlines the richness and real-worke relevance of the generated benchmark.
terms like solar eclipse frequently appear, indicating significant number
202-02201 20240220
20240220-
20240301
20240301 202400
202-0020- 20240101
202 20240501
202-0500 20240601
20240601 20240620
20240620
20240701
Subjects
Battle
AH
AH
Airport
SD Neger
LIVE
Mun.cipa Unid
eclipse
Edinoura
House
Church
House
SMA Negeri
House
nurch vaav
TRIING SEREAGO
Spoken WP
Country
Lipart taxon range
instance
work
subclas
YOrk
inst ance
historic County Anstance
Languager Cooken
Russian Enpire
popul aopllated.ro Place
Export FEO Test
Objeets
De Sovie Ion Onion, Russian
BeR3CCO0 pllnt
Kirker
Tibe: Autononous
Tarbay village
SATe Expor
SOLAY
Reg
Your 7411a6111
AmconNutica
Figure 8. Topic word clouds for cach timestep in the benchmark, illustrating the diversity of subjects, relations, and objects across different update intervals. Each row represents component of the factual triplets (subjects, relations, and objects, respectively), while each column corresponds to a specific timestep. Notable patterns include a focus on specific events, such as "solar eclipse" in Timestep 7 (2244062,,, and varying distributions of topics across timesteps, emphasizing the richness and real-world relevance of the henchmark.
B.2. Mhop Questions
To analyze the characteristics of multi-hop (mhop) questions, we evaluate the performance of LLMs on these questions without updating the models. In the top row of Figure 9, we plot the average accuracy of models on mhop questions as a function of their performance on the individual hops. The x-axis represents the accuracy on questions generated from the first part of the mhop tuple, while the y-axis shows the accuracy on the second part. The color of the markers corresponds to the overall multi-hop accuracy.
The results demonstrate a strong dependence on the accuracy of the second hop, which directly determines the final answer. These findings validate the feasibility of the mhop questions, showing that if the individual facts are known to the model, answering the multi-hop question is possible. To probe for shortcut-driven reasoning (Trivedi et al., 2020), we additionally compare multi-hop performance to performance on the individual hops (see Figure 9). While rare, we observe a small fraction of correctly answered multi-hop questions despite incorrect answers to one of the hops, indicating isolated cases of surface-level or heuristic reasoning.
B.3. Temporal Analysis
We provide an in-depth analysis of the temporal characteristics of the benchmark. Figure 10 shows the number of samples binned by year. The right subplot zooms into updates dated after 2000 for improved visibility. While the benchmark contains updates from past years, significant portion is dated to 2024, reflecting its focus current and relevant knowledge.
Figure 1l presents the accuracy of LLMs binned by the timestamp of the updates. For updates dated between 1800 and 2022, performance remains stable, albeit with higher variance for older samples due to fewer data points. However, updates from 2023 and 2024 show a marked drop in accuracy, emphasizing the benchmark's 's relevance for studying factual knowledge integration in areas where L.MSs struggle most.
19

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Llama 2
Liama 3
Gemma
Mistral
XGen
10.8
2.3-
Hop 2 Accuract
Hop Accuracy
Hop 2 Accuracy
Hop Accuracy
Accuracy
12
64
26
02
Hop Accuracy
Hop Accuracy
Accuraty
Hop Accuracy
Accuracy
10
Hop 2 Speciiiity
Hop 2 Specifiitty
Hop 2 Sp-ccnetty
Hop 2 Speciiiity
Hop 2 Speciiaty
15-
15
62
20
25
26
15 Hop 1 Specificity
13 Hop Specificity
26
Hop 1 Specificity
Hop Specificity
Hop Specificity
Figure 9. Multi-hop (mhop) question accuracy analysis for five models (Llama 2, Llama 3, Gemma, Mistral, XGen). The top shows relationship between the accuracy answering questions from the first (Hop 1) and second (Hop 2) parts of mhop factual tuples and overall mhop accuracy (color-coded). Hop 2 accuracy strongly correlates with mhop question accuracy. highlighting its critical role in multi-hoy hop reasoning. The bottom row explores the relationship between specificity (measured via entity-specificity scores) and accuracy for Hop I and Hop 2. Higher mhop accuracy is generally linked to lower specificity, emphasizing the challenge of highly specific entities.
Number of Questions per Year
Number of Questions per Year (after 2000)
3500
Number of Questions
Number Questions
3000
2500
2000
1500
200
1000
1250
2500
1750
2000
2000
2005
2010
2015
2020
2025
Year
Year
Figure Histograms showing the distribution the number questions per year in benchmark. The left plot provides an overview spanning all recorded years, highlighting a sharp increase in questions for recent years. The right plot zooms into the years after 2000, demonstrating steady growth in questions over time, culminating in significant spike in 2024, reflecting the focus on current updates in the benchmark.
B.4. Specificity Analysis
The benchmark contains a mix of general and highly specific questions. For instance, general questions like "Which continent is Tokyo part of?" contrast with highly specific ones like "What is the spacing of the rails Tire railway station?" Quantifying question specificity is crucial, as highly specific questions may be challenging for LLMs regardless of their knowledge cutoff.
We quantify specificity using two measures:
1. Subject and Object Counts: Frequency of the subject and object entities in the ....oomavvl.7.laame training corpus, derived using the InfiniGram API.
2. Subject and Object Views: 01. 01.07.2024, following Khodja et al. (2024).
Aggregated Wikipedia page views of the subject and object entities from 01.03.2023 to
Figure 12 shows the distribution of specificity scores. The count-based measure skews towards lower frequencies (higher specificity) for subjects, while objects are more evenly distributed. By contrast, the hew-based measure produces smoother distributions for both entities.
We also analyze the relationship between specificity and model performance. Figure 13 shows no clear link between Wikipedia page views and accuracy, while Figure 14 reveals consistent trend: questions with low specificity scores
20

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Avg, Accuracy by Year (after 1800)
Avg. Accuracy by Year (after 2000)
0.60
0.7
0.55
Accuracy
0.6
0.50
0.5
0.45
0.4
0.40
0.3-
0.35
1800
1850
1900
1950 2000
2000
2005
2010
2015
2520
2025
Year
Year
Llama-2 2-7b
Llama-3-4 3-80
Gemma-7b 7b
Mistral-7b
XGen-7b
Figure 11. Average accuracy of five evaluated models on questions binned by their fact timestamp. The left plot includes all questions dated
after 1800, showing stable performance across historical facts. The right plot focuses on questions after 2000, revealing performance decline for facts dated 2023 and 2024. This drop highlights the challenge of addressing more recent updates, emphasizing the relevance of the benchmark for studying factual knowledge integration modern contexts.
Subject Count Distribution
Object Count Distribution
Subject Views Distribution
Object Views Distribution
120000
20000
Numbor of Question
100000
30000
20000
250000
15000
20000
60000
10000
10000
10000
5000
5000
30
10
20
16
20
Subjuct
Object
Subject
Object Views
Figure 12. Distributions of specificity measures for subjects and objects in the benchmark. The left two plots show the frequency-basee specificity, measured as the count of occurrences in reference LLM training corpus (Infini Gram API). The distributions indicate that subjects tend to have lower frequency counts (higher specificity), while objects are more evenly distributed. The right two plots represent specificity based on Wikipedia view counts aggregated over months. These distributions are smoother and centered around a similar mean for both subjects and objects, highlighting different characteristics of the two specificity measures.
(frequent entities) achieve higher accuracy than those with high specificity scores (rare entities). This suggests that specificity, as measured by training corpus frequency, is a critical factor influencing model performance.
Further analysis revealed structural biases in the view-based specificity measure. For instance, questions about languages often receive high specificity scores based on page views despite being general in nature. By contrast, the count-based measure accurately reflects their generality due to the frequent occurrence of language names in training corpora.
C. Knowledge Integration
C.1. Problem Formulation
The task studied in this paper is the large-scale, sequential integration of factual knowledge into a pre-trained language model, denoted as fo. The model fo, with frozen parameters, is trained on dataset Dirain and deployed to process a stream of factual question qe and its corresponding factual answer a,.
updates [uj, u2..... ur]. where each update u, = (4t, a,) consists These updates are grouped into sequential batches, represented as [U1, U2 ........ UB). where U, = { (u,,1, 1, Ub.2...., U6, the set of updates in batch b, and Nú, is the number of updates in that batch.
At each timestep the model fhー1, which incorporates updates from all prior batches UcE - U, updated with the new batch Us, resulting in the updated model Sh, The goal of this process is enable f to integrate the new updates, retain previously integrated knowledge, and maintain its general performance. The objectives of this sequential knowledge integration task can be summarized as follows:
Successful Update Integration: After processing Ub the model must correctly answer all questions in U, ensuring that the newly introduced knowledge is successfully integrated:
{ = at. Vuu E Ub.
21

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Liema-2-7h
Liame-3-hb
Gemmaa-7
Mistral-bb
XGas- 7h
0.9
0.7
4.7
0.-
0.3
Mean Meennoonnos
6--
Oxjeds
6:6-
65-
0.51
65
6.5
4.4
64
45
033
Febzi Meel
Mart
Man kass
hel
hec hal
Mari40
Arl
Alyy
Jusl
Arll
Marise
300-0-4pp1
Apridtel
May Jist
Jest
Feb Pro3--del
May 1:
ala Apd
May! Memt
Fdhl
beedo Maeda-Ape Ap
Man Lual 1: tee ball
6-32 (Mre
16-35
25-52
557
2- CLeas Speaths
Figure 13. Mean accuracy of object-specific questions across specificity bins, measured using Wikipedia page views, for five evaluated models over different timesteps. The specificity bins range from the specific (0-10, blue) the least specific (90-100, red). While no clear trend emerges between specificity and performance, fluctuations in accuracy are observed across timesteps and specificity bins, indicating variability in how models handle questions of different specificity levels.
L:ams-2-7h
Lloma-3-88
Gomman7h
Mistrel-7h
XGan-7h
Mean Acomey
0.54
05-
0.5
0.-
83
03-
024
n2
024
Pakts
Falow
MactB- Aprl
Men deal
hut
Pattos
Felzs Mart
Mart-Sh
Me Aprl
Apviidae
Atxdt I
hazpdaal
Fabss Mar
Marl-30
MaDh-Ael
Apt
Melllall
hai-00
FelzssMell
Mart
Ma Appl
Apt Map!!
Mert hall
hel--5
hadio
frize Mail
Mat
Maze- Mazt-App
Apt Mart
May tail20
huita hil
(Mre Specitiy
16-35
10 daa Speans
Figure 14. Mean accuracy of object-specific questions across specificity bins, measured using Infinigram counts, for five evaluated models over different timesteps. Specificity bins range from the most specific (0-10, blue) the least specific (90-100, red). The results indicate a consistent trend where lower specificity (higher Infinigram counts) is associated with better model performance, with the least specific hin showing the highest accuracy across all models and timesteps. This suggests that questions involving more common entities in the training corpus are easier for the models to answer.
Retention of Prior Updates: The model must retain the factual updates from all previously processed batches, maintaining consistent predictions:
( = at, Vue E Uzb
The updated model must correctly answer rephrased questions from D rephrased which evaluate the generalization ability of the integrated updates:
Generalization of Updates:
(a) = as, Vg, E D Prephraedd
Application of Updates: The model must utilize integrated updates for reasoning tasks, answering multi-hop evaluation questions in Dinhop correctly:
(h( = at, Vqa E
Dimlop-
Locality of Updates: The model must preserve predictions on unrelated facts from a control dataset Diccaliyy ensuring that updates do not cause spillover or degrade performance on unaffected knowledge:
(a( (a))), Vg, E Diocality
To evaluate the performance of the model during this sequential update process, monitor five key metrics after processing each batch U,+ (1) the accuracy on the current batch Un. (2) the retention of accuracy on prior updates from Uea, (3) the accuracy on the rephrased questions Diocaliy. (4) the accuracy on multi-hop questions Dinhop and (5) the locality of
predictions on unrelated facts in Diocality Additionally, forgetting is monitored by continuously evaluating the model
on previously updated batches as new batches are integrated. This sequential process ensures that knowledge updates accumulate over time, creating challenging setting for lifelong factual knowledge integration.
22

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
C.2. Detailed Results
Approach
Model
Update
Rephrase
Personas
Mhop
Locality
lama-2-7b
49.90 (+2.30)
47.74 (+3.65)
41.18 (+2.26)
43.91 (+6.02)
48.62 (+2.411)
lama-3-8b
46.87 (#:3.30)
44.32 (::4.09)
40.12 (=2.85)
44.29 (#8.37)
46.92 (#2.334)
Pre-Update
mistral-7b
48.83 (#:2.26)
47.02 (#:3.32)
40.48 (#2.24)
42.19 (#6.18)
47.97 (+2.75)
gemma-7b
40.61 (+:3.33)
38.66 (+:4.14)
33.08 (=3.10)
35,40 (#7.36)
40.86 (+2.922)
xgen-7b
40.80 (.:1)33)
38.64 (4:3.33)
33.61 (#:2445)
34.87 (#5558)
40.49 (#:2.13)
llama-2-7b
96.17 (#:0.60)
84.92 (+5.76)
72.62 (#2.43)
51.08 (#6.66)
56.29 (#5.09)
Ilama-3-8b
99.18 (+:0.16)
86.19 (+6.37)
74.55 (#2.71)
49.67 (#8.66)
56.26 (+5.445
RAG
mistral-7b
99.25 (#:0.19)
85.89 (16.07)
74.83 (#2.50)
51,88 (=6.71)
56.97 (#5,38)
gemma-7b
95.06 (#:0.73)
81.98 (#6.15)
69.34 (#2.86)
44.84 (=8.02)
49.33 (#5.07)
xgen-7b
95.98 (+:0.46)
82.75 (16..1))
71.45 (#2.94)
45.54 (#6.82)
51.41 (#5,22)
llama-2-7b
63.07 (+8.10)
56.54 (+7.03)
47.19 (+7.05)
43.72 (0.177
46.10 (+3.42)
Illma-3-8b
62.84 (#:4.54)
56.27 (#:5.43)
46.74 (#5.27)
43.24 (=9.06)
45.96 (+2.88)
LoRA-FT
mistral-7b
29.21 (=22.71)
25.65 (+:0.855
21.89 (=:17,57)
20.29 (+18.72)
21.43 (#: 16.47)
gemma-7b
57.38 (+5)86)
51.33 (++610)
42.00 (+5.07)
(+10006)
38.35 (+2.333
xgen-7b
51.53 (#:8.0)
46.21 (#8.17)
38.07 (=8.76)
37.38 (#100.00)
36.48 (+22885)
llama-2-7b
67.36 (+:3.12)
62.01 (#:4.11)
56.64 (#2.93)
50.29 (#7.17)
58.40 (#4.44)
Ilama-3-8b
64.84 (+2.299)
59.79 (+4.56)
55.65 (+2.24)
47.68 (+8.10)
58.24 (+4.266
Merge
mistral-7b
61.84 (#:1.46)
57.17 (#3.82)
52.60 (=1.66)
49.24 (=6.92)
54.68 (#3.20)
gemma-7b
60.13 (#:4.57)
54.87 (#:6299
49.84 (#3.70)
44.64 (=8.09)
53.26 (=5.80)
xgen-7b
62.38 (+:2.50)
57.57 (#:4.49)
51.03 (=2.82)
45.16 (#7.51)
53.72 (+5.01)
Table Detailed evaluation results different approaches (Pre-Update, RAG, LoRA-FT, and LoRA-Merge) across five metrics: Update, Rephrase, Personas, Mhop, and Locality. Results presented for five models (Llama-2-7b, Llama-3-8b, Mistral-7b, Gemma-7b, and 7b) and are averaged across all timesteps, weighted by their respective sizes. Each value represents mean accuracy with standard deviation (+). This table highlights the performance variability of different approaches and models across metrics.
Table 5 provides comprehensive overview of the performance of the evaluated approaches Pre-Update, RAG, LORA-FT, and LoRA-Merge- across the five key metries: Update, Rephrase, Personas, Mhop, and Locality. The results are reported for five different models (Llama-2-7b, Llama-3-8b, Mistral-7b, Gemma-7b, and XGen-7b) and averaged across all timesteps, weighted by their respective sizes. Each entry in the table displays the mean accuracy with standard deviation over the eight timesteps, offering detailed insights into the effectiveness and stability of each approach under the lifelong knowledge integration setting.
C.3. Retrieval Augmented Generation (RAG)
The RAG baseline augments a pre-trained language model fo with a non-parametric memory M, enabling retrieval- augmented generation. The memory M stores factual updates and their embeddings, allowing the model to access relevant information dynamically. The components and process of the RAG baseline are described below:
Memory Construction. For each batch of updates U, {u,,1, 26,2. ub,N.) ). where us,s (qu, 06,)) the memory is constructed as follows:
Initialization: Each update question ib,; and its corresponding answer ab,; are stored in memory.
Embedding Computation: The embedding model g encodes each question into a fixed-dimensional vector: eb, g(qu.).
Memory Entry: Each memory entry is stored as a tuple (gb,s; 06,i, 6b,s), added to the memory M M + MU
23

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
(19b.) 0b,6 0b,))
Test-Time Retrieval. At test time, given input query Best, the RAG baseline retrieves relevant entries from M and augments the input context for the language model:
Embedding Computation for Query: The query embedding is computed as: étest - ((est))
Nearest Neighbor Retrieval: The top-k most similar entries are retrieved based on cosine similarity between eos and stored embeddings 6b,3
Retrieve {(Bb.) {b.))l 1s such that sim (Ciest, Cb.j) is maximized.
Efficient retrieval is implemented using the Annoy solver (Berhaadssson, 2018) for hierarchical navigable small world (HNSW) graphs.
Context Construction: The retrieved questions { and answers { are prepended to Gest to form the augmented context:
Cestt = Concat([qo.1 14 4b.1, Db,ks ab,k: Itess).
Response Generation. The augmented context Crea passed the language model f. which generates the response Arest Otest ((Cies).
This retrieval-augmenned approach provides simple yet scalable mechanism for integrating and accessing factual updates during inference, making it well-suited for the lifelong knowledge integration task.
Parameter Selection and Ablation. For the main experiments, &s 2 was chosen to enable effective multi-hop reasoning while keeping the context length manageable. Figure 15 shows ablation study over k, using subsets of 5,000 updates per timestep with k ranging from 1 to 100. The results indicate that increasing k improves performance across metrics, especially for rephrased questions, as it increases the likelihood of retrieving relevant entries. The performance on the update sets remains stable regardless of k. as they directly match the stored memory entries. Larger & values result in higher computational overhead, with i = 100 increasing the forward pass time by 3x compared to k = 1. Balancing performance gains and computational costs is thus critical.
Update
Rephrase
Pensonss
Mhop
Locality
Pre-ide
Top-1
Mcc
Top-10
Top-5
100
220
300
100 200
304
300
200
400
380
200 of 083
300
4.0
300
150
200
300
400
300
Ntather of Balt (K)
Naxdon of Bdit (K)
Nantber of Bdtto (8)
Natder of Edil (K)
Figure 15. Ablation results for the top-k parameter of the RAG baseline, evaluating update, rephrase, personas, mhop, and locality sets across 500k updates. Higher top-k values improve performance on rephrase, personas, and mhop sets due to increased retrieval coverage, while update accuracy remains consistent across top-k values. The locality set shows marginal gains with higher top-k values, indicating reduced spillover effects. However, increasing top-k also leads to greater computational overhead, highlighting the trade-off between retrieval depth and efficiency.
Computational Overhead. We evaluate the test-time compute of RAG compared pre-update models and LoRA fine-tuning. Corrected by batch size, forward pass times serve as the measure of computational overhead. Figure 16 shows that LoRA finetuning introduces negligible overhead, with minimal variations attributed to outliers. Meanwhile, RAG approximately doubles the forward pass time across all models due to the additional retrieval and larger context. This tradeoff highlights RAG's strength in dynamically integrating factual updates but emphasizes the need to balance retrieval efficiency and test-time compute for large-scale applications.
24

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
0.9
Edit Accuracy
0.6
0.5
0.4
0.025
6,030
0,640
Forward Pass Time (s)
Edit
RAG Post Edit
llama-3 3 -8b
gemma-7b
LoRA Bost-Bdit
llama-2-76
mistral-7b
xgon-7b
Figure 16. Trade-off between forward pass time (x-axis) and edit accuracy (y-axis) for RAG and LoRA across five models. Stars denote pre-edit performance, while post-edit performance for RAG and LoRA is represented by circles and crosses, respectively. RAG achieves higher edit accuracy the cost of increased forward pass time, nearly doubling the average inference latency compared to LoRA. LoRA introduces minimal additional computational overhead while maintaining moderate accuracy improvements over the pre-edit baseline, highlighting its efficiency in coostre-constraineed settings.
C.4. Continual Finetuning
We evaluate the task of continual finetuning using LoRA (Hu et al., 2021) under two distinct configurations: LoRA-FT, where adapters are continuously trained across sequential updates, and LoRA-Merge, which incorporates weight merging and interpolation to balance newly integrated knowledge with prior retained knowledge. Both configurations aim integrate large-scale factual updates efficiently while minimizing catastrophic forgetting and ensuring stable performance over time.
Continual LoRA Fine-Tuning (LoRA-FT)
1. Initialization: For the initial model fo, LoRA adapters A with learnable parameters are initialized and attached to specific layers in the model. The updated model fl is defined as: (z) (z) + A (x), where A (z) represents the output of the LoRA adapter.
2. Sequential Update Process: For each batch of updates Ub the LoRA adapters are trained on the corresponding question-answer pairs U,- ((9,,i " The LoRA parameters A from the previous timestep are reused and further optimized: A + A 1b-1 7/ A01 L(f (gb,s), ab,e), where L is the loss function, and r) is the learning rate, Cosine learning rate scheduling with warmup employed during training, with fixed number of 10 epochs per batch.
No,
3. Model Behavior:
The updated model at timestep b is defined as: (*) = (z) + Azxx,, where A accumulates adaptations from all previous timesteps Ucs.
LoRA Fine-Tuning with Merging (LoRA-Merge)
1. Initialization: Similar to LORA-FT, LoRA adapters A are initialized and attached to the model fd.
2. Weight Merging and Interpolation: After training the LoRA adapters A on the batch Us. the adapted model weights Wadapted are merged into the base model weights Whsse creating an interpolated model: Wmerged = artsss ((1 (1) Wadapted- where the interpolation factor.
3. Sequential Update Process: The merged weights Wmerged as the starting point for training LoRA adapters on the next batch Ub+1.
4. Model Behavior: This process ensures that each timestep balances newly integrated updates with previously retained knowledge, leveraging the interpolation factor a to control the degree of retention.
Ablations and Results
Parameter Selection LoRA-FT. We conduct an ablation on the rank of the LoRA matrices in the LoRA-FT setting,
comparing a high-rank configuration (rank 64) with low-rank configuration (rank = 4). As shown in Figure 17, the
25

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
low-rank setting consistently outperforms the high-rank setting across all metrics. Additionally, the high-rank configuration exhibits greater instability, with instances of catastrophic failure, such as breaking the Mistral model. The results also reveal increased forgetting in the high-rank setting compared to the low-rank setting.
Updatoo
Repheaso
Persumas
Mhop
Locality
Max BAt BACA
Proddi
Peetsae (LoRA-4)
0.4
Mold
Mard--8
92
02
ELAMA-2-18
Ga
12
10
16
Tizeskp T:
Taxsap T3
Tixeskay T2
Tixetty
Timedap
Pae-tinn
305
208
3X
309
(K]
Figure 17. Performance of continual LoRA fine-tuming with high-rank (LoRA-64) and low-rank (LoRA-4) configurations across evaluation sets. The top row shows the mean batch accuracy for update, rephrase, personas, mhop, and locality sets, while the bottom row displays accuracy differences for individual timesteps. Low-rank configurations maintain competitive performance initially but degrade over time, while high-rank configurations show instability, including catastrophic failures certain models.
Parameter Selection LoRA-Merge. We further investigate the impact of the interpolation factor a in the LoRA-Merge setting, as shown in Figure 18. We evaluate two configurations: one where the base model and adapted model are equally weighted (a 0.5) and another where the base model is given higher weight (a 0.75). The results indicate that higher weight given to the adapted model (a 0.5) leads to stronger initial adaptation but results in performance decay over time. Conversely, prioritizing the base model (a= 0.75) provides more stable performance across timesteps.
Update
Repheise
Persones
Milop
Loaelity
Pre-Bdit
Pust plyhe11
00
9.6
8.6
(ighee.5)
Model
Misall
62
92
8.2
8.2
6.2
3-31
XGax78
390
30
300
200
204
168
36
300
300
Tinesop
Tineag T5
Timwap
Timwap
Ao. Diltrroo
-18
-11
-10
Tinestap
Tinestp
Pro-Edi septami
Pex-Bii: (apheen
22
132
710
390
280
300
284
500
181
208
310
3.0
204
180
214
30
300
160
3/6
300
680
581
Namberr
Namber Edita EX]
Namber Edit [K]
Namber of
Namtorr
Figure 18. Ablation study on the interpolation factor a in the LoRA Mergee setting, comparing top row shows the mean batch accuracy for the update, rephrase, personas, mhop, and locality sets, while bottom row depicts accuracy differences (in percentage points) across timesteps. Higher weight on the base model (a 0.75) provides more stable performance and mitigates forgetting, while equal weighting (a = 0.5) achieves higher initial performance but leads to greater degradation, particularly on the mhop and locality sets. Bold lines represent average model performance, with lighter lines for individual models.
0.75 (red) and a = 0.5 (orange). The
C.5. Knowledge Editing
The knowledge editing task involves updating a pre-trained language model fü, with frozen parameters trained on Dunin- to modify its behavior for specific inputs qi to produce updated outputs a, These updates must be incorporated without retraining the entire model and while preserving the model's 's performance on unrelated tasks and previously integrated updates. The key requirements of the task are:
1. Successful Knowledge Integration: The edited model fd should correctly predict a, for all edited questions gt-
26

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
2. Retention of Unrelated Knowledge: The edits should not degrade the model's performance on unrelated queries Qi Procality ensuring (g.) (g:)-
3. Consistency Across Prior Edits: The model should maintain the correctness of previous edits q-t, ensuring (qu) dgr for all t/ < L.
The following describes three prominent knowledge editing methods- ROME, MEMIT, and WISE- this notation.
Local Modification Approaches. Local modification approaches focus on directly altering the parameters of the model specific layers to integrate new knowledge while minimizing unintended side effects. This category includes ROME and MEMIT.
Rank-One Model Editing (ROME). ROME (Meng et al., 2023) is a method for performing rank-one updates to a specific layer in the model. focuses on precisely editing the causal pathways responsible for a particular factual association.
Objective: Modify the model fo such that, for an update u (gt, at), the edited model satisfies: (a) ar, while minimizing interference with unrelated predictions.
Layer Selection: Identify the most causally relevant layer 1" for the given update by analyzing the model's causal mediation.
Update Rule: Modify the weights Wi. in the key-value projection as: Wi. + Wi. + AW, where AW ww is rank-one update computed from the desired output a and intermediate activations at layer l*
Multistep Editing for Memory-Intensine Tasks (MEMIT), MEMIT (Meng et al., 2022) extends ROME to handle multiple updates simultaneously and distributes them across multiple layers.
Objective: Modify the model l to incorporate batch of updates U, {gnhi a such that: (gr,,) ab,t, Vub, Us. while maintaining performance unrelated queries.
Layer Selection: Select set of layers {41, (2.,4....., ... [+} for distributing the updates to reduce interference.
Update Rule: For each selected layer 1, apply weight update: W1 Wi + AW, where AW; is computed via low-rank decomposition to ensure computational efficiency and scalability.
Local modification approaches are effective for small-scale or isolated updates but face challenges with scalability and catastrophic forgetting sequential update scenarios.
Lifelong Knowledge Editing Approaches. Lifelong knowledge editing approaches are designed to handle sequential, large-scale updates while mitigating catastrophic forgetting and maintaining stability over time. This category includes WISE.
Weight Injection for Structured Editing (WISE). WISE (Wang et al., 2024a) leverages semi-parametti memory to store layer-specific weight adjustments for each update, enabling reversible and modular edits.
Objective: For sequence of updates [w1, u2, ur]. modify the model f such that f*( at: ensuring (4: = (q)), Vg: Diocality, and retaining accuracy on prior updates u-r-
Vt S T, while
Memory Construction: For each update u, (g:: az), compute weight adjustments AW, for specific layers 1. Store these adjustments in memory: Mi <- M, U AW, VI L, where L is the set of edited layers.
Inference with Memory: During inference, apply the stored adjustments AWi to the forward pass: f* (x) + Eier (w(
Scalability: By maintaining modular memory, WISE avoids catastrophic forgetting and enables scalability to large sequences of updates.
Lifelong editing approaches like WISE are better suited for large-scale, sequential updates, though they may still face challenges with long-term consistency and retrieval efficiency in memory-base systems.
27

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
Knowledge Editing for Lifelong Knowledge Integration Figure 19 provides additional results on the application of knowledge editing techniques the lifelong knowledge integration task. The figure presents the accuracy the update set for the first 500 updates and the entire first timestep (26K updates) for the Llama-2 model. The results highlight the rapid performance deterioration of local modification approaches, with accuracy dropping significantly within the first 250 updates. While WISE performs comparably to RAG for fewer than 500 updates, its performance deteriorates over the first 10K updates, eventually converging to pre-update performance levels. This behavior underscores the challenges faced by knowledge editing approaches handling larger-scale, sequential updates.
8.8
Pre-Edtt
0.6
MEMIT
R-ROME
ROME
LoRA (Agg)
0.2
(AC())))
0.0
1003 240
740
2001
5000
1900
15303
2000
25100
Number of Ediis
Namber Edis
Figure 19. Update accuracy of knowledge editing approaches (MEMIT, R-ROME, ROME, and WISE) compared LoRA-FT and RAG baselines across the first 500 updates (left) and the full first timestep (26k updates, right). Local modification methods (MEMIT, R-ROME, ROME) rapidly degrade within the first 250 updates, converging to year-zero performance. WISE initially performs on par with RAG for fewer than 500 updates but declines over the first 10k updates, converging to pre-update accuracy, highlighting its limitations in maintaining update accuracy for larger-scale knowledge integration.
In the main experiments, MEMIT exhibits model collapse after approximately 250 sequential edits. This contrasts with the original MEMIT paper (Meng et al., 2022), which demonstrates successful integration up to 10K edits. To investigate this discrepancy, we conduct an ablation study where we vary the batch size (i.e., the number of edits processed simultancously) for MEMIT on Llama-2 using the first 10K updates of the Wik WikiBig BigEdit t benchmark. The results, presented in Figure 20, reveal that MEMIT avoids model collapse when edits processed in larger batches, with the performance remaining stable when all 10K edits are applied at once. However, the overall update accuracy in this setting remains below the pre-edit performance of the model, highlighting a trade-off between stability and efficacy.
1.0
MEMIT (Batch Size: 1)
0.8
MEMIT (Batch Size: 1000)
Edit Accuracy
MEMIT (Batch Size: 5000)
0.6-
MEMIT (Batch Size: 10000)
LoRA FT (Avg)
0.4
RAG (Avg.)
WISE
0.2
Pre-Edit (Avg.)
0.0
2000
6000
5000
1000
10000
Number Edits
Figure 20. Impact of batch size on MEMIT's performance for the first 10k updates of the Wiki gEddit benchmark using Llama- 2. Larger batch sizes (e.g., 10k edits) prevent model collapse but result in lower update accuracy compared to the model's pre-edit performance. Smaller batch sizes (e.g.. or edits) perform well initially but exhibit rapid degradation in subsequent updates, highlighting MEMIT's limitations for sequential, large-scale lifelong knowledge editing.
Interestingly, when applying smaller batch sizes (e.g., IK, 5K edits per batch), MEMIT performs well for the initial batch but collapses in subsequent updates. This behavior indicates severe limitation: while batch processing can temporarily mitigate collapse, it is impractical for real-world scenarios where edits arrive sequentially over time or at large scale. The need to process edits as they become available underscores MEMIT's unsuitability for sequential, large-scale lifelong knowledge editing. These findings emphasize the importance of developing methods that remain stable and effective under practical constraints, where timely incorporation of updates is crucial.
28

---

Understanding the Limits of Lifelong Knowledge Editing in LLMs
In Figure 21, we present additional experiments with WISE conducted three language models: Llama-2, Llama-3, and Mistral. These experiments focus the first timestep of the WikiBigEdit benchmark. The results consistently show that WISE converges to pre-update performance across all evaluated metrics within the first timestep. Although WISE avoids catastrophic forgetting, its inability to maintain the accuracy of newly integrated updates over time highlights its limitations for lifelong knowledge integration tasks.
Update
Rephease
Persones
Milop
Locality
Pie
Man Araah.nnnnnc
Pan-Bdz
Liana3--00
Mosal-1B
3099 3000 19KE 29.0 2580 Nomber (k)
sie I000 1388 2800 Nambee of (k)
2500
390 1981 1589) 2008 2500 Number Eitr [K]
5001 19:09 15000 20000 2008
5000 10800 Numbar biti (K]
1900 3000
Nommer
Figure 21. Results of WISE on three language models (Llama-2-7B, Llama-3-88 8B, and Mistral-7B) for the first timestep of the WikiBigEdit henchmark. Each subplot shows mean batch accuracy for the update, rephrase, personas, mhop, and locality sets over number updates. Post-update performance (blue) converges to pre-update levels (gray) across all metrics, highlighting WISE's limitations in sustaining accuracy after large-scale updates.
29


Reviews:
[{"review_id": "156kZTizve", "reviewer": "Reviewer_Qubt", "summary": "This paper proposes a large-scale knowledge editing benchmark *WikiBigEdit* based on the Wikidata, which contains over 500k question-answer pairs. At the same time, this work also constructs a pipeline that can automatically update data to adapt to the changes in real data, while mitigating the overfitting problem of pre-training processes for large-scale language models on information before the training time point. After construction, this article tests the effectiveness of several mainstream methods on *WikiBigEdit*, reflecting various problems of previous methods in terms of locality, multi-hop, and personalized questioning (generalizability).\n\n## update after rebuttal\nI'll keep my assessment unchanged. The authors have answered my questions clearly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and detailed evaluation. We appreciate the recognition of our key contributions: the construction of *WikiBigEdit* as the first large-scale, automatically extensible benchmark for real-world knowledge editing; the automated pipeline enabling continual updates; and the comprehensive evaluation across multiple LLMs and editing techniques. We are also grateful for the acknowledgment of our efforts to mitigate data leakage and pretraining overlap, as well as the clarity of our supplementary materials and code. We address the reviewer’s insightful questions below.\n\n**Length of Multi-hop Reasoning Chains**\n\nWe appreciate the reviewer’s observation regarding the scope of multi-hop reasoning. While multi-hop can involve longer chains or subgraphs in general QA settings, in the context of knowledge editing, it is commonly defined as reasoning over two connected facts, forming a two-hop chain [1–3]. This is consistent with prior work, such as HotpotQA [4] and EX-FEVER [5], where two-hop questions are standard for evaluating a model’s ability to integrate and reason over multiple facts. Our benchmark adopts this convention to align with established protocols. That said, we agree that exploring longer reasoning chains is a valuable direction for future work.\n\n**Triplet Quality and Filtering**\n\nWe thank the reviewer for their question on triple quality and filtering. To ensure a high-quality benchmark, our automated pipeline applies multiple filtering and QA steps (see Section 3.2). Below are average statistics from the first four timesteps:\n* Initial Input: 100% of new or changed factual triplets from consecutive Wikidata snapshots.\n* Initial Filtering: Remove cyclic, non-Roman, or overly long entities (~89% remain).\n* Filter Unwanted Relations: Keep only WikibaseItem relations (entity-to-entity), excluding media, URLs, etc., and a small list of low-value relations (e.g., “use”, “is a list of”), retaining ~39%.\n* Non-Deterministic Triplets: Remove (subject, relation) pairs with multiple objects (~24% remain).\nFinal Filtering: Further filtered to remove spurious entries (final ~24% retained).\n\nWe also conducted manual inspections, sampling triplets from various batches to confirm that relations and entity pairings are meaningful.\n\nThese steps help ensure that only high-quality, interpretable factual triples are included in WikiBigEdit. We will extend Section 3.2 to report these statistics in the final version.\n\n**Impact of LLM Choice on QA Generation**\n\nWe thank the reviewer for raising the important question of how the choice of LLM affects QA pair generation. In preliminary experiments, we evaluated the sensitivity of our generation pipeline to different GPT models.\n\nFor the Update and Locality sets, model choice had a negligible impact. These questions are directly derived from subject–relation–object triplets, and even GPT-3.5-turbo consistently produced accurate and well-formed outputs. For example, given the triplet (SS Heliopolis, manufacturer, Fairfield Shipbuilding and Engineering Company), GPT-3.5-turbo generated “Which company was the main manufacturer of SS Heliopolis?”, GPT-4o-mini produced “Who was the manufacturer of the SS Heliopolis?”, and GPT-4o generated “Which company was the manufacturer of the SS Heliopolis?” — all of which are stylistically comparable and semantically correct.\nThe Rephrase set showed similarly consistent quality across models, as paraphrasing proved to be a relatively simple task for most LLMs.\n\nIn contrast, the Multi-hop and Persona sets were more sensitive to model choice, as these tasks require compositional reasoning or stylistic transformation. For instance, in the multi-hop case combining (2004 VS75, discoverer, Marc Buie) and (Marc Buie, gender, male), GPT-3.5-turbo generated the incomplete question “Who is the discoverer or inventor of (525257) 2004 VS75?”, failing to incorporate both facts. Meanwhile, GPT-4o-mini and GPT-4o correctly generated queries such as “What is the gender of the discoverer of 2004 VS75?”\nBased on these findings, we selected GPT-4o-mini for the Multi-hop and Persona sets, as it offers a strong balance between generation quality and efficiency. We observed no significant difference between GPT-4o and GPT-4o-mini in these settings.\nWe hope this clarifies the robustness of our generation pipeline and our rationale for model selection. We will include an ablation on generation models in the Supplementary Material.\n\n[1] Cohen et al.,  Evaluating the Ripple Effects of Knowledge Editing in Language Models, 2023.\n\n[2] Zhong et al., MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions, 2023.\n\n[3] Zhong et al., MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations, 2025.\n\n[4] Yang et al., HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, 2018.\n\n[5] Ma et al., EX-FEVER: A Dataset for Multihop Explainable Fact Verification, 2024.", "claims_and_evidence": "Yes. \nThe text provides a detailed description of the construction process of the knowledge editing benchmark *WikiBigEdit*, and makes a detailed evaluation of the representative methods in the field on different open-source models, followed by analysis. The experimental results are very comprehensive.", "methods_and_evaluation_criteria": "The *WikiBigEdit* proposed in this paper is meaningful for knowledge editing tasks. On one hand, this is the first dataset of such a large scale constructed based on real data. On the other hand, the additional data update pipeline designed by the authors makes the benchmark keep pace with the times and avoids the overfitting or data leakage problems of newly emerged LLMs.", "theoretical_claims": "This paper is a work about a new benchmark and does not involve theoretical proof. All the formulas in the text merely describe the update process of model parameters and serve only as a narrative, not as proof.", "experimental_designs_or_analyses": "Yes, I have checked all the experiments, mainly including different models (7B level) under various knowledge editing methods on the *WikiBigEdit* benchmark proposed in this paper. Regarding the experiments, I have several questions I would like to ask the authors:\n1. As described in the paper, in the Multi-Hop Reasoning test, the authors only considered connecting two triplets end-to-end (which is actually a two-step question). But is this too short for normal multi-hop question answering? General multi-hop question answering is usually based on a subgraph of a knowledge base or a longer reasoning chain.\n2. Can some other results based on closed-source models  (considering GPT3.5, 4o-mini used in the experiment, the cost is actually not high) or larger open-source models (like Qwen-series) be supplemented in the experiment?", "relation_to_broader_scientific_literature": "This paper proposes a large-scale dataset *WikiBigEdit* for the knowledge editing task. This dataset addresses the insufficient amount of data in previous knowledge editing tasks, as well as the issue of overfitting in the pre-training process after the emergence of large-scale language models. More importantly, this benchmark proposes a pipeline for continuously updating data, constantly updating the dataset based on reality, which very effectively solves the problem of models overfitting to all knowledge before a specific time point.", "essential_references_not_discussed": "There is no such situation in my vision. This is the first knowledge editing dataset to reach this scale and can be continuously updated. At the same time, as a benchmark, this work has compared several very mainstream knowledge editing methods.", "other_strengths_and_weaknesses": "As described in \"Relation to Broader Scientific Literature\" and \"Experimental Designs or Analyses\".", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The previous common sense knowledge graph (such as ConceptNet) is a relatively dirty knowledge base, where the same head entity and relationship can correspond to multiple tail entities. In the narrative of this paper, such relationships are directly removed. I would like to know what proportion is removed? Can there be some analysis experiments to analyze the quality of the extracted triples? (Because in my previous understanding, the dirty relationships or meaningless triples in such common sense graphs account for the majority.) \n2. The construction of problems in the text is based on GPT3.5 or GPT-4o-mini. Have you tried other models for problem construction? Would the different distributions of different models lead to an impact on data quality? \nAnyway, this is a good piece of work, and answering these questions will make my understanding more complete.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ip3Q28QU5J", "reviewer": "Reviewer_j8NE", "summary": "Enabling LLMs to retain up-to-date knowledge is of significant practical importance. To avoid costly full-parameter retraining, recent research has proposed various lifelong knowledge editing methods to inject new knowledge into models at minimal expense. However, these knowledge editing approaches have two notable limitations: 1. The scale of edited knowledge is relatively small; 2. The complexity of the knowledge is limited. These constraints make existing knowledge editing methods difficult to apply directly in real-world scenarios.\nTo better simulate real-world conditions and investigate the limitations of knowledge editing techniques, the authors designed an automated data extraction pipeline to acquire the latest world knowledge from Wikipedia. In this pipeline, they employed LLMs to rephrase the data, generating five distinct datasets collectively named WikiBigEdit. These five datasets (four of which serve as test sets) evaluate model knowledge editing effectiveness from four perspectives: 1. Ability to answer rephrased questions; 2. Ability to answer questions written in specific personal styles; 3. Ability to answer multi-hop questions; 4. Retention of unmodified knowledge.\nThe authors used WikiBigEdit to test the performance of mainstream knowledge editing methods against general approaches in knowledge updating tasks. Experimental results demonstrate that models utilizing RAG methods achieved the highest accuracy on the test sets, while continued fine-tuning and model merging yielded second-best results, whereas knowledge editing methods performed unsatisfactorily. The findings indicate that existing knowledge editing techniques have significant limitations when applied to large-scale knowledge editing.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive review. We are pleased that the reviewer found the motivation and contributions of our work to be of practical importance — particularly our effort to investigate the limitations of existing knowledge editing techniques under realistic conditions. We appreciate the recognition of our automated pipeline for extracting real-world knowledge updates, as well as the comprehensive evaluation framework of *WikiBigEdit*, which tests factual retention, generalization, and reasoning across multiple dimensions. We are also grateful for the acknowledgment of our systematic comparison of knowledge editing methods with general update approaches such as RAG and continual finetuning. We address the reviewer’s suggestion for deeper analysis and provide further clarifications below.\n\n**In-Depth Analysis of Knowledge Editing Failure**\n\nWhile our study builds on the foundations of previous work, our experiments go beyond a simple dataset substitution. Specifically, we design a setting that reflects real-world, large-scale lifelong knowledge editing, a departure from the typically synthetic and small-scale settings used in prior studies (see also related works). Our benchmark introduces several important new dimensions: (1) the sequential nature of edits over time, (2) the use of real-world factual updates derived from Wikidata, and (3) a comprehensive evaluation protocol that includes not just edit accuracy but also generalization, locality, and multi-hop reasoning.\n\nMoreover, our analysis compares knowledge editing methods not only to one another but also to broader model update strategies, such as retrieval-augmented generation (RAG) and continual finetuning with adapter merging. This allows us to contextualize the limitations of editing methods in a practical deployment landscape - a comparison absent in prior work.\n\nWe agree that a deeper investigation into the specific mechanisms of failure in knowledge editing is an important direction. While our primary focus is on empirically establishing limitations at scale, we highlight recent work that examines these mechanisms in detail (see Section 2). For example, Hsueh et al. [1], Gupta et al. [2], and Yang et al. [3] demonstrate that even small-scale editing can induce instability or forgetting in LLMs. Gupta et al. [4] further attribute performance collapse during sequential edits to the lack of regularization, which leads to overfitting on individual updates. These insights complement our findings and underscore the challenges in designing robust editing techniques that scale.\n\nWe will clarify this distinction and discussion in Section 2 of the revision.\n\n[1] Hsueh et al., Editing the Mind of Giants, 2024.\n\n[2] Gupta et al., Model Editing at Scale Leads to Gradual and Catastrophic Forgetting, 2024.\n\n[3] Yang et al., The Butterfly Effect of Model Editing, 2024.\n\n[4] Gupta et al., Lifelong Sequential Knowledge Editing without Model Degradation, 2025.", "claims_and_evidence": "The authors designed comprehensive experiments to substantiate their arguments.", "methods_and_evaluation_criteria": "Regarding data, the authors employed a convincing approach to construct a large-scale dataset for testing whether models can retain the latest knowledge from Wikipedia. The test sets they developed comprehensively evaluate knowledge editing effectiveness from four distinct perspectives.\nIn terms of model selection, the authors tested several mainstream knowledge editing methods (including ROME, R-ROME, MEMIT, and WISE) as well as general-purpose RAG methods, continued fine-tuning approaches, and model merging techniques.", "theoretical_claims": "No theoretical contribution was presented by the authors.", "experimental_designs_or_analyses": "In the experimental section, the authors evaluated the effectiveness of various knowledge-updating methods across different LLMs (such as LLaMA and others). For each method, the authors provided detailed analyses of their performance on different datasets and presented several significant conclusions.", "relation_to_broader_scientific_literature": "This paper summarizes and emphasizes findings from existing literature, namely: as the number of knowledge entries to be edited increases, the ability of models trained with current knowledge editing techniques to memorize knowledge gradually deteriorates. Furthermore, through comparative experiments, this study reveals that knowledge editing techniques cannot outperform general methods (such as RAG, fine-tuning, etc.) and do not offer significant cost advantages.", "essential_references_not_discussed": "No missing key references were identified.", "other_strengths_and_weaknesses": "Strengths:\n- The authors constructed a large-scale and comprehensive knowledge editing dataset that effectively measures model performance on knowledge updating tasks.", "other_comments_or_suggestions": "The main focus of this paper is to analyze the limitations of existing knowledge editing methods. However, in the experimental section, the authors merely replicated similar experiments from previous literature (only changing the dataset) and obtained similar results. It would be desirable for the authors to conduct a more in-depth analysis of the specific reasons why existing knowledge editing techniques underperform compared to general methods.", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5G4SgcIWtg", "reviewer": "Reviewer_UH8c", "summary": "This paper proposes a benchmark called WikiBigEdit for evaluating knowledge editing. The benchmark is constructed from Wikidata edits and includes 500k question-answer pairs. They evaluate a number of existing methods for knowledge editing and find that current techniques have limitations where continual fine-tuning and retrieval augmented methods perform better. They suggest one of their contributions is to have a fully automated extraction pipeline which continuously extracts suitable factual edits from Wikidata. The resulting WikiBigEdit spans eight time-intervals over five months (February - July 2024).\n\nSection 3 discusses the design of WikiBigEdit. Their construction is divided into 7 steps:\n1. Periodic Snapshot Acquisition ($S_{changed}, S_{unchanged}$): downloading subject-relation-object relations from Wikidata for most recent snapshots. Triplets are divided into changed and unchanged sets.\n2. Initial Filtering: this step excludes triplets based on simple rules such as length of the subject or object.\n3. Generation of Locality Probes ($S_{locality}$): finds pairs of triplets in changed and unchanged sets where the subject-relation is similar but the object is distinct.\n4. Inclusion of multi-hop reasoning tuples ($S_{mhop}$): extracts pairs of factual triplets that are linked to each other by a shared subject/object. These pairs can be used to create multi-hop questions.\n5. Generation of QA edit pairs: for the sets changed and locality GPT-3.5 and for the set of mhop facts GPT-4o-mini are utilized to create question-answer pairs. For mhop, the questions omit the middle shared entity.\n6. Generation of personalized and rephrased question-answer edit pairs: to further evaluate the reasoning generalization, GPT-4o-mini is used to create question-answer variants by either rephrasing or mimicking a persona that rewrites the QA in various styles.\n7. Final filtering stage: filter QAs to make sure the question contains the subject/relation but not the answer/object and answers include the object.\n\nThe paper provides various analyses of the benchmark. For example, Figure 2.a shows that the generation QAs focus on recent events although questions about older events also exist as old as the 1900s.  Figure 2.b shows that the performance of five LLMs drops after their cut-off date on the proposed benchmark.\n\nSection 4 evaluates the performance of various knowledge editing methods. Figure 3 summarizes the performance of these methods on subsets of the evaluation. It shows while RAG outperforms all methods, continual fine-tuning methods based on LoRA perform better than other knowledge editing methods. The result about RAGs is expected as retrieving the correct answer from Wikipedia/Wikidata should be easy for RAG methods while multi-hop reasoning does not see significant improvement. Figure 4 also shows that performance of knowledge editing methods decays as the number of edits increase which is a limitation of such methods over long horizons.\n\n## Update after rebuttal\nI thank authors for their response. I encourage authors to incorporate their response into a revision, particularly the result of multi-hop evaluations and connection DiRe condition as well relation to other methods and benchmarks. I maintain a positive rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful feedback. We’re pleased they found the paper well-written, the benchmark well-designed, and the findings insightful. We appreciate the recognition of *WikiBigEdit* as a valuable contribution for evaluating real-world limitations of knowledge editing, as well as the clarity of our benchmark construction. We’re also grateful for the positive assessment of our method coverage and the relevance of our experimental insights, particularly regarding the challenges current editing techniques face over long update horizons. We address the reviewer’s suggestions and questions below.\n\n**Advanced RAG Approaches**\n\nWe thank the reviewer for the helpful suggestion of including more advanced RAG approaches and for highlighting HippoRAG [1]. While HippoRAG introduces a compelling long-term memory mechanism via structured memory graphs, it assumes access to full-text documents—unlike our setting, which focuses on factual QA pairs derived from structured triplets.\n\nWe agree that advanced retrieval-augmented approaches, including reasoning-augmented RAG or RAG agents, could enhance multi-hop reasoning. However, our goal is to evaluate the limits of lifelong knowledge editing in a controlled setting. We, therefore, use a simple, transparent RAG baseline to compare against editing and continual finetuning. Notably, even this baseline outperforms editing methods across most axes, though at higher inference cost.\n\nRAG’s difficulty with multi-hop reasoning is orthogonal to the challenge of integrating new knowledge over time — the focus of our work. More advanced RAG variants may improve multi-hop QA but likely require even more test-time compute.\nWe will add this discussion, along with a reference to HippoRAG, to Section 4.3 of the paper.\n\n**Shortcuts in Mhop Evaluation**\n\nWe appreciate the reviewer’s suggestion to consider the DiRe condition [2]. While we do not explicitly evaluate it in its original form, we conduct a related analysis in the Supplementary Material (Figure 9, top row). Specifically, we compare model performance on individual first-hop and second-hop questions to its performance on the combined multi-hop question. This serves to test whether the model relies on shortcut reasoning, i.e., answering the multi-hop question without resolving its components, which is central to the DiRe condition.\n\nOur results show that multi-hop accuracy depends heavily on correctly answering both hops. When a model fails on either, it rarely succeeds on the full question. However, a small fraction of questions (avg. 1.75% across models) are answered correctly despite failure on the first hop. These cases often involve surface-level cues, e.g.:\n* “What is the language of work or name associated with the spouse of Johann Caspar Richter?” → German\n* “In which country is the organization that Yamato Auto Works is a member of located?” → Japan\n\nThese align with shortcut-driven behavior as discussed in the DiRe framework. We will clarify this connection in Section 3.1 of the revision.\n\n**Reference of Concurrent Work TiC-Wiki**\n\nWe thank the reviewer for pointing us to the concurrent work TiC-Wiki [3], which shares similarities in spirit. We will include a discussion of it in the related works section. While TiC-Wiki is a valuable benchmark for continual pretraining, it covers Wikipedia/Wikidata revisions from 2008–2022 — a period largely included in the pretraining data of modern LLMs. This makes it less suitable for evaluating updates beyond the knowledge cutoff, which is the central focus of our work. In contrast, WikiBigEdit is constructed from post-cutoff updates, allowing direct evaluation of newly emerging knowledge. We view the two efforts as complementary and appreciate the chance to clarify this distinction.\n\n[1] Gutiérrez, B. J., et al. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models, 2024.\n\n[2] Trivedi, Harsh, et al. Is multihop QA in DiRe condition? Measuring and Reducing Disconnected Reasoning. 2020.\n\n[3] Li, Jeffrey, et al. TiC-LM: A Multi-Year Benchmark for Continual Pretraining of Language Models, 2024.", "claims_and_evidence": "The paper introduces a new benchmark to evaluate knowledge editing capabilities. The paper shows the importance of the problem, discusses the construction with details, and evaluates existing methods and provides interesting insights.", "methods_and_evaluation_criteria": "The selection of methods covers a wide spectrum of methods. The authors may consider including evaluation of more recent RAG methods that potentially utilize an LLM for reasoning on facts that may improve on multi-hop reasoning evaluations proposed in the paper.\n- Gutiérrez, Bernal Jiménez, et al. \"Hipporag: Neurobiologically inspired long-term memory for large language models.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.", "theoretical_claims": "No theoretical claims made.", "experimental_designs_or_analyses": "The design of the evaluations and comparison of models is well designed. The authors may consider testing their evaluation against the DiRe condition. See paper below:\n- Trivedi, Harsh, et al. \"Is multihop QA in DiRe condition? Measuring and reducing disconnected reasoning.\" In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)", "relation_to_broader_scientific_literature": "The question of how to keep the knowledge of LLMs up to date is an important research direction that requires stronger benchmarks. This work is a helpful evaluation for better understanding the limitations of LLMs and designing improved methods to keep their knowledge up to date.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- The paper is well-written, the benchmark is well-designed and provides insightful findings.\n\nWeaknesses\n- Please see above for potential comparisons and related works. Specifically, performing a variant of DiRe test may be helpful in understanding whether the multi-hop reasoning questions could be answered with a shortcut. It may be hard to apply the test directly as the QAs are written by GPT models. One potential way is to manually remove the mention of the subject from the question and evaluate if the question would be still answerable without it. Some questions about general knowledge may have this property.", "other_comments_or_suggestions": "Authors may consider discussing the relation to recent concurrent work TiC-LM and particularly TiC-Wiki where a related automatically generated continual benchmark based on Wikipedia and Wikidata over a span of 10 years has been proposed.\n- Li, Jeffrey, et al. \"Tic-lm: A multi-year benchmark for continual pretraining of language models.\"", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces WikiBigEdit, a large-scale, continuously updating benchmark designed to evaluate the capabilities of knowledge editing in large language models (LLMs). Built from Wikidata, it comprises over 500,000 question-answer pairs generated through an automated extraction and rewriting pipeline. The benchmark spans multiple knowledge editing dimensions, including multi-hop reasoning, stylistic variation, and knowledge retention, enabling comprehensive evaluation of modern LLMs. Reviewers (Qubt, j8NE, UH8c) universally acknowledge the novelty and practical relevance of this benchmark, especially in light of real-world challenges such as outdated knowledge and the limitations of retraining. They also commend the meticulous data construction pipeline, strong empirical evaluation across a range of methods (e.g., ROME, MEMIT, RAG, fine-tuning), and clear experimental insights that reveal current weaknesses in existing knowledge editing techniques (e.g., poor scalability and generalization). The benchmark’s alignment with real-world data and its reproducibility—via shared code and data—further enhance its value to the community.

While the overall contribution is strong, reviewers suggest several areas for refinement. First, the benchmark’s multi-hop evaluation may be too simplistic, as it only considers two-hop reasoning; more complex chains could better represent real-world multi-step inference (Qubt). Additionally, there is limited exploration of whether different LLMs used in the QA construction phase (e.g., GPT-3.5 vs. GPT-4o-mini) introduce distributional biases that affect benchmark quality (Qubt). The current evaluation focuses largely on well-known open-source models, and broader comparison with larger or closed-source systems (e.g., Qwen or GPT-4) could offer deeper insights (Qubt, UH8c). Reviewers also recommend testing the benchmark against recent baselines (e.g., HippoRAG, DiRe tests) and concurrent datasets like TiC-LM for multi-hop and long-term reasoning comparisons (UH8c). Lastly, while j8NE appreciates the benchmark’s value, they point out that much of the experimental analysis replicates existing setups on a new dataset and suggest that more in-depth failure analysis would improve the interpretability of performance gaps. Despite these concerns, all reviewers agree that the benchmark is timely, impactful, and fills a critical gap in the evaluation of dynamic knowledge in LLMs.
Decision: Accept (poster)