ICML 2025

WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Lukas Thede 1 2 3 Karsten Roth 1 2 3 Matthias Bethge 1 Zeynep Akata 2 3 4 Thomas Hartvigsen 5
Abstract
Keeping large language models factually up-to-
date is crucial for deployment, yet costly retrain-
ing remains a challenge. Knowledge editing of-
fers a promising alternative, but methods are only
tested on small-scale or synthetic edit benchmarks.
In this work, we aim to bridge research into life-
long knowledge editing to real-world edits at
a practically relevant scale. We first introduce
WikiBigEdit; a large-scale benchmark of real-
world Wikidata edits, built to automatically ex-
tend lifelong for future-proof benchmarking. In
its first instance, it includes over 500K question-
answer pairs for knowledge editing alongside a
comprehensive evaluation pipeline. Finally, we
use WikiBigEdit to study existing knowledge
editing techniques’ ability to incorporate large
volumes of real-world facts and contrast their ca-
pabilities to generic modification techniques such
as retrieval augmentation and continual finetun-
ing to acquire a complete picture of the practical
extent of current lifelong knowledge editing.1
1. Introduction
Keeping large language models factually up-to-date is a
critical challenge in their practical deployment (Hartvigsen
et al., 2023; Meng et al., 2023; 2022; Wang et al., 2024a;
Zhang et al., 2024) due to given knowledge cutoffs and
limited tail-end concept coverage (Cheng et al., 2024; Zheng
et al., 2024; Zhang et al., 2024; Udandarao et al., 2024)
incurring inaccuracies. Finding ways to inject or update
new and revised facts is crucial; Particularly for factuality-
critical applications like medicine (Bao et al., 2023; Jeong
et al., 2024), law (Colombo et al., 2024; Lai et al., 2024) or
education (Team et al., 2024c; Wang et al., 2024b).
1T¨ubingen AI Center, University of T¨ubingen 2Helmholtz Mu-
nich 3Munich Center for Machine Learning (MCML) 4Technical
University, M...