Lifelong Sequential Knowledge Editing without Model Degradation
Authors: name: Akshat Gupta, name: Phudish Prateepamornkul, name: Maochuan Lu, name: Ahmed Alaa, name: Thomas Hartvigsen, name: Gopala Anumanchipalli

Prior work in parameter-modifying knowledge editing has shown that
large-scale sequential editing leads to significant model degradation. In this
paper, we study the reasons behind this and scale sequential knowledge editing
to 10,000 sequential edits, while maintaining the downstream performance of the
original model. We first show that locate-then-edit knowledge editing methods
lead to overfitting on the edited facts. We also show that continuous knowledge
editing using these methods leads to disproportionate growth in the norm of the
edited matrix. We then provide a crucial insight into the inner workings of
locate-then-edit methods. We show that norm-growth is a hidden trick employed
by these methods that gives larger importance to the output activations
produced from the edited layers. With this "importance hacking", the edited
layers provide a much larger contributions to the model's output. To mitigate
these issues, we present ENCORE - Early stopping and Norm-Constrained Robust
knowledge Editing. ENCORE controls for overfitting and the disproportionate
norm-growth to enable long-term sequential editing, where we are able to
perform up to 10,000 sequential edits without loss of downstream performance.
ENCORE is also 61% faster than MEMIT and 64% faster than AlphaEdit on
Llama3-8B.