Should We Really Edit Language Models? On the Evaluation of Edited
  Language Models
Authors: name: Qi Li; name: Xiang Liu; name: Zhenheng Tang; name: Peijie Dong; name: Zeyu Li; name: Xinglin Pan; name: Xiaowen Chu
Published: 2024-10-24 14:36:48

Abstract: Model editing has become an increasingly popular alternative for efficiently
updating knowledge within language models. Current methods mainly focus on
reliability, generalization, and locality, with many methods excelling across
these criteria. Some recent works disclose the pitfalls of these editing
methods such as knowledge distortion or conflict. However, the general
abilities of post-edited language models remain unexplored. In this paper, we
perform a comprehensive evaluation on various editing methods and different
language models, and have following findings. (1) Existing editing methods lead
to inevitable performance deterioration on general benchmarks, indicating that
existing editing methods maintain the general abilities of the model within
only a few dozen edits. When the number of edits is slightly large, the
intrinsic knowledge structure of the model is disrupted or even completely
damaged. (2) Instruction-tuned models are more robust to editing, showing less
performance drop on general knowledge after editing. (3) Language model with
large scale is more resistant to editing compared to small model. (4) The
safety of the edited model, is significantly weakened, even for those
safety-aligned models. Our findings indicate that current editing methods are
only suitable for small-scale knowledge updates within language models, which
motivates further research on more practical and reliable editing methods. The
details of code and reproduction can be found in
https://github.com/lqinfdim/EditingEvaluation.