Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model
  Editing with Llama-3
Authors: name: Junsang Yoon; name: Akshat Gupta; name: Gopala Anumanchipalli
Published: 2024-05-01 17:50:37

Abstract: This study presents a targeted model editing analysis focused on the latest
large language model, Llama-3. We explore the efficacy of popular model editing
techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
interventions. We identify the most effective layers for targeted edits through
an evaluation that encompasses up to 4096 edits across three distinct
strategies: sequential editing, batch editing, and a hybrid approach we call as
sequential-batch editing. Our findings indicate that increasing edit
batch-sizes may degrade model performance more significantly than using smaller
edit batches sequentially for equal number of edits. With this, we argue that
sequential model editing is an important component for scaling model editing
methods and future research should focus on methods that combine both batched
and sequential editing. This observation suggests a potential limitation in
current model editing methods which push towards bigger edit batch sizes, and
we hope it paves way for future investigations into optimizing batch sizes and
model editing performance.