PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for
  Multimodal Large Language Models
Authors: name: Zhaopan Xu; name: Pengfei Zhou; name: Weidong Tang; name: Jiaxin Ai; name: Wangbo Zhao; name: Xiaojiang Peng; name: Kai Wang; name: Yang You; name: Wenqi Shao; name: Hongxun Yao; name: Kaipeng Zhang
Published: 2025-03-16 15:26:20

Abstract: In recent years, Multimodal Large Language Models (MLLMs) have demonstrated
remarkable advancements in tasks such as visual question answering, visual
understanding, and reasoning. However, this impressive progress relies on vast
amounts of data collected from the internet, raising significant concerns about
privacy and security. To address these issues, machine unlearning (MU) has
emerged as a promising solution, enabling the removal of specific knowledge
from an already trained model without requiring retraining from scratch.
Although MU for MLLMs has gained attention, current evaluations of its efficacy
remain incomplete, and the underlying problem is often poorly defined, which
hinders the development of strategies for creating more secure and trustworthy
systems. To bridge this gap, we introduce a benchmark, named PEBench, which
includes a dataset of personal entities and corresponding general event scenes,
designed to comprehensively assess the performance of MU for MLLMs. Through
PEBench, we aim to provide a standardized and robust framework to advance
research in secure and privacy-preserving multimodal models. We benchmarked 6
MU methods, revealing their strengths and limitations, and shedding light on
key challenges and opportunities for MU in MLLMs.