Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for
  Text-to-Image Diffusion Model Unlearning
Authors: name: Saemi Moon; name: Minjong Lee; name: Sangdon Park; name: Dongwoo Kim
Published: 2024-10-08 03:30:39

Abstract: As text-to-image diffusion models become advanced enough for commercial
applications, there is also increasing concern about their potential for
malicious and harmful use. Model unlearning has been proposed to mitigate the
concerns by removing undesired and potentially harmful information from the
pre-trained model. So far, the success of unlearning is mainly measured by
whether the unlearned model can generate a target concept while maintaining
image quality. However, unlearning is typically tested under limited scenarios,
and the side effects of unlearning have barely been studied in the current
literature. In this work, we thoroughly analyze unlearning under various
scenarios with five key aspects. Our investigation reveals that every method
has side effects or limitations, especially in more complex and realistic
situations. By releasing our comprehensive evaluation framework with the source
codes and artifacts, we hope to inspire further research in this area, leading
to more reliable and effective unlearning methods.