ReLearn: Unlearning via Learning for Large Language Models
Authors: name: Haoming Xu; name: Ningyuan Zhao; name: Liming Yang; name: Sendong Zhao; name: Shumin Deng; name: Mengru Wang; name: Bryan Hooi; name: Nay Oo; name: Huajun Chen; name: Ningyu Zhang
Published: 2025-02-16 16:31:00

Abstract: Current unlearning methods for large language models usually rely on reverse
optimization to reduce target token probabilities. However, this paradigm
disrupts the subsequent tokens prediction, degrading model performance and
linguistic coherence. Moreover, existing evaluation metrics overemphasize
contextual forgetting while inadequately assessing response fluency and
relevance. To address these challenges, we propose ReLearn, a data augmentation
and fine-tuning pipeline for effective unlearning, along with a comprehensive
evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR)
and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and
Linguistic Score (LS) to evaluate generation quality. Our experiments show that
ReLearn successfully achieves targeted forgetting while preserving high-quality
output. Through mechanistic analysis, we further demonstrate how reverse
optimization disrupts coherent text generation, while ReLearn preserves this
essential capability. Code is available at https://github.com/zjunlp/unlearn.