ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging
Authors: name: Haoming Xu; name: Shuxun Wang; name: Yanqiu Zhao; name: Yi Zhong; name: Ziyan Jiang; name: Ningyuan Zhao; name: Shumin Deng; name: Huajun Chen; name: Ningyu Zhang
Published: 2025-03-27 02:03:25

Abstract: This paper presents the ZJUKLAB team's submission for SemEval-2025 Task 4:
Unlearning Sensitive Content from Large Language Models. This task aims to
selectively erase sensitive knowledge from large language models, avoiding both
over-forgetting and under-forgetting issues. We propose an unlearning system
that leverages Model Merging (specifically TIES-Merging), combining two
specialized models into a more balanced unlearned model. Our system achieves
competitive results, ranking second among 26 teams, with an online score of
0.944 for Task Aggregate and 0.487 for overall Aggregate. In this paper, we
also conduct local experiments and perform a comprehensive analysis of the
unlearning process, examining performance trajectories, loss dynamics, and
weight perspectives, along with several supplementary experiments, to
understand the effectiveness of our method. Furthermore, we analyze the
shortcomings of our method and evaluation metrics, emphasizing that MIA scores
and ROUGE-based metrics alone are insufficient to fully evaluate successful
unlearning. Finally, we emphasize the need for more comprehensive evaluation
methodologies and rethinking of unlearning objectives in future research. Code
is available at https://github.com/zjunlp/unlearn/tree/main/semeval25.