ICML 2025

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Kunda Yan * 1 Min Zhang * 2 Sen Cui † 1 Zikun Qu 3 Bo Jiang 2 Feng Liu 4 Changshui Zhang † 1
Abstract
Model merging aims to integrate the strengths of
multiple fine-tuned models into a unified model
while preserving task-specific capabilities. Ex-
isting methods, represented by task arithmetic,
are typically classified into global- and local-
aware methods. However, global-aware meth-
ods inevitably cause parameter interference, while
local-aware methods struggle to maintain the ef-
fectiveness of task-specific details in the merged
model. To address these limitations, we propose
a Consensus-Aware Localized Merging (CALM)
method which incorporates localized informa-
tion aligned with global task consensus, ensur-
ing its effectiveness post-merging. CALM con-
sists of three key components: (1) class-balanced
entropy minimization sampling, providing a
more flexible and reliable way to leverage un-
supervised data; (2) an efficient-aware frame-
work, selecting a small set of tasks for sequential
merging with high scalability; (3) a consensus-
aware mask optimization, aligning localized bi-
nary masks with global task consensus and merg-
ing them conflict-free. Experiments demonstrate
the superiority and robustness of our CALM, sig-
nificantly outperforming existing methods and
achieving performance close to traditional MTL.
1. Introduction
Multi-task learning (MTL) facilitates knowledge transfer
across different tasks through a shared backbone, improving
*Equal contribution † Corresponding author 1Institute for Artifi-
cial Intelligence, Tsinghua University (THUAI), Beijing National
Research Center for Information Science and Technology (BN-
Rist), Department of Automation, Tsinghua University, Beijing,
P.R.China 2East China Normal University (ECNU) 3The Chinese
University of Hong Kong, Shenzhen 4School of Computing and In-
formation Systems, The University of Melbourne. Correspondence
to: Sen Cui <cuis@ma...