Model Merging with Functional Dual Anchors
Authors: name: Kexuan Shi; name: Yandong Wen; name: Weiyang Liu
Published: 2025-10-24 07:54:06

Abstract: Model merging is an efficient post-training strategy for integrating
knowledge from multiple finetuned checkpoints of a shared foundation model.
Existing methods operate in the parameter space, combining task vectors to
mitigate conflicts, but remain constrained by parameter inconsistencies. We
propose Functional Dual Anchors (FDAs), a framework that instead models the
input-representation space. FDAs are synthetic inputs whose induced gradients
align with task vectors, capturing task-specific functional shifts relative to
the pretrained model. This perspective bridges joint multi-task training and
post-hoc merging, offering both robustness and flexibility. We further
introduce a principled initialization scheme and show that FDAs are
complementary to parameter-space model merging. Comprehensive experiments
demonstrate the effectiveness of FDAs in model merging.