Title: SAMerging: Sharpness-aware Model Merging via Multi-Teacher Knowledge Distillation
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: model merging, multi-task learning, flat minima, knowledge distillation, generalization

Abstract: Model merging offers a lightweight alternative to joint multi-task learning (MTL), which is often costly or data-prohibitive. While the task arithmetic seems promising, it is brittle to coefficient scaling, and we observe that recent approaches, such as AdaMerging, that learn these coefficients, remain sensitive to initialization. This raises a key question: can merging coefficients be learned in a principled, label-free way? We introduce SAMerging, a method that learns coefficients by seeking flat minima. Our approach is grounded in two theoretical contributions. First, we derive a flatness-aware PAC-Bayes generalization bound for the merged model, featuring a novel cross-task heterogeneity term that quantifies expert-task mismatch. Second, this analysis guides us to frame merging as multi-teacher knowledge distillation on a small, unlabeled dataset. We formally show that minimizing the student-teacher KL divergence tightens an upper bound on the merged model's excess risk. We then employ Sharpness-Aware Minimization (SAM) to find robust solutions that generalize better. Empirically, SAMerging establishes a new state of the art on vision and NLP benchmarks. Notably, it surpasses AdaMerging with accuracy gains of $+4.5\%$ on TA-8 and $+11.7\%$ on TALL-20. This is achieved with remarkable data efficiency, using $10\times$ fewer calibration data and proving effective even in data-scarce settings with as few as $16$ examples per task. Furthermore, it requires no original training data and incurs no additional inference-time or memory overhead.

TL;DR: We introduce SAMerging, a method for model merging that does sharpness-aware multi-teacher knowledge distillation.