Title: Gram-weighted Mahalanobis Fréchet Mean: A Hyperparameter-Tuning-Free Solution for Model Merging
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Model Merging, Multi-task Learning, Task Arithmetic

Abstract: Model merging has emerged as a promising technique for integrating multiple fine-tuned models into a single unified model without additional training. This paradigm is particularly appealing in resource-constrained scenarios where data access or retraining is infeasible. Existing techniques—such as Task Arithmetic, Ties-Merging, and AdaMerging—achieve competitive results but typically rely on extensive hyperparameter tuning, which is costly and impractical for large-scale models. In this work, we introduce a hyperparameter-free merging method that reframes the problem as estimating a merged task vector capturing the principal directions of each task (e.g., dominant singular vectors). We formalize this process as the Gram-weighted Mahalanobis Fréchet mean (GMF-Mean), a convex optimization problem that admits a closed-form solution. Our theoretical analysis shows that GMF-Mean naturally adapts to both orthogonal (non-interfering) and conflicting (collinear but opposing) task interactions by leveraging the magnitude of singular values. This adaptation eliminates the need for scaling coefficients that are commonly required in Task Arithmetic-based methods. Empirical results on both vision, language, and vision-language models show that GMF Mean delivers competitive performance with state-of-the-art baselines, while maintaining the advantages of being training-free, data-free, and hyperparameter-tuning-free. These properties make GMF-Mean a practical solution for real-world deployment.