Title: From Coefficients to Directions: Rethinking Model Merging with Directional Alignment
Primary area: foundation or frontier models, including LLMs
Keywords: model merging

Abstract: Model merging is a recently emerging paradigm that aims to integrate multiple independently trained models into a single model without joint retraining. Previous studies have demonstrated the effectiveness of combining parameters from multiple models through strategies such as parameter decomposition, coefficient optimization and subspace learning. These methods significantly reduce the need for expensive joint training and have shown strong empirical performance across diverse tasks.    However, these approaches predominantly treat merging as a problem of parameter or coefficient adjustment, while largely overlooking the role of directional information in both parameter space and feature space. In this work, we emphasis the role of directional alignment and introduce a unified framework, \emph{Merging with Directional Alignment}~(MDA), which aligns direction in both parameter and feature spaces. Building on insights from neural collapse, we show that naive merging disrupts the underlying simplex \emph{equiangular tight frame} (ETF) geometry, resulting in performance degradation. Our theoretical analysis demonstrates that directional alignment improves structural coherence and tightens generalization bounds. Extensive experiments across benchmarks, model scales, and task configurations confirm the effectiveness of our approach, yielding consistent gains over prior merging techniques.