Title: MIN-MERGING: MERGE THE IMPORTANT NEURONS FOR MODEL MERGING
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Model Merging, LLM, Router, Pruning

Abstract: Recent advances in deep learning have led to a surge of open-source models across diverse domains. While model merging offers a promising way to combine their strengths, existing approaches often suffer from parameter conflicts that degrade performance on domain-specific tasks. We propose MIN-Merging, a router-based framework that selectively merges the most important neurons to reduce such conflicts.Extensive experiments on Computer Vision and Natural Language Processing benchmarks show that MIN-Merging achieves consistent gains on in-domain tasks while retaining the generalization ability of pretrained models on out-of-domain tasks. These results highlight its effectiveness as a practical solution to the parameter conflict problem in model merging.

TL;DR: We propose a comprehensive solution to address the parameter conflicts in model merging.