## 1 Introduction  
Large Language Models (LLMs) are advanced AI systems, containing billions or even trillions of parameters, that leverage deep learning and Natural Language Processing (NLP) techniques to understand and generate natural language text [1]. Trained on extensive textual datasets, these models capture intricate language structures and semantic relationships, enabling a wide range of tasks such as language comprehension, text generation, knowledge-based question answering, code generation, and conversational systems [2, 3]. Recent advancements in models such as GPT [4] and the LLaMA series [5] have resulted in significant performance improvements, while also addressing challenges related to security, efficiency, and interpretability. A key optimization strategy is model fusion, which combines parameters from multiple independent models to form a more versatile, general-purpose model. This approach, akin to ensemble learning, allows for knowledge transfer without requiring access to the original training data or incurring substantial computational costs. Model fusion enhances both accuracy and robustness, thereby expanding the applicability of LLMs to complex tasks. Model fusion techniques are broadly categorized into pre-fusion and in-fusion methods [6]. Pre-fusion methods, such as Linearization Fine-tuning [7], improve model weight decoupling in the tangent space of pre-trained models. Architecture Transformation [6] standardizes models with varying architectures into a unified structure, enabling parameter-level fusion. Techniques such as GAN Cocktail [8] map models into a shared parameter space, followed by averaging and fine-tuning for domain-specific applications. [PERSON] et al. [9] fused heterogeneous neural networks of varying depths using cross-layer alignment, without increasing network size or requiring access to original training data. In contrast, in-fusion methods [6] dynamically merge models during the fusion process, providing greater flexibility and adaptability, particularly for varying tasks or data samples. These methods adjust weight importance during fusion, allowing more precise control over the merging process compared to basic averaging or linear combinations. [PERSON] et al. [10] introduced an evolutionary algorithm to identify optimal merging combinations for various open-source models, producing new foundational models with specific capabilities. [PERSON] et al. [11] employed Bayesian optimization to determine the optimal merging weights during pretraining, aiming to reduce resource consumption while improving performance. [PERSON] et al. [12] conceptualized model merging as a multi-task learning problem, deriving optimal merging coefficients through local linearization and task vector orthogonality. Subspace-based Merging Methods [6] mitigate task interference by transforming models into sparse subspaces. [PERSON] et al. [13] addressed parameter interference by pruning fine-grained parameters, resolving symbol conflicts, and merging consistent parameters, thus enhancing fusion efficiency. Despite their advantages, in-fusion methods continue to face challenges in cross-domain applications and in determining the optimal fusion model weights and parameters, especially for complex or heterogeneous model structures. An important avenue in model fusion research involves enhancing domain-specific capabilities during the fusion process. This approach emphasizes tailoring models to optimize performance in specialized domains, as different tasks may benefit from distinct parameters. For instance, [PERSON] et al. [14] introduced the Joint Autoregressive Mixture (JAM) framework, which fuses large autoregressive text-to-image and language models into a unified system capable of generating high-quality multimodal outputs. [PERSON] et al. [15] presented a method for fusing multimodal LLMs spanning image, audio, video, and point cloud modalities, reducing cross-modal interference through parameter decoupling and adjustment of modal fusion coefficients. [PERSON] et al.