Knowledge Fusion By Evolving Weights of Language Models (2024)

Abstract: Fine-tuning pre-trained language models, particularly large language models, demands extensive computing resources and can result in varying performance outcomes across different domains and datasets. This paper examines the approach of integrating multiple models from diverse training scenarios into a unified model. This unified model excels across various data domains and exhibits the ability to generalize well on out-of-domain data. We propose a knowledge fusion method named Evolver, inspired by evolutionary algorithms, which does not need further training or additional training data. Specifically, our method involves aggregating the weights of different language models into a population and subsequently generating offspring models through mutation and crossover operations. These offspring models are then evaluated against their parents, allowing for the preservation of those models that show enhanced performance on development datasets. Importantly, our model evolving strategy can be seamlessly integrated with existing model merging frameworks, offering a versatile tool for model enhancement. Experimental results on mainstream language models (i.e., encoder-only, decoder-only, encoder-decoder) reveal that Evolver outperforms previous stateof-the-art models by large margins. The code is publicly available at https://github.com/ duguodong7/model-evolution.

heading: 1 Introduction
text: Due to the high training costs of large language models, it is common practice to fine-tune already pre-trained language models to adapt them for specific applications. This fine-tuning approach often allows us to achieve excellent performance in specific data domains or tasks at a relatively lower cost (Chen et al., 2021). However, the challenge lies in the fact that fine-tuning (Dodge et al., 2020)
* Corresponding authors.
the same model in different task scenarios may result in performance variations, meaning that the results may not be satisfactory when testing the same model in different contexts. Therefore, our objective is to integrate knowledge from models trained in different scenarios to enhance the model’s performance in cross-domain or cross-task scenarios (Wortsman et al., 2022b), without the need for further training or extra training data.
Mainstream knowledge fusion methods can be divided into two main categories. The first involves extensive training on large datasets across multiple tasks to learn new model parameters with shared representations, such as in multi-task learning. The second relies on fusing existing models from specific scenarios without requiring extensive data. While multi-task learning generally improves overall performance, it has significant drawbacks: the need for abundant annotated data for all tasks and the complexity and time consumption of the training phase, especially with dataset combinations (Ruder, 2017). In contrast, model merging methods do not require retraining models and do not raise concerns about data privacy. In this paper, we primarily delve into the second category of methods and introduce an innovative model evolution approach inspired by Darwinian evolution theory (Shafiee et al., 2018). In short, we compare our model evolution approach with other prevalent knowledge fusion methods, detailing their distinct features in Table 1.
In fact, the problem of model merging can be reformulated as an optimization problem that aims to find the most valuable information for knowledge fusion in order to achieve better outcomes than any individual model alone. For instance, (Jin et al., 2023) employed a simple linear regression approach for optimization, while model soups (Wortsman et al., 2022a) implemented a greedy search method. In this paper, we consider the adoption of a more robust evolutionary algorithm for optimiza-
11727
tion. Evolutionary algorithms offer several advantages, including their outstanding performance in handling complex, high-dimensional, and nonlinear problems, as well as their relative insensitivity to local optima. Traditionally, evolutionary algorithms are primarily used in neural architecture search (NAS) (Awad et al., 2020). However, in this paper, we pioneer their application to the selection of important weights in language models for knowledge fusion.
As shown in Figure 1, our approach first processes models fine-tuned in different environments as an initial population. We then generate a new population through mutations and recombination among different individuals within the population. Subsequently, we validate the performance of the new population on development environment datasets and preserve elite individuals for updating. After evolving with enough generations, we select individuals with the best performance as the evolved model. Our evolutionary algorithm is firmly rooted in the task vector (Ilharco et al., 2022) methodology, wherein we derive the differential vectors between two individuals during the mutation operation. We delve deeper into the task vector concept from two distinct perspectives. Firstly, we employ a scaling factor (F ), to search and regulate the weights of the difference vectors. This is crucial for aligning the expectations of model outputs, as illustrated in (Yu et al., 2023). Secondly, we utilize the crossover ratio (Cr), to maintain a certain level of sparsity, thereby preventing interference among different individuals, as demonstrated in (Yadav et al., 2024). Additionally, by conducting random searches for crossover parameters, we are able to obtain more optimal per-parameter coefficients for model merging.
We conduct knowledge fusion experiments across various difficulty levels, employing different types of models, such as RoBERTa, DeBERTa, T5 and GPT2. These experiments cover sentiment classification tasks in diverse data domains, as well as benchmark tasks from the GLUE dataset. The
experimental results consistently demonstrate that our proposed method effectively enhances performance across all experimental settings. Furthermore, our approach can be synergistically combined with previous model merging methods (e.g., fisher (Matena and Raffel, 2022), regmean (Jin et al., 2023)), resulting in further improvements in knowledge fusion performance. This combined approach significantly outperforms baseline methods and previous techniques. Notably, our method also exhibits superior generalization performance when applied to data domains beyond the scope of multiple datasets. To summarize, our key contributions include:
• Innovative model evolution method: We propose a novel knowledge fusion method from an evolutionary perspective, by evolving weights of language models. • Improved knowledge fusion performance: Our method consistently enhances knowledge fusion performance across a broad spectrum of data domains and tasks.
• Effective integration with existing model merging methods: Our approach can be effectively enhanced and augmented through the integration of existing model merging techniques.

heading: 2 Related Work

heading: 2.1 Knowledge Fusion
text: Numerous studies have shown that aggregating knowledge from multiple datasets can enhance the performance of a single model across various data domains and different tasks (Poth et al., 2021). This approach is also applicable to out-of-domain data (Wang et al., 2020b). (Frankle et al., 2020) demonstrated the effectiveness of simple weight averaging in model fusion, exhibiting better performance than pre-training methods. (Matena and Raffel, 2022) proposed fisher-weighted averaging to merge models with different architectures, taking into account the importance of each parameter. (Jin et al., 2023) investigated model fusion using regression mean, re-weighting, and linearly combining rows within the weight matrix. (Wortsman et al., 2022a) introduced greedy soup, a technique to obtain robust results by searching for different average weights from multiple fine-tuned models. (Ilharco et al., 2022) proposed the concepts of task vectors to improve pre-trained models on multi-tasks and (Yadav et al., 2023) worked on the interference problems of task vectors to avoid the loss of valuable information. In addition to these knowledge fusion methods that do not require training, there are many knowledge fusion strategies that require complex training environments. Multi-task learning, as explored by (Ruder, 2017), improves performance across various tasks by sharing knowledge. Federated learning (Wang et al., 2020a) is a collaborative decentralized privacy-preserving technology designed to overcome the challenges of data silos and data sensitivity. Our evolutionary algorithm is firmly rooted in the task vector (Ilharco et al., 2022) methodology, wherein we derive the differential vectors between two individuals during the mutation operation. The methodology of task vectors has recently emerged as a promising approach for model merging and knowledge fusion, as evidenced by studies such as (Yadav et al., 2024; Ortiz-Jimenez et al., 2024). These investigations underscore the capacity of differential vectors, acquired through differentiation of pre-trained and fine-tuned models, to facilitate model editing.
Overall, our approach harnesses the power of task vectors to enhance model merging and knowledge fusion, offering insights into both the theoretical and practical aspects of this methodology.

heading: 2.2 Evolutionary algorithms
text: Of particular relevance to our work is evolving algorithms (EAs), which provide an alternative path for addressing optimization problems in deep neural networks (DNNs) without relying on gradient information. The fundamental idea behind EAs is to combine the structures and weights of a group of neural networks and continually evolve them in the direction of improved global fitness to enhance model performance. These methods encompass genetic algorithms (Montana et al., 1989), genetic programming (Suganuma et al., 2017), differential evolution (DE) (Pant et al., 2020; Jiang et al., 2024), and evolutionary strategies (Salimans et al., 2017), among others. Neuro-evolution techniques, such as NEAT (Neuro Evolution of Augmenting Topologies) (Stanley and Miikkulainen, 2002), have demonstrated the ability to design simpler neural network architectures for improved performance, particularly in reinforcement learning tasks. However, it’s important to note that EA methods typically perform well on small datasets and small-scale DNNs (Piotrowski, 2014). When applied to large-scale datasets, these methods tend to converge slowly and may even fail to converge (Piotrowski, 2014). In this paper, we propose model evolution which is motivated by the fact that model merging is neither amenable to traditional gradient-based optimization methods, nor are simple techniques like grid search sufficient. Therefore, we turn to evolutionary algorithms, which show promise for effectively addressing the model fusion problem.

heading: 3 Evolving Weights of Language Models
text: The goal of model evolution is to combine multiple fine-tuned language models into a more powerful single model. We achieve this by simulating the evolution process of a neural network population, as shown in Figure 2. We use the same pre-trained checkpoint and fine-tune it in different environments to create the initial population. As all individuals share the same model architecture, this enables our evolution algorithm to perform mutations and recombinations among individuals within the parameter space.
3.1 Evolutionary Strategy: Evolver
Population Initialization. For the optimization problem of model merging, an original set of individuals (population) is initialized. The parameters of each of N models are flattened into a onedimensional vector, forming a set of candidate solutions. In this way, we obtain a set of candidate individuals represented by θ = θi, i = 1, ..., N . Here, N denotes the size of the population, and θi = (θi,1, θi,2, ..., θi,d) represents each candidate individual, where d is the dimension.
Evolution Process. We simulate the evolution process of a population of neural networks using the differential evolution algorithm (Pant et al., 2020). Each generation consists of three main steps: mutation, crossover, and updating.
Mutation: For each candidate individual θi, we randomly select two other candidate individual θr1 and θr2 , where r1 and r2 are two distinct random integers less than or equal to m. We use a scaling factor F to adjust the differences between θr1 and θr2 , and then add them to θi to obtain the mutated solution θ⋆i = θi + F × (θr1 − θr2), where F is used to control the weights of the difference vector in the new parameter set.
Crossover: The computation for crossover is as follows:
θ⋆i,j = { θ⋆i,j if rand(0, 1) ≤ Cr, θi,j otherwise.
(1)
where Cr is the pre-set crossover degree threshold between the new individual and the parent solution, and the setting of the threshold Cr can impact the ratio of elements selected in a mutated solution.
Updating: Throughout the process, we convert the offspring population vectors into models and conduct inference to get performance scores for these models on the development dataset. As demonstrated in the equation below, We sequentially evaluate the performance scores of offspring individuals in comparison to their parent one by one. If an offspring performs better, we then replace the corresponding parent individual with it, thereby updating the parent population.
θi =
{ θ⋆i score(θ ⋆g i ) > score(θ g i )
θi otherwise. (2)

heading: 3.2 Integration with Other Merging Methods.
text: Our proposed model evolution method can be integrated with any other model merging technique. Specifically, we can select the optimal individual from the updated population as the outcome of the evolution process, which we refer to as a simple Evolver. Furthermore, we can also apply other model merging techniques to the updated population as a further improvement measure. Such integration can be implemented both at the stage of obtaining the final evolved model and during the calculation of the updated population’s scores for iterative updates. This process has been summarized as Algorithm 1 and is detailed in the flowchart provided in Appendix A.

heading: 3.3 Computation Efficiency
text: Memory Analysis: The memory expanse during our model evolution is mainly related to the size of the population: ∑N i=1 d, where N represents the number of populations, d is the dimension of the model parameter space. Since we avoid inner product matrices computing as in previous model
Algorithm 1 Integrating Model Evolver with Other Model Merging Methods
1: When inference with the updadted population 2: if this is a simple evolver 3: then 4: evaluate the performance of individual θ⋆i , 5: else 6: merging θ⋆i with other individuals with a
specific merging method, 7: evaluate the performance of merged model.
merging methods such as fisher and regmean, and the parameters is updated mainly through forward propagation of greedy models, the peak GPU memory consumption is consequently lower.
Time Consumption: We hereby provide the formula and key definitions required to calculate the runtime. The total evolving time can be calculated as T = G×N × (t1+L× t2) ≈ G×N ×L× t2, where t1 ≪ t2 in practice. Here, G is the total generations for updating, N is the population size, t1 is the time for mutation and crossover for each individual, L is the number of samples of development datasets, t2 is the time for inference of a sample on one model.

heading: 4 Experimental Setup

heading: 4.1 Evaluation Settings
text: Problems. We primarily consider the following three main advantages when testing our proposed model evolution method: Firstly, we anticipate that our evolved model fM , created by integrating knowledge from individual models f1..N finetuned on diverse datasets D1..N , will have competitive performance across various data sources without necessitating separate models for each domain or task. Then, by evolving different models excelling in various tasks Dt1..Nt , we aim to enhance multitask handling capacity, avoiding the complexity of retraining as in MTL, while enabling cross-task inferencing within a single model. Lastly, our goal is for the evolved model fM to excel in generalizing to out of distributio (OOD) test sets Do1..No , thereby enabling it to effectively handle new and unforeseen data from domains or tasks not encountered during training. D1..N .
Datasets. We use the GLUE datasets (Wang et al., 2018) as the cornerstone of our investigation into the performance of evolved models. This inquiry
encompasses two key dimensions: training for nonindependent and identically distributed (non-i.i.d.) partitions and training for disparate tasks. Detailed dataset information and additional experimental results are available in Appendix D.
Implementation. We use Hugging Face’s transformer library (Wolf et al., 2019) to access pre-trained language models and conduct finetuning. All our models, denoted as fi, follow the same architecture and employ identical pretrained model weights θ for initialization, as described in (McMahan et al., 2017). Our experiments include various pre-trained models as starting points, such as RoBERTa-base (Liu et al., 2019), the lightweight DistilBert (Khanuja et al., 2021) and well-established model DeBERTa-largev3 (He et al., 2022). Besides the models with encoder-only architecture, we also conduct experiments with encoder-decoder architecture, T5-basev1.1 (Raffel et al., 2020) and decoder-only architecture, GPT2 (Radford et al., 2019) and large language model MiniCPM (Hu et al., 2024).
Population Initialization. The initial population for model evolution is created through fine-tuning a model with the same initialization but on different data domains or various tasks. While fine-tuning DistilBERT-base, RoBERTa-base, and DeBERTalarge, we maintained a constant initial learning rate of 1e-5. Throughout our experiments, we consistently utilized the AdamW optimizer with a warm up learning rate during the initial 6% of training. Our model training utilized a batch size of 16 and encompassed 10 epochs for the GLUE task and 30 epochs for the emotion classification task.

heading: 4.2 Compared Methods
text: We mainly compare Evolver with existing merging methods, including simple, fisher (Matena and Raffel, 2022) , regmean (Jin et al., 2023) , greedy soup (Wortsman et al., 2022a) and TIES (Yadav et al., 2023). To gain a better grasp of the advantages of model merging, we show the performance prior to model evolution, the average performance of the population (Avg.f1..N ) and the bestperforming individual (Best.f1..N ), more details is shown in Appendix D.3. Moreover, we provide the performance for the model trained on a specific task domain-specific. We also compare with model ensembling, where the logits from predictions are extracted, averaged, and then subjected
to the argmax operation. Lastly, we use multi-task learning (MTL) as a benchmark.

heading: 5 Experimental Results
text: We assess the performance dynamics of the model evolution method across a range of scenarios with varying levels of complexity. These scenarios include: (1) performance across different data domains used for fine-tuning individual models. (2) performance across different tasks, when individual models are specialized in only one task. (3) OOD generalization performance on datasets from previously unseen domains.

heading: 5.1 Model Evolving Across Data Domains
text: Evolving All Domain-Specific Models. We conduct experiments of evolving five domain-specific models for emotion classification, and the results are recorded in Table 2. Multi-task learning (MTL) achieves performance similar to that of domainspecific models, suggesting that a single model has the capability to acquire knowledge from multiple domains. Besides, model soup approach, which greedily selects fusion objects, leads to some improvements over the best individual. However, these improvements are relatively marginal compared to model merging methods.
We compare model evolution with three other knowledge merging methods. The basic version of model evolution outperformed fisher method and achieved performance that is comparable to regmean on some tasks. Furthermore, we explore the combined use of model evolution and model merging methods, demonstrating that our approach further enhances existing model merging methods and consistently yields improvements across different models. Also, we demonstrate results with shared and different classification head initialization (Same Head Init/Diff Head Init). It can be observed that fisher and regmean produce unstable and highly variable results with different initialization, while the performance of the model evolution method is less affected by this factor. Therefore, our proposed model evolution method shows more stable performance when deploying and maintaining a single model across multiple domains.
Evolving Pairwise Domain-Specific Models In addition to the fusion experiment involving all the models, we also consider pairwise domain-specific models for knowledge fusion in the context of the emotion classification task. After pairing models
from a set of 5, we conduct 10 (C25 ) runs with all the same methods described in section 5.1. The result of these 10 runs are averaged and recorded in Table 3. We observe clear differences between model evolution and model merging methods, with the TIES-evolver achieving the best performance when combining pairwise models finetuned from different domains.
Evolving Models Trained on Non-i.i.d. Partitions. We adopt synthetic data divisions to simulate non-independent and identically distributed (non-i.i.d.) partitions of the same dataset, across the 8 tasks included in the GLUE benchmark. Given the inconsistency in the performance of regmean and fisher methods under different seeds, we choose to average the result of eight different random seeds. The results in Figure 3 indicate that the implementation of model evolution outperformed previous methods on all tasks, with clear improvements observed particularly on cola and mnli datasets. Details of this section are shown in Appendix D.4.

heading: 5.2 Evolving Models Across Different Tasks
text: Here we examine the effectiveness of model evolution in merging models finetuned on different tasks. We use the RoBERTa-base model and train indi-
vidual models with the complete training data for each task in the GLUE benchmark. Following this, we randomly select two task-biased individuals to conduct pairwise model evolution. Specifically, we exclude the parameters in task-specific headers due to their potential dimension variance depending on tasks. We summarize the results of eight different task pairs in Figure 4, which show that our model evolution strategy performs effectively when fusing knowledge from diverse tasks.

heading: 5.3 Model Evolving for Out-of-Domain Generalization
text: For Out-of-Domain (OOD) generalization, we can obtain the same conclusion as our in-domain experiments, indicating that model evolving leads to improvements in OOD generalization performance, as summarized in Table 4. We notice that in the case of RoBERTa-base and DistilBERT models initialized with different heads, the basic Evolver has outperformed fisher-evolver and regmean-evolver.
A plausible explanation for this is that previous methods may suffer from the negative impact of extremely poor-performing individual models. In contrast, model evolution possesses an elimination mechanism that effectively removes poorlyperformed individual models during the competition process. This ensures that only models with superior performance are retained and merged. Consequently, model evolution can reduce the negative impact of under-performing models on the merged results.

heading: 5.4 Mutation and Crossover
text: We also test the impact of different values of scale factor F for mutation and crossover ratio Cr, as shown in Figure 5. Due to the inherent randomness in the search process of evolutionary algorithms, we conducted each experiment using four different random seeds and then calculated the average results. In general, the study findings indicate that the performance of the evolutionary algorithm improves as the parameters F or Cr increase until reaching 0.5. However, when F or Cr exceeds 0.5, there is minimal improvement in performance, and no clear pattern is observed. Therefore, in all experiments conducted in this paper, we have consistently used F = 0.5 and Cr = 0.5.
In addition, regmean method requires decreasing the non-diagonal items of the inner product matrices by multiplying a scalar α. Since the effectiveness of the regmean-evolver method can be influenced by the hyperparameter α, we also test
the performance of regmean-evolver under different α parameters, as shown in Figure 6.

heading: 5.5 Integration with Coefficient Search
text: The coefficient search is a promising scheme to improve the model merging performance, by searching the optimal α. The proposed model evolution can also be integrated with the coefficient search method by searching the optimal scale factor f . We have performed the grid search of α and f with intervals of 0.05 from 0.1 to 0.9. We present the result in Table 5. In the setting of Simple, Fisher and RegMean, the results show that a default version of evolver (with scale factor f = 0.5, cr = 0.5) outperforms the coefficient search results. Notably, the integration with scale factor search further boosts the performance of the evolver, which is worth further investigation. Especially, the crossover ra-
tio Cr in model evolution could also be the subject of coefficient search. Many adaptive schemes are also available in the realm of evolution algorithms, like SADE (Qin and Suganthan, 2005) and SHADE (Tanabe and Fukunaga, 2013), providing a possibility of future algorithmic development.

heading: 5.6 Analysis
text: We demonstrate the evolutionary process when combined with other model fusion methods. From Figure 7, we can observe that when model evolver is combined with fisher or regmean method, the upper bounds of the evolutionary approach can be enhanced. Additionally, we present the test results of the evolutionary algorithm on the development dataset. It is evident that as individual models are trained on the development dataset, their performance on the test set gradually improves. This indicates that our model evolution method indeed has the ability to optimize and learn.
In addition, we visualize a two dimensional slice of the average test accuracy landscape when evolv-
ing pairwise fine-tuned T5 models, as shown in Figure 8. In this experiment, we use the zero-shot initialization and fine-tune twice, independently, to produce task vectors on MRPC and STSB datasets. We analyze the positions of the model results after fusion by different methods in their corresponding landscapes. Methods like TIES and other merging approaches typically process the task vectors of models on different tasks and then perform a direct weighted average. This results in a fixed relative scale among different tasks during model fusion. However, from the landscape, we can see that the optimal fusion performance usually has different preferences for different tasks. Our proposed model evolution method can dynamically update the scales of different tasks to promote better model fusion effects.
The proposed model evolution also has some advantages in other aspects: (1) Model evolution can leverage the benefits of a larger population size without being significantly affected by individuals with extremely poor performance. This advantage is a result of the survival of the fittest mechanism in the model evolution process. By favoring those with most effective performance, we reduce the influence of individuals with extreme differences, leading to a more robust and reliable system. (2) Model evolution method can effectively maintain low peak GPU memory usage. This is primarily attributed to its sequential forward inference of individual models, as opposed to most previous model merging techniques that require additional GPU memory for computing inner product matrices in
the model parameter space. This advantage significantly reduces GPU memory consumption and extends the range of feasible solutions for largescale language models.

heading: 6 Conclusions
text: We introduce a novel knowledge fusion method, called model evolution, inspired by evolutionary algorithms. This approach significantly boosts the performance of model merging in diverse NLP contexts. Model evolution stands out by aggregating model weights into a population and updating it with superior offspring models, all without requiring extra training data. The most significant contribution of our work lies in our experimental discovery that knowledge fusion can benefit from searching per-model and per-parameter coefficients. This type of search-based, gradient-free optimization algorithm proves to be an effective tool for model merging and warrants greater attention in the research community. Our extensive experiments validate its superiority over previous techniques.
Limitation
The limitations of the model evolution method can be summarized in three main aspects: (1) the necessity for a high-quality development dataset with consideration for data privacy, (2) the requirement for a cautious selection of hyperparameters F and Cr, (3) and the significance of conducting further theoretical analysis of evolutionary algorithm principles. Future research offers several promising directions. Firstly, exploring advanced optimization strategies within evolutionary algorithms, including adaptive approaches and hyperparameter selection based on historical performance, holds the potential for enhancing the method’s effectiveness. Secondly, extending knowledge fusion to a more complex training environment by considering hyperparameters, multimodal and exploring different training methods like unsupervised or supervised learning can provide a comprehensive understanding of its applicability. Thirdly, arithmetic operations in (Ilharco et al., 2022) for model edit can be analyzed. Lastly, evaluating the approach on larger language models can provide insights into its scalability.
Ethical Considerations
Our approach has been evaluated on GLUE. We explicitly claim that the applicability of our method and findings may be confined to similar datasets or domains. The performance of our method on other specific datasets or domains remains uncertain. Thus, there are potential risks when applying our method to privacy-sensitive or high-risk datasets. We should be cautious and verify whether the method generates correct answers.

heading: Acknowledgements
text: This work was supported in part by Shenzhen College Stability Support Plan (GXWD20231128103232001), Department of Science and Technology of Guangdong (2024A1515011540), Shenzhen Start-Up Research Funds (HA11409065), National Natural Science Foundation of China (12204130), the Ministry of Higher Education Malaysia through the Fundamental Research Grant Scheme (FRGS/1/2023/ICT02/XMU/02/1), and Xiamen University Malaysia through Xiamen University Malaysia Research Fund (XMUMRF/2022-C10/IECE/0039 and XMUMRF/2024-C13/IECE/0049).

heading: A Explanation of Model Evolution
text: In this section, we provide a more comprehensive explanation of the principles underlying the Differential Evolution algorithm, furthermore, we offer an in-depth elucidation of the mutation process, providing a visual representation in Figure 9 to enhance clarity and understanding. Overall, the fundamental principle of the differential evolution algorithm involves randomly selecting three distinct individuals, performing a mutation operation to create a new candidate solution, using a crossover operation to refine the solution, and replacing the original solution if the new one performs better. This iterative process continues until certain stopping criteria are met.
To better illustrate our model evolution method, we have created a flowchart as shown in algorithm 2. The details of combining our proposed method with other models are provided in Step 3. The approach involves calculating an overall score by using model merging on the mutated individuals along with the non-mutated individuals in the current evolution. In contrast, the simple evolver determines the success of mutation based on the score of individual entities, while the combined approach assesses it based on the score of the entire population after individual mutation.

heading: B Preliminaries
text: Fisher-weighted averaging (fisher) (Matena and Raffel, 2022) examines the importance of each weight Fi associated with each label by computing the norm of the logarithmic likelihood gradient. Specifically, the posterior probabilities of
each model are interpreted as gaussian distributions p(θ|θi, Fi), where the parameters θ for model i correspond to the Fisher information matrix Fθ. Finally, the fisher information for each parameter is used to perform a weighted average of the parameters, integrating the parameters of different models into a single model.
Regression mean (regmean) (Jin et al., 2023) expands the solution of a linear optimization problem to K models where Wi, i ∈ K, denoted as WM = ( ∑i∈K i X T i Xi) −1∑i∈K i (X T i XiWi). Each transformer model f (j) linear layer’s corresponding X(j)i is captured along with per-weights and its input inner product matrix, to compute the merged weights and produced merged model fM (x) = W T Mx. The scale of X T i Xi exhibits substantial variation across different models. Additionally, a control mechanism is applied by multiplying XTi Xi by α = 1 1+γ .
Model soups (greedy soup) (Wortsman et al., 2022a). Initially, models are ranked based on their development dataset scores. Subsequently, model parameters θi are chosen through a greedy search, and their inclusion in the gradient is determined by comparing the average validation set accuracy before and after their addition. The merged model’s parameters can be represented as θS = average(ingredients).
TIES merging (TIES) (Yadav et al., 2023). This method is based on the concept of task vectors (Ilharco et al., 2022) and address the problem of two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter’s values across models. This method proposed TRIM, ELECT SIGN & MERGE (TIES-MERGING), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign
C Impact of Development Dataset
The availability of development datasets directly impacts the effectiveness of our model evolution approach. However, many publicly available datasets either do not provide development sets or widely use them as test sets. In our case, the development set of the GLUE dataset is used as a test set, so we
Algorithm 2 Model Evolution 1: Step 1 - Initializing the Population 2: Initialize population Θ ▷ A population of candidate solutions 3: generation← 0 ▷ Initialize generation counter 4: converged← False ▷ Convergence flag 5: while not converged do 6: Step 2 - Evolution Process: Mutation and Recombination 7: for each candidate solution θi in Θ do 8: Randomly select θr1 and θr2 ▷ Select random solutions 9: F ← Random scaling factor ▷ Control parameter for mutation
10: Cr ← Random crossover rate ▷ Control parameter for recombination 11: Compute mutated solution θ⋆i using θi, θr1, θr2, and F 12: Perform recombination of θ⋆i based on Cr and θi 13: Step 3 - Model Inference 14: if not combined with other model merging methods then 15: Evaluate the performance of θ⋆i on development data 16: else 17: Merging θ⋆i with other models 18: Evaluate the performance of merged model on development data 19: end if 20: end for 21: Step 4 - Updating the Population 22: converged← True ▷ Assume convergence 23: for each candidate solution θi in Θ do 24: if θ⋆i outperforms θi then ▷ Comparing performance 25: Replace θi with θ⋆i ▷ Update population 26: converged← False ▷ Reset convergence flag 27: end if 28: end for 29: generation← generation+ 1 ▷ Increment generation counter 30: end while
utilize a small portion of the training dataset (approximately 5%) for model evolution. For non-i.i.d. partition methods, we also use only a subset of the same training data samples. In the case of the unified emotion dataset, we separately extract 10% of data from each of the five high-resource datasets for model evolution, following the same partitioning method as employed in (Jin et al., 2023).
For our model evolution approach, the quality of the development dataset can significantly impact performance, making the selection of a highquality development dataset a crucial consideration. To address this, we conducted experiments on the emotion dataset using different lengths of model evolution methods. We performed experiments with both the simple evolver and regmean evolver, evolving all five domain-specific models. The experimental results are shown in Table 6, indicating that even with a short development dataset, model evolution can still be effective. However, as the length of the development dataset increases, the performance of model evolution tends to improve. Additionally, we included the test scores of simple methods as baselines for comparison.

heading: D Dataset, Metrics and Training Details
text: D.1 Emotion Classification Datasets
In order to investigate the performance of the sentiment classification task, we selected a diverse and challenging set of datasets. Among them, DailyDialogs (Li et al., 2017), CrowdFlower, TEC (Mohammad, 2012), Tales-Emotion (Alm et al., 2005), and ISEAR (Scherer and Wallbott, 1994) is utilized to train domain-specific model. For acessing OOD generalization performance, we use Emoint (Mohammad and Bravo-Marquez, 2017), SSEC (Schuff et al., 2017), ElectoralTweets (Mohammad et al., 2015), GroundedEmotions (Liu et al., 2017), and AffectiveText (Strapparava and Mihalcea, 2007). For OOD evaluation, we focus exclusively on the fundamental emotions: anger, disgust, fear, joy,
sadness, and surprise. A detailed overview of the datasets and statistics is provided in Table 7.
D.2 GLUE Benchmark Datasets
For the GLUE dataset, we utilized multiple tasks, including CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017), MNLI (Williams et al., 2018),QNLI (Rajpurkar et al., 2016), QQP, and RTE (Giampiccolo et al., 2007). These tasks cover various natural language understanding problems such as text classification, text similarity, and natural language inference. To assess our merged models, we tested them on the official development sets. We performed experiments by training models on non-i.i.d. partitions, creating various partition scenarios through random sampling. Each partition is uniformly sub-sampled to yield a total of 1,000 training examples per partition.
D.3 Definitions and Metrics
The performance of individual models involved in merging are reported: (1) the average performance of all individual models (Avg. f1..N ); (2) the performance of the best single individual model (Best. f1..N ), as determined by using the validation set; (3) the performance of the individual models corresponding to the training data set for each test set (Domain-Specific).
In evaluating merged models trained for noni.i.d. partitions of the same dataset, we assessed their performance using a unified test set characterized by a joint distribution of all partitions. For merged models trained across different domains or tasks, we measured their performance across individual domains or tasks incorporated into the
merger and derived their macro-average. Similarly, when evaluating out-of-domain performance, we computed the macro-average of their performance across the out-of-domain test set.
D.4 Merging Models Trained on Non-i.i.d. Partitions.
Merging models initially trained on non-i.i.d. partitions of the same dataset is started, which is achieved by simulating synthetic data splits across the 8 tasks within the GLUE benchmark. Each task involves dividing the training data into two partitions, each containing 1,000 training examples with distinct label distributions. Following this, we perform fine-tuning on these two partitions for 8 pairs of individual models and merge each pair of models. The evaluation of these merged models takes place on the official validation sets, which portray a joint distribution of both partitions.
D.5 Time Cost We report the time cost of the scheme of model evolution, as shown in Table 8. T5-base model and MiniCPM model are tested on single NVIDIA A800 80G GPU and other models are tested on single RTX A6000 48G GPU. We find that all task of model evolution can be completed within half an hour, which is very cost-efficient in improving the model performance without further training.

heading: Future Work
text: Future research offers several promising directions. Firstly, exploring advanced optimization strategies within evolutionary algorithms, including adaptive approaches and hyperparameter selection based on historical performance, holds the potential for enhancing the method’s effectiveness. Secondly, extending knowledge fusion to a more complex training environment by considering hyperparameters, multimodal and exploring different training methods like unsupervised or supervised learning can provide a comprehensive understanding of its applicability. Thirdly, arithmetic operations in (Ilharco et al. , 2022) for model edit can be analyzed. Lastly, evaluating the approach on larger language models can provide insights into its scalability. Ethical Considerations
Our approach has been evaluated on GLUE. We explicitly claim that the applicability of our method and findings may be confined to similar datasets or domains. The performance of our method on other specific datasets or domains remains uncertain. Thus, there are potential risks when applying our method to privacy-sensitive or high-risk datasets. We should be cautious and verify whether the method generates correct answers