Improving Model Merging with Natural Niches

Authors: joão p abrantes
robert tjarko lange
yujin  tang
sakana  ai
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 11:55:26

Abstract: Model merging is a powerful technique to combine specialized knowledge of multiple machine learning models into a single unified model.
However, current methods require manually partitioning the model parameters into a fixed number of groups to be merged, which constraints the exploration of potential combinations and limits performance.
To address these limitations, we propose an evolutionary algorithm with three key features:
(1) dynamically adjustment of merging boundaries to progressively explore a broader range of parameter combinations;
(2) a diversity preservation mechanism inspired by nature, which maintains a population of diverse, high-performing models that are particularly effective for merging;
and (3) a heuristic-based \textit{mate selection} strategy to identify the most promising pairs of models for merging. Our experimental results show, for the first time, that model merging can be used to evolve models from \textit{scratch}.
Specifically, we evolve MNIST classifiers from scratch using our method, and achieve comparable performance to CMA-ES, while being computationally cheaper.
Additionally, we use our method to merge specialised language models and obtain state-of-the-art performance.
Our code is available at https://github.com/AnonScientist/natural_niches.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have evolved MNIST classifiers from scratch and achieved comparable performance to CMA-ES. Additionally, we applied our method to LLM merging and achieved state-of-the-art performance.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
model merging: 
 Model merging introduces an innovative approach for integrating the strengths of multiple pre-trained models. In contrast to fine-tuning, which focuses on refining a single pre-trained model, model merging can leverage several models concurrently without requiring backpropagation. This has allowed the method to combine extremely large models for tasks involving subjective goals, like customizing an image generation model to reflect personal tastes.Notably, the release of Stable Diffusion (SD) [21] and open-source interfaces [3] enabled practitioners to merge different SD fine-tunes manually, using techniques like linear and spherical linear interpolation (SLERP) [25] . These early efforts demonstrated the potential of model merging in combining specialized capabilities into a single unified model.Subsequent research has approached the model merging problem from two complementary directions: minimizing interference between models and automating the merging process. Methods such as TIES [29] and DARE [31] introduced strategies to balance the contributions of individual models while minimizing interference, ensuring that the strengths of each model are preserved without mutual disruption.Evolutionary algorithms like CMA-ES [11] were later applied to automate the search for optimal merging coefficients. As explored in [1] , these methods not only automate what was previously a manual, iterative process but also significantly improve merging efficiency.While previous research centered on merging pre-trained models, we show that merging can efficiently be used to evolve models from scratch. Additionally, unlike earlier methods that required manual partitioning of model parameters, we automate and optimize this process during the evolutionary process.Diversity preservation in Genetic Algorithms (GA) is crucial for finding multiple solutions to multimodal problems [26] and to prevent premature convergence. We believe this is particularly important when using crossover operations (such as model merging). These operations benefit from diversity while at the same time reducing it, which may lead to premature convergence if not counter-acted by a diversity increasing mechanism. In this section, we provide a quick overview of the two main methods for diversity preservation in GA: 1) crowding [7, 27] and 2) fitness sharing [10, 8, 9, 20] .Crowding methods involve first applying mutation and crossover to produce new candidate solutions. These candidates then compete for inclusion in the population, but only against other candidates that are similar, based on a predefined criteria such as genetic or phenotypic distances. This selective competition helps maintain diversity within the population by preventing any single solution type from becoming overly dominant. A similar mechanism for selective competition is used in the popular algorithm of MAP-Elites [18] . In MAP-Elites, the solution space is divided into a multidimensional grid, with each cell representing a species defined by one or more predefined behaviour descriptors. New candidates are placed into cells based on their descriptors and replace existing solutions only if they perform better. The real challenge of this method lies in defining descriptors that promote the desired type of diversity.Fitness sharing requires each individual to share its rewards with others. In explicit fitness sharing, the researcher defines a distance function that is used to cluster similar individuals into a species, each individual then shares its fitness with other members of its species, making it more difficult for any single species to grow excessively large. A notable example is the NEAT [24] algorithm, known for evolving neural networks topologies, which clusters solutions into species by measuring genotypic differences (distances in network topologies). Implicit fitness sharing [23, 6] , is seen as the more natural method because, as in Nature, it protects niches rather than species. A niche is a group of individuals that compete for the same resources, while a species is defined as group that can interbreed and typically have small genetic and phenotypic differences. Usually, members of the same species compete for the same resources (e.g., food, partners, shelter), but vastly different species can also compete for the same vital resources like nesting sites or food sources (e.g., birds and bats, lions and hyenas). Implicit sharing does not rely on custom distance metrics. Instead, it simulates natural competition for limited resources, promoting diversity as individuals who can derive their fitness from less contested resources are favoured. We provide more details on Section 3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have a section that addresses the limitations of this work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
natural niches: 
 In model merging, the goal is to find the optimal parameters θ * for a merged model from a set of K seed models, each of which characterized by its model parameters θ i (i = 1 • • • K), such that optimization goal, normally in the form of task scores summation or average, is maximized. The following equation expresses this description mathematically:EQUATIONwhere h w is the model merging function parameterized by w's that correspond to fixed model merging boundaries (e.g., one scalar w k,l for the l-th layer in the k-th seed model), s is the score function for a certain task, x j is a task example, and N is the number of examples to be evaluated. In Natural Niches (NN), we propose modifications to the merging function h to enable evolution of the merging boundaries, and adjustments to the optimization goal to promote diverse solutions.In the formulation above, finding θ * boils down to searching for the optimal model merging parameters w in h w . To get rid of the constraints of fixed model merging boundaries and thus allows more flexibility, we propose to include these boundaries together with the mixing parameters into the evolutionary process. Concretely, NN maintains an archive of models, which is initialized with the K seed models. At each training step, NN randomly picks two models A and B from the archive, and samples two parameters (w m , w s ) that determines the mixing ratio and the split-point in the models' parameters space. It then merges models A and B with the following formula, and inserts the new model into the archive if it outperforms the worst individual.EQUATIONHere, θ <ws and θ ≥ws indicate the sub-arrays of model parameters before and after the split-point indexed by w s . f t (θ A , θ B ) is a spherical linear interpolation of rotations (SLERP) function that interpolates (θ A , θ B ) with t. As shown in the right part of Figure 1 , our method incrementally expands the search space by exploring a broader set of boundaries and coefficients. This gradual introduction of complexity ensures a wider range of possibilities while maintaining computational tractability.Competing for limited resources naturally promotes diversity, favoring individuals who can tap into less contested resources. In the context of the optimization goal in Equation 1, where a sum of scores from all the examples is being maximised, each score is a "resource" that contributes to the fitness of a solution. By limiting the resource supply, NN sparks competition which naturally favors individuals that take over new niches. Concretely, we limit the total fitness a population can extract from a data point x j by a capacity c j . The amount of fitness a candidate solution derives from a data point is proportional to its score relative to the aggregate score of the population. Our modified goal becomes:EQUATIONwhere ϵ in the denominator is a small number to prevent the zero-division error. In the term that defines z j , P is the archive size. The capacity c j , is task dependent and can be defined in multiple ways. For example, in binary scoring tasks (i.e. s(•) ∈ {0, 1}) we simply set c j = 1. In one experiment (WebShop) the environment offers a continuous reward from 0 to 1. Here, we define c j = max i s(x j |θ i ) to ensure that partially solved data points (where max i s(x j |θ i ) < 1) do not distribute the same amount of fitness points as fully solved data points (where max i s(x j |θ i ) = 1).Many evolutionary algorithms use the crossover operation to combine the strengths of both parents.In biology, this combination (i.e., reproduction) is very expensive, and therefore, animals invest many resources in the process of mate selection. We believe that as we make use of more expensive crossover operations, like model merging, algorithms for mate selection become increasingly important.In contrast to conventional methods that put more sampling probability mass on top performing models in the archive, NN adds an extra layer of consideration that takes into account the complementarity of the parent models. Specifically, we sample the first parent based on their weighted sum of scores defined in Equation 3, and then sample the second parent based on a "matching score" generated by function g that is specifically tailored for the first parent. The equation below gives the definition of this matching score, it straightforwardly expresses a desire to choose a model B that performs well in the data points where model A performs less well, while giving an extra preference to resources with high capacity c j and low competition z j .EQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The link to our code repo is on the abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We verify the effectiveness of our proposed method on two challenging tasks: First, we evolve image classifiers from scratch and from pre-trained models on the MNIST dataset, and then we scale up the experiment to merging LLMs.Model. The model being optimized is a two-layer feedforward neural network with 19,210 parameters in total. When starting from scratch, we randomly initialize the models. For pre-trained models, we develop two specialized models: one is trained on digits 0 through 4, and the other is trained on digits 5 through 9.Dataset. We've used the MNIST [15] dataset from scikit-learn [19] where each digit is a 8x8 gray scaled image. 80% of the data was used as the training split, and 20% as the testing split.Baselines. For the MAP-Elites algorithm, we used two diversity dimensions to create a 10 by 10 grid: the accuracy of the model in odd and even numbers. When starting from scratch, we use CMA-ES [11] as a baseline, even though it does not perform model merging here. Since the models are randomly initialized, optimizing mixing coefficients alone would be insufficient. Instead, CMA-ES directly optimizes model weights, which incurs a cubic computational cost O(n 3 ) with respect to the number of parameters. While this method doesn't scale to larger models, it serves as a benchmark for how a popular evolutionary algorithm performs in this experiment. When working with pre-trained models, we use a brute-force search baseline that merges the two seed models by adjusting a mixing coefficient that ranges from 0 to 1 in increments of 0.00001. This baseline is first evaluated on the training data, and the best coefficient is subsequently evaluated on the test data.Evolutionary Operators and Variables. All model merging methods (which excludes CMA-ES) sample a new candidate at a time and decide sequentially whether to insert the candidate into the archive. NN and GA use an archive of 20, sampling each candidate sequentially and deciding whether to insert it, similar to MAP-Elites. MAP-Elites, uses a 10x10 grid, resulting in an archive size of 100. CMA-ES uses a population of 20, sampling and updating its parameters in batches. When starting from scratch all model merging methods use the same mutation operation (Gaussian noise) and the same crossover operation (SLERP with split-point, as described in section 3.1). However, when dealing with pre-trained models, mutation is omitted because random alterations don't scale well to larger models. By avoiding mutation, we can assess which method is most promising for achieving efficient merging in larger pre-trained models.Compute Resources. The 10 independent runs took about 15 hours for CMA-ES, and about 1h for each of the other methods. We ran this experiment using only CPUs. When starting from scratch, NN achieves the highest test accuracy by a substantial margin when compared to the other model merging methods, as shown in Figure 2 (left). Interestingly, GA with crossover performs better early on (before step 12,000) than GA without crossover, however, it converges faster to an inferior solution. The early convergence happens because GA can't maintain a diverse population which is crucial for effective crossover operations. Crossover reduces population diversity, and without a strong counteracting force, it diminishes exploration. In contrast, NN leverages the crossover operation effectively, benefiting significantly from the diversity it manages to retain. GA is an extreme case where there is no competition, we observed that by progressively decreasing competition in NN, we progressively converge earlier to worse solutions (section 4.1.3). MAP-Elites clusters individuals by their accuracy on odd and even numbers. This means it will always keep individuals who perform poorly on those tasks because there is a slot reserved just for them. Even though those individuals add to the diversity of the population, this is clearly not the type of diversity that leads to strong solutions and it highlights the difficulty of hand-engineering useful diversity metrics.For models trained from scratch, the split-point and matchmaker have a minimal impact (ablations omitted for clarity). However, as seen in Figure 2 (right), the split-point becomes crucial when starting from pre-trained models, while the matchmaker significantly improves performance throughout the training process. GA has a low average test accuracy with large error bars as its performances is highly dependent on the quality of the first merges. Note that when starting from pre-trained models the mutation operator was not used (as explained in Section 4.1.1), and therefore, the performance is worse.This section focuses exclusively on the experiment where models were evolved from scratch, as the later sections will provide ample discussion on evolving models starting from pre-trained LLMs.Diversity. If either all models correctly or incorrectly classify a data point, the entropy is 0 (no diversity). In contrast, when the models are evenly split on a prediction, entropy reaches its maximum value of 1. The plot displays the average entropy across all data points. For NN we see a sharp initial rise in entropy followed by a gradual decline as low-performing models go extinct. In contrast, MAP-Elites continually increases diversity by retaining lower-performing models, but it fails to achieve a high coverage. The Genetic Algorithms, lacking a diversity preservation mechanism, reduce coverage early on and show a sharp drop in entropy as they converge prematurely on the best solutions.Overall, the graphs show that NN maintains an archive of models with complementary strengths that facilitate effective merging, while systematically discarding weaker models as training progresses.Competition. Figure 4 left, shows that smaller archives perform better in the beginning but converge faster to inferior solutions. This suggests that we should scale the archive size along the number of forward passes we want to make. Note that in our plot the computational cost does not increase with the archive size since the number of forward passes remains the same, however, the memory footprint does increase with larger populations. For very large models we can always store the archive on disk instead of keeping them all in the RAM.For a fixed population size P , we can adjust the intensity of competition by introducing a hyperparameter α ≥ 0, as described in the fitness function in eq. 5.fEQUATIONWhen α = 0, there is no competition because the total fitness available per data point becomes unlimited. When α = 1, the total fitness distributed among different individuals is limited to the capacity c j . For α > 1, the total fitness distributed decreases with increasing competition (z j ), this scenario can be thought of as individuals needing to "fight" for resources, spending some fitness points in the process. Figure 4 right, shows that smaller values of α (i.e. lower competition) have a similar effect to decreasing the population size: it performs better in the beginning but it converges faster to inferior solutions. In our LLM Merging experiments we don't use a mutation operator since random mutations don't work well on large models. Moreover, in these experiments we initialise the NN and GA archives with seed models, followed by a short warm-up period (50 iterations or less) where the seed models merge randomly amongst themselves and populate the archive. For these experiments, we used 4 H100 GPUs to run each method for around 24h.Models. We reproduce the experiment done in [1] , where one Japanese specialist LLM, shisa-gamma-7b-v1 [2] , and two Math specialists LLMs, WizardMath-7B-V1.1 [16] and Abel-7B-002 [4] are combined to create a hybrid model that can answer math problems in Japanese.We used exactly the same datasets and evaluation method from [1] , but for completeness, we provide the details here.Dataset. The test split, consists of the Japanese test set of the MGSM dataset [22] , which is a Japanese translation of a subset of the test set of the GSM8k dataset [5] consisting of 250 samples.The training set, consists of a translation of the remaining 1069 (out of 1319 examples) of the GSM8k test set that were not included in the MGSM Japanese test set.Baselines. The CMA-ES baseline implemented in [1] optimised the parameters for a TIES-Merging [29] with DARE [31] between the three seed models. CMA-ES used a population of 9 and it made 9,000 evaluations, while here we run NN and GA with an archive size of 20 but limited them to 4,000 evaluations.Evaluation. A correct answer must meet the following criteria: 1) the final numerical value must be correct, and 2) the reasoning text must be written in Japanese. To determine the language of the output, the library fasttext was used [17, 13] .Models. We combine a math specialist, WizardMath-7B-V1.0 [16] , with a specialist on agentic enviroments, AgentEvol-7B [28] , to achieve an agent that performs well on the math benchmark GSM8k [5] and on the web shopping benchmark WebShop [30] .Datasets. For the math task, we used the test split of GSM8k as our test split (1319 samples). For the training split, we used the first 1319 samples of GSM8k train dataset. In the web shopping task, we used the WebShop environment implemented in [28] . The test split consistent of the first 100 tasks, while the training split were the next 100 tasks. We allowed the agents to take up to 7 steps.Baselines. The CMA-ES optimised 32 mixing coefficients (one for each layer) for a SLERP merge between the two seed models. All methods were run for a 1000 evaluations on the training set. For the MAP-Elites we used two dimension to create a 4 by 4 grid: the accuracy on the math and on the web shopping training splits.Evaluation. In this experiment, all methods used 1000 evaluations on the training set. NN and GA used an archive size of 15. CMA-ES used a population size of 25.Tables 1 and 2 show that NN achieves the highest score. Both the matchmaker and the split-point techniques play a crucial role, however, the splitpoint seems to be slightly more important. Note that on Table 1 CMA-ES was run 2.25x longer than the other methods and used a more advanced merging technique (DARE-TIES), while on Table 2 all algorithms were run for the same amount of time and used the same merging method (SLERP). When combining the Math and Agentic skills, CMA-ES yielded a low score, likely due to suboptimal parameter partitioning, highlighting the need to include the merging boundaries in the optimization process. As shown in Figure 5 , the findings from the MNIST dataset generalize to LLM merging. The Natural Niches method maintains high training coverage, as seen on the left side of the figure. The entropy rises early on as the models explore diverse niches (right), followed by a gradual decrease as low-performing models are removed, and the strengths of the models are aggregated. In contrast, MAP-Elites focuses on maximizing entropy at the cost of training efficiency and coverage, as it retains low-performing models. GA quickly reduces both coverage and entropy as it greedily converges on its top solution, ultimately collapsing the entire archive onto a single solution, with entropy nearing zero.  
limitations & future work: 
 The feasibility of model merging strongly depends on the degree of similarity between models. As demonstrated in [31] , when fine-tuned models deviate significantly from their base models-often due to extensive, divergent training-merging becomes impractical. We hypothesize that models with divergent state representations are incompatible for merging. However, a standardized metric for model compatibility has yet to be established. Defining such a metric could allow it to be used as a form of regularization during preprocessing (e.g., fine-tuning), enabling better control over model compatibility and ensuring the success of merging.Moreover, we believe there is a strong evolutionary pressure for models that are co-evolving together to remain compatible for merging. Should one model, diverge and become incompatible with others, it would no longer produce viable offspring, halting its improvement and leading to its eventual extinction. Testing this hypothesis through further research would provide valuable insights into the dynamics of model co-evolution.Finally, incorporating a compatibility metric into the matchmaker heuristic could facilitate the coevolution of distinct species of models, defined as groups that can merge with one another but not with others. 
conclusion: 
 We've shown that model merging can significantly speed up the evolution of image classifiers when combined with a diversity-preservation technique. Moreover, this technique scales up to pre-trained LLMs. Our ablation studies show that both our proposed matchmaker heuristic and the use of crossover with split-point significantly improve the performance of the proposed method and have the potential to improve other evolutionary algorithms that use the crossover operation.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We describe that on our Setup sections and it can also be verified by inspecting our code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: On the MNIST experiment we did 10 independent runs for each method and report the error bars. On LLM merging, each run is very computationally expensive so we only did one run for each method and do not report error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In our Setup subsections we specify the compute resources used.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We present a generic algorithm to improve evolutionary algorithms that use model merging or crossover operations.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
seed models: 
 ModelOpen-source generative models have allowed the proliferation of thousands of specialised variants, fine-tuned by practitioners to solve their specific needs. In such an environment, where numerous diverse models are freely accessible, the ability to merge and aggregate that wealth of knowledge into a single model becomes important. This process, known as model merging [14] , has gained popularity as evidenced by the current prevalence of merged models on the Open LLM Leaderboard [12] .Model merging initially relied on manually adjusting coefficients to combine seed models, a process guided by intuition and requiring significant trial and error to optimize performance for specific tasks. Recently, this has been streamlined with evolutionary algorithms that automatically search for optimal coefficients [1] and increase the merging efficiency.However, one manual step persists: developers must group model parameters into fixed sets before merging, which restricts the search for potential combinations (see Figure 1 left). To overcome this, we propose an evolutionary algorithm with three key features:1. Evolving the Merging Boundaries. Existing methods partition the parameters of each seed model into fixed groups (e.g., layers) and search for optimal coefficients for merging these groups, which confines exploration to predefined boundaries. In contrast, our approach merges two models at a time, using arbitrary split points to divide parameters. Rather than working with fixed models, we maintain an evolving archive of models. As the number of generations increases, the method progressively explores a broader set of boundaries and coefficients (see Figure 1 right), allowing for increasingly complex combination if needed. This optional and gradual increase in complexity ensures a wider range of possibilities while maintaining computational tractability.2. Managing Diversity. Merging models only makes sense when they differ, making it essential to maintain diversity within the evolving population. However, the challenge of diversity preservation lies in determining which characteristics should be diverse. While many approaches require developers to manually define a diversity metric, we employ a nature-inspired method to automatically preserve diverse high-performing models that are particularly effective for merging.3. Matchmaker. We introduce a heuristic for pairing models based on their complementary strengths, enhancing both the efficiency and quality of our method. Mate selection remains an under-explored area in genetic algorithms, becoming increasingly important as computational costs of crossovers (merging) rise. This work encourages further research on this topic.We name our method Natural Niches and show that it performs well across two vastly different experiments: 1) evolving small classifiers from scratch and from pre-trained models (section 4.1), and 2) merging Large Language Models (LLMs) (section 4.2).2 Related Work 
 
 Appendix: Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Preprint. Under review. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Method for model merging with impressive results on LLMs
main_review: main_review: ### Strengths:

1.
The paper is well-written, easy to read and proposes a nature-inspired model merging performance with solid results on  LLMs.

2.
The idea of splitting the layer for model merging is intuitive and novel. 

3. Experiments in section 4.1.3 clearly shows that the proposed method (NN) helps maintain a more diverse set of model archive, showing the efficacy of the method.

4. The method is scalable, and strong results on different language tasks using LLMs make a strong case in favour of the proposed method. 


### Weaknesses and suggestions:
1. Sections 3.1 and 3.2 are currently difficult to understand, and notations can be further improved to make it easier for readers. 


2. Since the method is inspired by the genetic algorithm, a more detailed background would make it easier to follow the intuition behind the method. 

3. Eqn 3 is not clear --- over which variable the equation is maximized?

4. More detailed discussion is required to explain why pre-trained models perform worse in Fig 2.
title: title: Improving Model Merging with Natural Niches
main_review: main_review: # Paper Summary
The paper addresses the model merging problem: combining multiple (possibly pre-trained) neural network parameters into an optimal one that inherits capabilities from its predecessors. The authors introduce _Natural Niches_ (NN), an evolutionary algorithm that dynamically adjusts merging boundaries, preserves diversity among models, and uses a heuristic for mate selection to optimize model merging. The paper builds upon previous work on model merging and genetic algorithms; particularly, the work by [Akiba _et al_, 2024](https://arxiv.org/abs/2403.13187) required manual partitioning of model parameters and static merging boundaries. The authors extend this by automating the exploration of merging boundaries and preserving diversity more effectively using techniques inspired by evolutionary processes.

Regarding the presented method, the algorithm iteratively selects two parent models using a matchmaker heuristic, which prioritizes complementarity in their strengths and then performs a parameter-space crossover using spherical linear interpolation (SLERP) at a randomly chosen split-point. This merging process produces new offspring models that combine the characteristics of both parents. This merging process produces new offspring models that combine the characteristics of both parents. The algorithm preserves population diversity by introducing competition for resources during fitness evaluation, ensuring diverse solutions are maintained. It replaces the worst-performing models in the archive with new offspring if they outperform existing ones, and over multiple generations, the algorithm progressively explores more complex merging boundaries, leading to the discovery of high-performing, merged models.

## Contributions
- A novel evolutionary algorithm called Natural Niches that includes:
    - a diversity preservation mechanism that enhances the effectiveness of merging models, and
    - a heuristic mate selection strategy to optimize the combination of models.
- Experiments include:
    - evolving small MNIST classifiers from scratch;
    - evolving pre-trained large language models (LLMs), showing state-of-the-art performance;
    - comparison with baselines, such as CMA-ES and MAP-Elites

## Quality, Originality, Significance
The authors' line of work is very relevant within the model merging community and has interesting implications when it comes to understanding how different parameter spaces relate to each other. The experiments are exhaustive, that is, they try to address all of the claims posed by the authors in the introduction. While the proposed research question seems to be a follow-up from a very recent paper, the authors do a good job stating how the question they address contributes to the state-of-the-art.

## Clarity
The paper is well written, following the NeurIPS format and sticking to the workshop's restrictions. Relevant related work is presented about model merging and evolutionary algorithms. However, the method section seems a bit short, for the number of details that the reader needs to understand to fully grasp the proposed algorithm.

# Pros
- The results have similar conclusions when it comes to different scales of model archives; this gives a pathway to extend the results to multi-model merging.
- Almost all experiments come with ablations that help to assess how both matchmaking and splitting weights contribute to NN.
- The authors provide a path to follow-up this work and hint its applicability to other related areas of machine learning.

# Cons
- The method’s feasibility seems to heavily depend on the similarity of models being merged.
- Big open questions are left on how to thoroughly address the time and space complexity of the algorithm.
- Experiments leave open the question on how would these evolutionary approaches compare to fine-tuning merged models.

# Further Questions and Suggestions
- The methodology section presents the main parts of the NN algorithm as 3 subsections; yet it seems that it is up to the reader to put everything together to fully describe the algorithm from start to end. Hence, I would suggest to add another descriptive figure of the pipeline, a pseudo-code block, or even a small section that puts the contents of 3.1, 3.2, and 3.3 together.
    - For instance, it is not clear how does the algorithm go from 1 split-point to many boundaries (Fig. 1) by looking at Eqs. 2, 3, and 4.
- While SLERP interpolations seem to preserve certain geometric properties, a small question is left on how their usage compares to (Euclidean) linear interpolations -- which often appear within model merging literature. Therefore, adding an ablation would be nice to address this concern; although I would leave this to the Appendix section, for the purposes of this workshop. Additionally, it could help pointing out the preference of SLERPs withing the evolutionary merging literature, as well.
- Do you have an explanation/intuition on why would evolving models from scratch achieve better performance than starting from pre-trained models (Fig. 2)? I am assuming pre-trained models are optimal so I am curious to know how this result relies on the complexity of the task itself (MNIST).
- Could you clarify the statement on not using mutations on pre-trained models because _"random alterations don't scale well to larger models"_?
- Could you clarify if the _training coverage_ concept relates to ensembling accuracy (from all the archive)? In any case, do we expect it to be a hard upper-bound even for the best performing NN-generated archive?

All in all, it is a very interesting approach to model merging, with a nice set of experiments. I look forward to discussing more with the authors at the workshop :).
title: title: This paper designs a merging method using an evolutionary algorithm. Benefiting from the dynamic adjustment of merging boundaries and some evolutionary techniques, the experiments achieve SOTA performance.
main_review: main_review: ##### Summary of Strengths

- The writing is clear and easy to follow.
- The use of a evolutionary algorithm for model merging is both interesting and novel. 
- The effectiveness of the method is validated through experiments on Evolving MNIST classifiers and LLM merging.

##### Summary of Weaknesses

- As noted by the authors, the mutation operator is not particularly effective in LLM merging.
- While the paper includes ablation studies, it offers limited comparisons with other methods, only comparing with CMA-ES.
- Although faster than CMA-ES, the computational cost seems high compared to other merging methods.
- There is a minor citation error: "Ties-merging" should be cited as NeurIPS 2023.
title: title: Natural Niches: evolutionary algorithm for model merging
main_review: main_review: This paper proposes a method for model merging using an evolutionary algorithm. In particular, authors claim three features: dynamic adjustment of merging boundaries,  a diversity preservation mechanism, and a heuristic-based mate selection strategy. 
Weaknesses: 
1. (2) assumes that model A and B should have compatible parameters for concatenation. Also, merging w/o boundaries can be  computationally-demanding
2. (3) is just the weighted and normalized version of (1) which is an incremental contribution
3. Downsampled MNIST experiments show extremely slow evolution process (15 hrs)
4. Extensive usage of vague notions such as "diversity" instead of scientific ones such as entropy
5. Code for language experiments is not available