A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
Authors: Prateek Yadav
Colin Raffel
Mohammed Muqeeth
Lucas Caccia
Haokun Liu
Tianlong Chen
Mohit Bansal
Leshem Choshen
Alessandro Sordoni
Venue: Accepted by TMLR ()

Abstract: The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. Model MoErging methods aim to recycle expert models to create an aggregate system with improved performance or generalization. A key component of MoErging methods is the creation of a router that decides which expert model(s) to use for a particular input or application. The promise, effectiveness, and large design space of MoErging has spurred the development of many new methods over the past few years. This rapid pace of development has made it challenging to compare different MoErging methods, which are rarely compared to one another and are often validated in different experimental setups. To remedy such gaps, we present a comprehensive survey of MoErging methods that includes a novel taxonomy for cataloging key design choices and clarifying suitable applications for each method. Apart from surveying MoErging research, we inventory software tools and applications that make use of MoErging. We additionally discuss related fields of study such as model merging, multitask learning, and mixture-of-experts models. Taken as a whole, our survey provides a unified overview of existing MoErging methods and creates a solid foundation for future work in this burgeoning field.

Published in Transactions on Machine Learning Research (04/2025)
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
Prateek Yadav* 1
Colin R 2.3
Mohammed Muqeeth Lucas Caccia Haokun Liu Tianlong Chen'
Mohit Bansal' Leshem Choshen 4,5
Alessandro Sordoni
Equal Contribution
UNC-Chapel Hall" 2 University of Toronto, Vector Instilue
MIT, MIT-IBM Watson AI Lab, Microsoft Research," Mila
Correspondence Emoil: (praty@cs.unc.ddu, unc.edu, craffel@gmail.com com, absondon@mieroooft.comm com}
Reviewed on OpenReview: https: // openreview. net/ porm? id=
Abstract
The availability of performant pro-trained models has led to proliferation of fine-tuned expert models that are specialized to particular domain task. Model MoErging meth- ods aim to recycle expert models to create an aggregate system with improved performance or generalization. A key component of MoErging methods is the creation of a router that decides which expert model(s) to use for a particular input or application. The promise, effectiveness, and large design space of MoErging has spurred the development of many new methods over the past few years. This rapid pace of development has made it challenging to compare different MoErging methods, which rarely compared to one another and are often validated in different experimental setups. To remedy such gaps, present comprchensive survey of MoErging methods that includes a novel taxonomy for cataloging key design choices and clarifying suitable applications for each method. Apart from sur- veying MoErging research, we inventory software tools and applications that make use of MoErging. We additionally discuss related fields of study such as model merging, multitask learning, and mixturo-of-experts models. Taken as whole, our survey provides a unified overview of existing MoErging methods and creates solid foundation for future work in this burgeoning field.
1 Introduction
The development of large-scale pre-trained models increasingly aims to create general-purpose AI systems that can perform any task without requiring task-specific training. Improvements in these models are often driven by scale, i.. training a larger model on a larger dataset (Hestness et al., 2017; Kaplan et al., 2020). However, even with increased scale these models are not yet truly "general purpose" and often struggle with certain tasks and/or domains (McCoy et al., 2023; Ling al., 2023; Kandpal et al., 2023a). Unfortunately, pre-training new model in hopes of improving capabilities can be incredibly compule-intensive (Li et al., 2023; Workshop et al., 2022) and is therefore impossible for most of the research and practitioner community, In addition to high computational costs, it can be difficult to localize which parameters of the model might be more useful for a specific use case and adaptively improve performance or reduce computation based on that information (Pfeiffer et al., 2023).
Fortunately, often possible make targeted improvements to a pre-trained model via fino-tuning (i.c. fur- ther training on specialized dataset). In addition, parameter-efficient fine-tuning (PEFT) techniques (Ding
We maintaining github repo with list of related papers and their taxonomy https:// http:://gtttubb.oo///uuusss axesone-noerging.

---

Published in Transactions on Machine Learning Research (04/2025)
et al., 2022; He et al., 2021; Mangrulkar et al., 2022) further increase fine-tuning efficiency and decrease the cost of serving such specialized models. PEFT introduces small components like Low-Rank Adapters (Hu et al., 2022) or (LA)a vectors (Liu et al., 2022) that surgically modify the original model while adding a negligible amount of parameters. Due to their compact size, these specialized PEFT modules can be cheaply shared, facilitating the dissemination of an ever-growing number of adapters across various platforms. The effectiveness of fine-tuning, combined with the recent release of performant open-weight pre-trained models like Llama (Touvron et al., 2023) or Stable Diffussin (Podell et al., 2023), has fostered the creation and release of multitude of fine-tuned expert models.
This proliferation of expert models has led to the development of methods for re-using such experts to improve performance or generalization. Central to these approaches are routing mechanisms that adaptively select relevant experts for a particular task or query. These methods have been referred to as "MoErging" since they frequently share methodology and ideas with mixture-of-experts (MoE) models (Shazeer et al., 2017; Fedus et al., 2022; Du et al., 2022) and model merging (Matena & Raffel, 2022; Ilharco et al., 2022; Yadav el al., 2024). However, MoErging methods are distinct from MoE approaches that jointly train all the experts from scratch (Gupta et al., 2022). Instead, experts are provided by a distributed and decentralized community of contributors and are not trained by a centralized body. In addition, unlike merging methods that typically produce a static combination of models, MoErging methods adaptively combine models to improve performance on a per-query or per-task basis.
Model MoErging has several attractive properties compared to typical paradigms for monolothic model development. First, reusing and routing among gudepeddently-rrinee expert models can enable decentralized model development, alleviating the need for extensive centralized data and compute resources, while still ingesting large amount of data and computation from different collaborators. Second, the modular use of expert models facilitates expansion of capabilities by adding experts or making localized updates by changing individual experts. Such changes are more "transparent" than standard model training by virtue of the fact that experts are often specialized specific functionalities. Finally, these systems can idcally generalize compositionally by identifying and "remixing" fine-grained skills from expert models in various ways, thereby extending their abilities beyond the experts' initially intended scope (Pfeiffer et al., 2023).
The promisc of decentralized development of modular AI systems by recycling experts has led to explo- sion of recent. work on MoErging. The rapid pace of development in this budding subfield has often led to papers being unaware of and/or omitting comparison to one another. In addition, differences in assumptions, experimental setups, and problem settings across papers can further conflate comparison. Hence, the goal of this survey is to review the many recent publications and projects that can be considered a form of Mo- Frging. To facilitate comparison and clarify assumptions, we categorize existing works in a novel taxonomy encompassing possible design choices at three distinct levels: i) the erperts that are independently trained and shared by contributors in decentralized fashion; ii) the routing strategy that determines how to select and aggregate experts; and iii) the downstream application targeted by end users. In addition, we discuss related lines of inquiry, tools that support this approach to model development, and areas for future work.
2 A Taxonomy for MoErging Methods
Broadly, our survey focuses on "MoErging", a new paradigm for decentralized model development that aims to recycle expert models trained asynchronously by distributed contributors. We can organize the stages and components of MoErging methods into three categories: (1) experts, (2) routing, and (3) application. The experts are the specialized models that are trained and shared by individual contributors. Importantly, experts are trained independently, i.c. the expert contributors do not have access to one another's compute or data. Once expert models have been shared, MoErging methods perform routing, which aims to select and aggregate the contributor-providd
expert models in order to improve performance or generalization. To process given query or adapt to a larget dataset, routing can operate in various ways, for example: (I) adaptively select a single expert model, (2) route different examples or processing steps to different experts, (3) learn a layer to extract relevant information from all experts, and/or (4) combine the expert models in an adaptive way. Some MoErging methods assume that the expert contributors share not only their expert
28ee e.g. https: http:///wwagggaaeeccc///////e//s 11m- http://ceeeopppemnneeeeeeeeeerer 11m leaderboard
2

---

Published in Transactions on Machine Learning Research (04/2025)
models but also their training datasets so that they can be used to design create the routing strategy. Finally, the aggregate system is applied to some particular use case, e-g. processing a query or solving a target task. Different MoErging methods are designed for different use-cases, including zero- or few-shot adaptation to in-distribution or out-of-distribution task (i.c. a task for which there is or is not a trained expert model in the system).
While MoErging methods share an overarching goal, the large design space and range of possible use cases have led to wide diversity of design choices across different methods. At the same time, scemingly disparate methods sometimes only differ in single assumption target application. In addition, most MoErging methods were developed over the past year so. This contemporancousnes has resulted in studies rarely discussing or citing one another, further conflating comparison. To address this state of affairs, we introduce a taxonomy of MoErging methods that precisely enumerates the various design choices and use cases of MoErging. Our goal in designing this taxonomy is not only to elucidate the previously undocumented connections and commonalitics among methods, but also to provide a framework to situate future work on MoErging. Our laxonomy is shown in Figure 5, with descriptions of each design consideration below.
2.1 Expert model design choices
MoBrging involves recyeling specialized expert models. Contributors of the expert models do their training independently, i.c. without access to one another's 's data or compute, and subsequently share their models. Design choices for the expert models include:
2.1.1 Expert Training
While contributors must. train and share a model for it to be used as part of a MoErging system, a given MoErging method may further stipulate that the expert models are trained in a specific way. For example, PHATGOOSE (Muqeeth et al., 2024a) requires that expert model training includes an additional stage where gates are trained that are later used for routing. If a MoErging method stipulates a specific expert training procedure, we label it Custom; otherwise, we label it as Standard. We note that many MoErging methods require access to statistics of each expert training dataset (c.g. each expert training set's average activation at some particular layer). We consider this a modification because il would not otherwise be done as part of standard expert training.
The choice between Standard and Custom expert training procedures involves trade-offs between expert training complexity and better integration within the MoErging system. Custom training procedures, while potentially more complex to implement, can allow for better integration of experts with the routing mochanism, potentially leading to better model (Muqeeth et al., 2024a; Wu et al., 2023; Diao et al., 2023). In contrast, Standard training leverages existing fine-tuning methods, and allows to reuse trained experts available online (Huang et al., 2024b; Wang et al., 2024). However, it may necessitate more sophisticated routing and aggregation strategies to effectively utilize these off-the-shelf expert models.
Considerations:
2.1.2 Expert Data
A major motivation of the field of MoErging is to recycle the huge number of fine-tuned models being shared on model lubs. Such models are typically shared without their associated training data. However, certain MoErging methods assume access to expert training data, e.g. for learning the routing procedure. When expert data is shared, is no longer a requirement that the experts must be trained independently. Furthermore, it would be possible to e.g. perform multitask training on all expert datasets simultaneously carry out modified expert training procedure. In the scenario where expert data needs to be shared, the benefits of MoErging methods are: (1) recycling of the compute required to train the expert models, and (2) decentralized training. In addition, apart from the reality that training data is often not shared alongside fine-tuned expert models, contributors may prefer to keep their training data private. We therefore categorize whether each method requires that expert training data is Shared or can remain Private.
Considerations: The choice between Shared or Private Expert Data fundamentally shapes MoErging methods. Shared data allows leveraging data embeddings for similarity-based -ased routing (Chronopoulou et al.,
3

---

Published in Transactions on Machine Learning Research (04/2025)
Routing Input Granularity
Step / Token Level
Routing
Task
Example Level Routing
Culput
Routing
Output
Oulput
Expert
Expert
Expert 3
Expert
Expert
Expert
Expert
Expert
Expert
Rouae
Rtazer
Router
Activations
Activations
Expert
Expert
Expert 3
Export
Expert 2
Expert
Expert
Expert
Expert 3
Router
Example
Task
Example
Example Tokens
Figure Different levels of granularity for routing decisions in MoErging methods. Left: Task Level Routing selects a single expert for all examples belonging Routing chooses an expert independently for each imput example. Right: Step/Tooken Level Routing makes routing decision (i.e., selects expert) at each processing step or for each generated token. The purple elements indicate the imput used by the router to make its decisions.
specific task. Middle: Example Level
2023; Zhao et al.; 2024b; Cheng et al., 2024; Jang et al., 2023), training a routing classifier (Lu et al., 2024a; Ong et al., 2025), and training module level routers on expert dataset distributions (Xu et al., 2024; Sukhbaatar et al., 2024), potentially leading to more informed expert selection (Zhao et al., 2024b; Bansal et al., 2024; Sukhbaatar et al., 2024; Zhao et al., 2024a). However, this necessitates data sharing, raising privacy and legal concerns with proprietary datasets. Private data methods are more broadly applicable and univacy-preserving relying on open data sources (Shnitzer et al., 2024; Lu et al., 2024a; Ong et al., 2025) and expert models for routing and aggregation (Muqeeth et al., 2024a; Ostapenko et al., 2024). The core trade-off is between the routing utility gained from expert data and the practical limitations of data availability and privacy.
2.2 Routing design choices
In MoErging, expert models collected to create aggregate system to improve performance general- ization. A key step in this process is to create a "router" that can adaptively choose which model(s) should be used for particular query or dataset. The creation of the aggregate MoErging system involves a large range of design choices, including:
2.2.1 Routing Dataset
To learn to route or select among expert models, MoErging methods often require a training dataset that we refer to as the "routing" datasct. Some MoErging methods make use of the Expert's training datascts for the routing dataset, while others assume access to a Target-lask dataset or General dataset that covers wide range of tasks. In addition, some MoErging methods do not explicitly train router and therefore use None.
The choice of Routing Dataset is critical as it dictates the data used to train the routing mechanism and significantly impacts the router's generalization capabilities and data requirements. Methods utilizing Expert datasots assume that the expert training data contains sufficient signal to learn effective routing for new inputs (Bansal et al., 2024; Zhao et al., 2024b; Sukhbaatar et al., 2024). This approach minimizes the need for additional data collection but may limit the router's ability to generalize beyond the
Considerations:
4

---

Published in Transactions on Machine Learning Research (04/2025)
Routing Depth Granularity
Model Level
Module Level Routing
Routing
Expert 1
Expert 2
Expert 3
Expert
Expert 2
Expert 3
Module Router
Copy
Expert 1
Expert 2
Expert 3
Expert
Expert 2
Expert 3
Model Router
Module Rouler
Input
Input
Different levels of granularity for routing depth in MoErging methods. Left: Model Level Routing applics single routing decision to select experts that are then used across all applicable modules or layers of the model. Right: Module Level Routing makes independent routing decisions al each layer or module where experts are integrated, allowing for different experts to be active at different depths. The turquoise boxes represent the routers operating at either the model level or the individual module level.
Figure 2:
distribution of expert tasks. Employing a Target-task dataset allows for optimizing the router specifically for the intended application, potentially leading to superior in-distribution performance (Tang et al., 2024c; Wang et al., 2024; Lin et al., 2024). However, this necessitates the availability of a labeled target dataset, which may not be feasible in zero-shot or few-shot settings, and may lead to overfitting to the target task, hindering broader generalization. Using a General dataset aims to train a more broadly applicable router, capable of gencralizing across a wider range of tasks (Shnitzer et al., 2024; Lu et al., 2024a). These approaches rely on the assumption that the general dataset adequately represents the diversity of potential target tasks and requires access to such diverse and representative dataset. Methods that use None for the routing dataset rely heuristic routing mechanisms (Maxine, 2023; Durbin, 2024) or information stored from modified expert training (Jang et al., 2023; Chronopoulou et al., 2023; Mugeeth et al., 2024a), circumventing the need for router training data altogether. While data-efficient, the effectiveness of these methods hinges on the quality of the heuristics and their applicability to diverse tasks and expert models. The selection of the routing dataset thus involves trado-off between data availability, task-spociffcity, and the desired generalization scope of the MoErging system.
2.2.2 Routing Input Granularity
Different MoErging methods make routing decisions at different levels of granularity. At the finest level, routing can be done per-Step (e.g. choosing different expert model for each of a language model's generated tokens). In addition, routing can be performed once for each Example or query, or single expert model can be chosen for all examples from some particular Task.
Routing Input Gramularity balances adaptability and cost, influencing methods' effective- ness. Higher frequency per-Step routing maximizes muanced input adaptation but bocomes computationally expensive and potentially unstable, with the risk of early routing errors propagating to later routing de- cisions (Mugeeth et al., 2024a; Shen al., 2024). Example-based routing offers a balance (Zhao et al.,
Considerations:
5

---

Published in Transactions on Machine Learning Research (04/2025)
Expert Selection
Dense Selection
Sparse Selection
top-1
Expert
Expert 2
Expert
Expert
Expert 2
Expert 3
Router
Routor
Input
Input
Different strategies for expert selection in MoFrging methods. Left: Dense Selection utilizes the output of all available experts, often through weighted combination. Right: Sparse Selection activates only subset of the experts (e.g,, the top-k most relevant ones) based on the router's decision. The router's output distribution indicates the selection strategy, with "All" implying dense selection and 'top-1' implying sparse selection of the single most relevant expert.
Figure 3:
2024b; Tang et al., 2024c). Lower frequency, Task- routing prioritizes efficiency but assumes consistent expert needs across lask, potentially limiting adaptability for diverse task inputs (Huang et al., 2024b; Wu et al., 2023; Chronopoulou et al., 2023). Selecting the routing frequency involves trading off adaptability, computational efficiency, and the consistency of expert needs within and across inputs.
2.2.3 Routing Depth Granularity
Parameter-effiiiet fine-tuning methods like LoRA (Hu et al., 2022) or (IA)3 (Liu et al., 2022) insert trainable modules at different layers throughout a model. Some MoErging methods therefore make per-Module routing decisions (i.e. with different routing decision at each layer where modules have been inserted, as in mixtuur--f-expprts models (Shazeer et al., 2017)), while others make a single routing decision for the entire Model.
Per-Module routing, operating at the level of individual modules or layers, allows for fine- grained control over expert utilization within different parts of the model (Diao et al., 2023; Ostapenko et al., 2024; Tang et al., 2024b). For instance, different experts might be selected for attention layers versus feed- forward layers. However, per-module routing significantly increases the complexity of the routing mechanism by making decisions at multiple depths, potentially leading unstable routing as early routing errors can affect later routing decisions. Model-level routing, in contrast, applies a single routing decision the entire expert model (Lu et al., 2024b; Ong et al., 2025). This simplifies the routing process, as only decision is made per inpul or example, and reduces routing overhead. However, it assumes that single expert (or combination) is globally optimal for all the layers of the model, potentially limiting adaptability. The choice between per-module and model-level routing thus involves a trade-off between routing granularity, architectural complexity, and the assumption of a few experts' relevance across model depths.
Considerations:
2.2.4 Expert Selection
When routing among experts, some MoErging methods make Sparse selection (i.c. choosing only subset of the experts) while others perform Dense routing (i.e. making use of all experts at once).
Expert selection strategy governs the number of experts used, impacting computational efficiency and knowledge aggregation. Sparse selection (subset of experts) enhances efficiency by activating only the most relevant experts, crucial with large expert pools to avoid processing all experts (Muqeeth al., 2024a; Xu et al., 2024). However, sparse selection requires precise routing to identify the relevant experts,
Considerations:

---

Published in Transactions on Machine Learning Research (04/2025)
Expert Aggregation
Output Aggregation
Parameter Aggregation
Expert
Merged Expert
Expert
Expert
Expert
Expert 2
Expert 3
Rouler
Router
Input
Input
Figure 4: Different methods for expert aggregation in MoErging methods. Left: Output Aggregation combines the outputs of multiple selected experts, often using weights determined by the router. Right: Parameter Aggregation merges the parameters of multiple selected experts into a single, aggregated expert model before processing the input.
any inaccuracy in selection lead to information loss and reduced performance. Dense routing (all experts) maximizes knowledge aggregation potential and is advantageous when experts have overlapping, but complementary, skills for richer representations (Cheng et al., 2024; Zhao et al., 2024a). Dense routing is computationally expensive, particularly with many experts, and may degrade performance irrelevant experts introduce noise and redundancy. Choosing between sparse and dense expert selection thus a trade-off between computational efficiency, routing accuracy, and desired knowledge aggregation.
2.2.5 Expert Aggregation
If a MoErging method selects more than one expert, must aggregale the experts or their oulputs some way. The Expert Aggregation method defines how the information from selected experts combined, inftuencing the nature of knowledge integration and computational complexity. Aggregation methods include mixing the Output of experts, combining the expert's 's Parameter values before processing inputs, or None for methods that perform no aggregation (e.g. because they select single expert).
Expert Aggregation impacts knowledge mixing and computational complexity. Output aggregation flexibly combines expert predictioos/reppesentttooo by mixing their outputs (Lin et al., 2024; Bansal et al., 2024). Dynamic output combination is basod on input relevance and can use techniques like weighted averaging, attention, ensembling. Output aggregation performs post-expert processing allowing for independent expert operation. Parameter aggregation integrates knowledge the parameter level by combining multiple expert parameters into single aggregated expert (Lu et al., 2024b; Zhao et al., 2024a) by using techniques like parameter averaging (Wortsman et al., 2022), merging (Yadav et al., 2024), or task vector arithmetic (Ilharco et al., 2022). Parameler aggregation yields a more compact model than output aggregation.
Considerations:
2.3 Application design choices
Once the expert models have been recycled into an aggregate system, users can then apply the system their tasks or queries of interest. Different MoErging methods produce systems that support different usage patterns and incur different requirements applications. Relevant design choices include:
7

---

Published in Transactions on Machine Learning Research (04/2025)
2.3.1 Generalization
MoErging can aim to produce systems that improve performance on In-Distribution tasks (i.e. the tasks that the experts were trained on) or enable generalization to Out-of-Distribution tasks (i.e. those tasks for which there is no corresponding expert). However, many systems are applicable to both settings.
Considerations: The Generalization goal, whether In-Distribution (ID) or Out-of-Distribution (OOD), shapes design and evaluation criteria. For ID generalization, the primary objective is to enhance performance on tasks domains that are similar to within the distribution of the experts (Cheng et al., 2024; Sukhbaatar et al., 2024; Shen et al., 2024). Effective methods for ID generalization should excel at retrieving and aggregating relevant experts, potentially through fine-tuning routing mechanisms on ID data or leveraging expert specialization within the known distribution. However, ID-optimized methods may de- grade OOD performance. contrast, OOD generalization aims to enable performance on novel tasks (Wang et al., 2024; Muqeeth et al., 2024a; Mohammadshahi et al., 2024). Achieving OOD requires strategies that extrapolate expert knowledge, potentially via meta-learning, task similarity routing, or compositional gen- eralization. OOD-focused methods prioritize robustness and adaptability over ID peak performance, and may require sophisticated routing and training. Many systems demonstrate applicability to both settings, highlighting potential for both specialized and general knowledge integration. The generalization goal guides design and evaluation, inffuencing routing data, training, and metrics.
2.3.2 User Dataset
MoErging methods may require a training datasct in order to be applicd to a target task, which may be a Few-Shot dataset with a small number labeled examples or a Full dataset with many labeled examples. Other methods require no target-task training dataset (i.e. they can be applied Zero-Shot). We make slight misnomer and also refer to MoErging methods where an unlabeled target-task training dataset is required as "zero-shot".
Considerations: User Dataset availability dictates MoErging application and adaptation and impacts its usability and data needs. Zero-shot methods are appealing for their simplicity and speed, especially when labeled data is limited and costly. These methods inherently rely on the generalization capabilities of the pre-trained expert models and routing mechanism. Hence, without task-specific adjustments, their effectiveness can be limited in specialized downstream tasks. Few-Shot methods adapt the model using few labeled examples, often via prompt tuning or fine-tuning, balancing zero-shot usability and full fine-tuning performance with data-efficient task-specific optimization. The effectiveness of a few-shot method depends on the informativenoss of examples and the adaptation technique. Full dataset methods use fully labeled data for end-to-end fine-tuning, potentially maximizing performance given sufficient data, but increase data and computational costs, reducing the data efficiency in some contexts. The choice of dataset, thus trades off data needs, usability, performance, and applicability across data-scarce or data-rich scenarios.
3 Broad Categorization of MoErging Methods
Given the taxonomy Section 2, we provide some high-level discussion about the commonalities and differ- ences among MoErging methods, First, we note that many MoErging methods are remarkably similar, and that can broadly group most of them into four categories based on how routing is learned: embedding- based hods, classifier-based methods, task-specific routing, router-free methods. Next, we explore these method categories, their strengths, weaknesses, data needs, and computational costs. These are crucial for practitioners seeking to apply MoErging effectively.
3.1 Embedding-Band -Base Routing Methods
Embedding-be based routing methods such as AdapterSoup (Chronopoulou et al., 2023), Retrieval of Experts (Jang et al., 2023), Token-Level Adaptation (Belofsky, 2023), LoraRctricver (Zhao et al., 2024b), Mo MoAL (Maxine, 2023). the embedding baaseddin--baed approach of Airoboros (Durbin, 2024), and Dynamic Adapter Merging (Cheng al., 2024) rely on comparing the input query embeddings with embedding that represents
8

---

Published in Transactions on Machine Learning Research (04/2025)
Expert Traiuing (2.1.1)
Standard
1,7,8,9,10, 12,
18, 14, 15, 17, 18, 26, 27,27,22222 27,
Custom
11,
Expert Design (2.1)
Expert Duta Privacy (2.1.2)
Private
12
23,
Shared
12, 28,
None
11, 22,
Routing Dataset (2.2.1)
Target
14, 23, 27,28
Expert
12, 25,1,
General
9, 14,22
Task
2,3,4, 7,9,4,7,9,122 12,
Routing Iaput Granularity (2.2.2)
Example
5,6,8, 10, 10, 27,2
Step
1,11, 16, 18, 20, 20, 24,
Routing Design (2.2)
Routing Depth Granularity (2.2.3)
Module
1,5, 13, 15, 16, 18, 19, 21
26
Model
2,3,4,6,7,8,9
10,
20,
27,27,
Expert Selection (2.2.4)
Dense
1, 6, 11,
Sparse
2,3, 4,3,4,
13,
24,
25,
Expert Aggregation (2.2.5)
Outpat
1,5, 13, 16,
Parameter
3,4,6, 7, 11, 18, 15, 17, 26,28,
ID
1,5,8, 15,
Generalization (2.3.1)
Application Design (2.3)
OOD
2,3,
10, 10, 14344667.8888810112111441444311,
Zero-Shot
2,4,6,8,9 10, 11, 16,,1,22,,,,11,,,2,,,,,,,,
User Dataset (2.3.2)
Few-Shot
19,
Full
1,5, 17, 19, 26,
Taxonomy of model MoErging design choices. References in the leaf nodes link to sections for specific papers that make some particular design choice. We omit references to methods for which given choice is not applicable.
Figure 5:
9

---

Published in Transactions on Machine Learning Research (04/2025)
the expertise of each available expert model. Often, these expert embeddings are derived from the experts training data, representative samples, or summary statistics thereof. Routing is then typically performed by selecting the expert(s) whose embeddings exhibit the highest similarity (e.g., cosine similarity) to the input embedding.
A key strength of embedding-based routing methods is their intuitiveness and minimal need for additional routing-specifie training data. This makes them effective in zero-shot and few-shot settings and enables generalization to new tasks, provided the embedding space is capable of capturing the relevant task similar- ities. Moreover, efficient inference can be achieved through pre-computed expert embeddings and optimized similarity search techniques like FAISS (Douze et al., 2024). Furthermore, it is not necessary for all experts to follow the same architecture. However, their performance is critically dependent on the quality of the embedding space. A primary weakness is the reliance on pre-trained embedding models to capture maanced task relationships. Suboptimal embeddings can result in poor routing, especially for semantically subtle, specialized, or out-of-distribution tasks where simple embedding similarity may not accurately reflect expert applicability.
Despite these limitations, embedding-basse routers are well-suited for tasks where semantic similarity is a reliable performance indicator. They are most effective when expert training data distributions are relatively distinct, facilitating clearer separation in the embedding space. Conversely, they may falter in highly special- ized domains where generic embeddings are insufficient, and where semantic similarity alone is inadequate for expert selection. They are also beneficial when expert models have different architectures.
3.2 Classifier- BBaee Routing Methods
Classifier-based routing offers an alternative to embedding- methods by utilizing dedicated, trained classifier to predict the optimal expert for each input. Methods like Zooter (Lu et al., 2024a), Branch- Train-Mix (Sukhbaatar et al., 2024), Routing with Benchmark Datasets (Shnitzer et al., 2024). Routoo (Mohammadshahhi et al., 2024), and RouteLLM (Ong el al., 2025) exemplify classifier-based routing. Instead of relying solely on embeddings, these methods train a dedicated classifier to predict the optimal expert(s) for given input for routing. This classifier is typically trained on labeled data, which could be the original expert training data (if shared), general-purpon dataset covering a range of tasks, or a dataset specific to the target task. Downstream performance metrics can also serve as training signals, guiding the classifier to optimize expert selection based on desired outcomes. Effective training and performance hinge the availability of sufficient, high-quality labeled data.
Classifier-based routing offers notable advantages, primarily in its capacity to learn and implement complex routing functions. Unlike embodding-bant based methods constrained by similarity mcasures, classifier-based ap- proaches can capture nuanced and intricate relationships between input features and expert specializations. This enhanced complexity translates to potentially superior routing accuracy and a greater ability to bal- ance performance and cost considerations within expert systems. The inherent flexibility of classifier-based routers further renders them more adaptable to wider spectrum of tasks and domains, accommodating di- verse expert architectures and application contexts. Classifier-based routing, while beneficial, faces inherent limitations. The foremost constraint is its data-dependent nature, which relies on labeled training data a practical challenge in novel or unseen tasks where such data may be scarce or absent, thereby restricting generalizat ion in zero- or few-shot learning scenarios. Furthermore, the dedicated classifier component in- creases computational requirements during both training and inference, cnlarges the model size, and adds to the complexity of training. This added complexity is further affected by the routers architectural intricacy and the scale of the training dataset.
Classifier-based routing is best suited for data-rich scenarios with complex routing needs such as those explored in Routing with Benchmark Datasets (Shnitzer et al., 2024). They are ideal when nuanced expert selection beyond simple similarity is required, and can accommodate diverse expert designs across various applications given sufficient data. Practitioners are advisod to adopt classifier- routing when high precision is paramount and the target domain offers ample data, though its inflexibility in zero-shot or few-shot settings limits its broader applicability.
10

---

Published in Transactions on Machine Learning Research (04/2025)
3.3 Task- Sak--Specfiic Routing Methods
Task-specific routing methods- like LoraHub (Huang et al., 2024a), LoRA-Flow (Wang et al., 2024), Adapter- Fusion (Pfeiffer et al., 2021); ); æ--Tuning (Wu et al., 2023); Co-LLM (Shen et al., 2024); Weight-Ensemblligg MoE (Tang et al., 2024c), MoLE (Wu et al., 2024), MetcoRA (Xu et al., 2024), PEMT (Lin et al., 2024), MixDA (Diao et al., 2023), and Twin-Merging (Lu et al., 2024b) are characterized by their focused approach to expert selection, learning routing strategies tailored for individual target tasks. Training is intrinsically linked to data from the target task or related datasets, often demonstrating data efficiency and requiring only limited task-specific examples to achieve good performance for the target task.
This focused training allows the router to become highly specialized for the nuances of a particular task, making it ideal for production environments where performance on a task is prioritized over adaptability. Often, only a relatively small amount of target-task data is needed to fine-tune the routing. Nevertheless, this specificity comes at the cost of generalizatio;; without retraining, these methods cannot, effectively handle new or diverse tasks, and their reliance on task-specific data may exclude them from applications where such data is unavailable. Computationally, training the router for cach task can be time-consuming, particularly with large datasets or complex architectures, though the inference phase remains efficient once trained.
Task-specific routing methods are most appropriately applied when the primary goal is to maximize per- formance specifie, clearly delincated task, and when some task-specific data is available. They offer robust solution for targeted optimization in such contexts. However, practitioners should avoid these methods when broad generalization, adaptability to new tasks without retraining, or zero-shot capabilities are required. Their inherent trade-off necessitates a careful evaluation of application demands, prioritizing tpsk-specific performance at the expense of wider applicability and flexibility.
3.4 Router-Free Approaches
Router-free methods offer an alternative to explicitly trained routers by directly determining expert choice through pre-existing mechanisms. Methods like Arrow (Ostapenko et al., 2024), PHATGOOSE (Muqoeth et al., 2024a), and based routing (Airoboros, LlamaIndex) which leverage expert properties, LLM capabilities, or precomputed information for routing. Arrow (Ostapenko et al., 2024) uses a zero-shot linear router with LoRA prototypes, while PHATGOOSE (Mugeeth et al., 2024a) employs sigmoid gates for token importance routing. Airoboros (Durbin, 2024) and LlamaIndex (Liu, 2024) depend on the inherent knowledge and reasoning abilities of pre-trained LLMs, and they only need descriptions of available expert models to perform routing. Critically, these methods bypass dedicated post-hoe router training by using precomputed gates or by using pro-trained LLM knowledge. This reliance on cxisting information significantly roduces data and computational demands compared to traditional router-based approaches.
The main advantage is that this design enables zero-shot functionality, allowing immediate deployment in settings where labeled data is absent, as demonstrated by Arrow (Ostapenko et al., 2024) and PHAT- GOOSEs (Muqeeth et al., 2024a) reasonable performance in initial testing phases. The absence of router training reduces computational overhead, with inference typically being fast despite potential precomputa- tion costs for embeddings or LLM queries. These methods involve no training and do not require additional data. However, router-free methods may necessitate modifications to expert model training, increasing de- velopment complexity. Zero-shot routing can be inherently unstable, potentially leading to inconsistent performance. Furthermore, accurate expert retricval for specific tasks, even when relevant experts cxist, can be challenging. These factors highlight the need for careful implementation and thorough evaluation to mitigate potential drawbacks in router-free expert selection.
Router-free methods are particularly well-suited to applications featuring highly diverse querics and de- manding zero-shot generalization to novel tasks. However, practitioners must carefully consider the inherent trade-off: while offering broad applicability through zero-shot capabilities, these methods may not achieve the task-spocific performance attainable with dedicated routing schomes optimized for target tasks. Therefore, thorough evaluation remains crucial fully characterize the nuanced performance of router-free approaches across diverse applications and to inform the future evolution of expert selection strategies.
11

---

Published in Transactions on Machine Learning Research (04/2025)
3.5 Summary and Takeaways
Given these broad categorizations, note that the differences between methods within a particular category frequently come down to the routing and expert gramularity, the way experts are selected and aggregated, and the evaluation performed. We consider these differences to be relatively superficial compared to the way the router is built, which has significant implicat ions in lerms of what data is required and what settings a given method is applicable to. We therefore argue that methods within each category should, in most cases, compare to one another. However, such comparisons are rare in past work; for example, within the first routers, only LoraRetriever compares to AdapterSoup. Comparison between
category of embedding- groups would not make sense in some cases, though it could in principle be possible by swapping out the routing dataset of given method (e.g., using general dataset or expert datasets to learn the router in the third "task-specific router" category to enable comparison to methods in the other categories).
Apart from comparison to other methods, we reemphasize that assumptions about data access can make other often omitted baselines important. For example, if all expert datasets are assumed to be simultaneously available for the purpose of, e.g. training the router, then multitask training of the base model on the expert datasets should be considered as a baseline. More broadly, we consider data access to be a primary considcration in terms of which methods are applicable and/or realistic in different settings. For example, methods that require labeled larget-lask dataset are, by definition, not applicable to improving zero-shot generalization, and methods that require that the expert training datasets are shared alongside adapters are not applicable to reusing the huge number of publicly shared adapters (which seldom have their training dataset released alongside parameter values). Next, we discuss many MoErging methods in detail and categorize them according to our taxonomy.
4 A Survey of MoErging Methods
Having established our taxonomy, now provide detailed survey of a few dozen recent papers that propose and study MoErging methods. Precisely delineating what is and is not a MoErging method is challenging because many past methods share the same basic motivation but differ in their application and framing, However, we believe that the papers we cover in this section provide a reasonably comprehensive overview of MoErging and doEEgingg-adjacent methods. Notably, most of the papers we discuss here only cite a small fraction of the other papers, suggesting that there is a general lack of awareness about relevant papers. Our survey aims to address this gap in knowledge.
For each method described in this section, include an "infobox" cataloging the design choices made by each method according to our taxonomy. These infoboxes provide point of reference to quickly understand each method and how it relates to others. However, there are cases where a given paper does not clcanly map onto our laxonomy. In such cases, we may denote that a paper considers Multiple options for a given design choice or that some design choice is N/A (not applicable).
4.1 AdapterFusion
Expert Training: Standard
Expert Data: Private
Routing Dataset: Target
Input Granularity: Step
Depth Granularity: Module
Expert Selection: Dense
Expert Aggregation: Output
Generalization: In-Distribution
User Dataset: Full
Pfeiffer et al. (2021) propose a two-stage algorithm for sharing knowledge across task-specific adapters that consists of an extraction stage and subsequent combination stage. In the extraction stage, the adapters (Houlsby et al., 2019) are trained independently module is added to the top of all single-task adapters. The fusion module is form of attention module (Vaswani el al., 2017), with its query from the input representation of adaplers and the key and value from the output representation of the adapters. Then, the model trains only the fusion module parameters on a target task, therefore learning to combine all the individually trained adapters. Their experiment on natural language understanding tasks shows in-distribution performance improvement on 12 tasks, compared to standard full model fine-tuning on the target task.
individual tasks. In the combination stage, new fusion
12

---

Published in Transactions on Machine Learning Research (04/2025)
4.2
Retrieval of Experts
Expert Training: Custom
Expert Data: Private
Routing Dataset: None
Input Granularity: Task
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation:
None
Generalization: Out-of-Distriben Distribution
User Dataset: Zero-Shot
Jang al. (2023) argue that multitask training may underperform individually trained task experts oquipped with a retrieval mechanism. Their proposed retrieval step encodes unlabelled examples from the target task, compares it to data encoded from each training task, and assigns each target datapoint to a specific trained expert. The expert with the most datapoints assigned to it is retrieved. Experiments are conducted using TO-3B and its associated training and evaluation sets (Sanh et al., 2022). This retrieval approach is shown to outperform TO-3B. Moreover, for certain benchmarks there exists a single oracle expert that performs significantly better than multitask training, showing the potential for better performance with a better retriever.
4.3 AdapterSoup
Expert Training: Custom
Expert Data: Private
Routing Dataset: Nonc
Input Granularity: Task
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation: Parameter
Generalization: Out-of-Distribution
User Dataset: Few-Shot
Chronopoulou et al. (2023) combine different PEFT adapters trained independently over 21 website domains to enable few-shot transfer to novel domains. In order to select which domain adapters the most relevant to the downstream task, the authors explore two approaches. The first uses a pretrained sentence- BERT (Reimers & Gurevych, 2019) representation averaged over 100 samples for each training domain and downstream task to compute a similarity metric. The second approach trains a gaussian mixture model using the representation of 100 samples from each training domain and then maps few-shot samples from the downstream task their closest cluster. In either case, chosen adapters are retrieved and their parameter are averaged to produce aggregate adapter for the downstream task. The authors show that both these approaches oblain belter perplexily on 11 unseen domains than uniformly averaging all experts, and picking single expert according to the same metrics.
4.4 Tuning
Expert Training: Custom
Expert Data: Private
Routing Dataset: Target:
Input Cranularity: Task
Depth Cranularity: Model
Expert Selection: Sparse
Expert Aggregation: Parameter
Generalization: Out-of-Distribune
User Dataset: Multiple
To transfer knowledge from similar lasks to a larget task, Wu al. (2023) make use of the Fisher Information Matrix (FIM)-based Task Vector method (Achille et al., 2019). Specifically, given pool of adapters, they construct new expert for target task by finding the adapters whose FIM is among the top-k most similar and averaging weights (including target task-specific adapter) according to FIM similarity. The experts and their interpolation weights are jointly optimized to improve the target task loss. They also introduce a zero-shot variant, where the single adapter with the highest FIM is picked. Their results show improvement in multiple language and vision tasks.
4.5 MixDA
Expert Training: Custom
Expert Data: Private
Routing Dataset: Target
Input Cranularity: Example
Depth Cranularity: Module
Expert Selection: Dense
Expert Aggregation:
Output
Generalization: In-Distribution
User Dataset: Full
Diao et al. (2023) propose a two-stage algorithm to transfer knowledge from self-supervised domain adapters
target tasks. The first stage involves training domain-specific adapters with masked language modeling objectives on unlabeled data. In addition, mean-square-error auxiliary loss added maintain the similarity between output representations of the domain adapter and the base model's feedforward network.
13

---

Published in Transactions on Machine Learning Research (04/2025)
In the second stage, domain adapters are all added to the model and always activated. A series of MLP. sigmoid gates following the domain adapters control the weight to aggregate their outputs. This aggregated output is fed through a newly-introdueed task adapter. Training in the second stage freezes the base model and domain adapters and updates the gates and task adapter.
4.6 Mo'LoRA
Expert Training: Custom
Expert Data: Private
Routing Dataset: None
Input Granularity: Example
Depth Granularity: Model
Expert Selection: Dense
Expert Aggregation: Parameter
Generalization: Out-of-Distrilbun
User Dataset: Zero-Shot
Mo'LoRA (Maxine, 2023) considers the case where base LLM (specifically, Llama 2) is being fine-tuned on a diverse general dataset (specifically,
(Xu et al., 2023)). To train specialized models, the generalist dataset is first clustered based on embeddings produced by a sentence transformer (Reimers & Gurevych, 2019) and a LoRA is trained on each cluster. Then, the cosine distance between the embedding of a given query and the cluster centroids is used to produce routing distribution. The parameters of the LoRAs are then averaged, weighted according to the routing distribution, and the query is processed using the aggregate LoRA.
4.7 LoraHub
Expert Training: Standard
Expert Data: Private
Routing Dataset: Target:
Input Granularity: Task
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation:
Parameter
Generalization: Cuttoof-Distribution
User Dataset: Few-Shot
Huang et al. (2024b) train one LoRA expert per task collection of 200 tasks from the Flan collection (Longpre et al., 2023), starting from the -lan-T5-Large as the base model (Chung et al., 2024). The experts are used to test few-shot generalization on a suite of 27 tasks from BIG-Bench Hard (Suzgun et al., 2022). LoraHlub performs routing in two steps: first, 20 adapters are chosen at random from the full set of 200 training adaplers; then, for each new lask, the authors learn a fixed routing distribution over the randomly chosen adapters using a gradient-free method over a small task-specific training dataset. The routing probabilities used to compute weighted average of the chosen adapters' parameters to create single specialized adapter. LoraHub is therefore focused on few-shot out-of-distribution tasks, i.e. it evaluates performance on separate set of tasks but requires tasc-ipecific training data to learn routing weights.
4.8 Airoboros and Llamalndex
Expert Training: Multiple
Expert Data: Private
Routing Dataset: None
Input Cranularity: Example
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation: None
Generalization: Multiple
User Dataset: Zero-Shot
Airoboros (Durbin, 2024) is an open-source tool for generating and training on instruction tuning data. It includes functionality for selecting among a pool of expert models. Adaptive routing is supported in two ways: either by embedding 1,000 samples from each expert training dataset and retricving the expert whose embedding is nearest to the query (composed of the system prompt and instruction) embedding via a FAISS index, or by asking an L.M which model use for the query given a list of descriptions of each model. LlamaIndex (Liu, 2024) is an open-source library for connecting LLMs with data sources and other tools. Like airoboros, it includes functionality for building model-level router by querying an LLM, with flexible choicses of the routing model and selection prompt.
4.9 Routing with Benchmark Datasets
Expert Training: Standard
Expert Data: Private
Routing Dataset: General
Input Granularity: Task
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation: None
Generalization: Out-of- Distribution
User Dataset: Pero-Shot
14

---

Published in Transactions on Machine Learning Research (04/2025)
Shnitzer et al. (2024) reuse a collection of benchmark datasets (specifically HELM, Liang et al., 2022) to determine routing among L.Mss on unseen dataset. Specifically, they hold out one dataset while using the remaining datasets, called "benchmark data", for learning the routing. The evaluation is performed on all the LLMs in the pool on the benchmark data, and they define the correctness of cach LLM for a given query with a binary score indicating whether the LLM can provide an acceplable answer to the given query. They embed the benchmark data using a sentence embedder, and for a query from the holdout dataset, the averaged correctness score from the k nearest neighbors in the benchmark data is assigned as the score for this query and LLM. The average of all scores for all queries in the dataset is then taken to estimate how accurate an LLM is for the task. They propose three estimators: the first that takes the arg max of the previously computed correctness scores over all the queries of the dataset. The second estimator applies a threshold on the correct ness score of samples when averaging over queries in the dataset and accounts only for those that cross the threshold. To address out-of-distriuttion tasks, a third proposed estimator takes into account unlabeled out-of-distribution test samples by estimating the probability that the per-test-sample correctness score is accurate. The estimator defaults to the best LLM on the benchmark in cases of low confidence.
4.10 Zooter
Expert Training: Standard Input Granularity: Example
Expert Data: Private
Routing Dataset: General
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation: None
Generalization:
User Dataset: Zero-Shot
Lu et al. (2024a) propose Zooter, a learned router that aims to send each query to the best generalist language model (LM) within a pool of possible modcls. To train the router, predictions over a set of unlabelled instruction data are first collected for all LMs in the pool. The predictions are then scored by a reward model and the normalized scores across models are used as a training signal for the router. The router is kept relatively small (3 orders of magnitude smaller) compared to the LMs to keep routing cost low. Given the inherent noise in the scoring of queries using a reward model, the authors use a form of label smoothing: the reward for a given query is averaged with other queries with the same tags (e.g. "math", "creative writing") obtained from a pretrained tagger (Lu et al., 2023). When evaluated on generalist benchmarks like -eT-Bench (Zheng et al., 2023) or FLASK (Ye et al., 2023b), Zooter performs similarly to naively routing each query to every LM the pool and selecting the response with the highest reward.
4.11 Token- Level Adaptation of LoRA Adapters
Expert Training: Custom
Expert Data: Private
Routing Dataset: None
Input Cranularity: Step
Depth Cranularity: Model
Expert Selection: Dense
Expert Aggregation: Parameter
Generalization:
In-Distribution
User Dataset: Zero-Shot
Bolofsky (2023) formulato a routing approach for independently-trriindd LoRA experts. After training experts on a small set of specialized tasks, they form an expert representation by leveraging the experts' training data. Specifically, for each dataset, they compute the rentroid of the embeddings of the dataset prompts. At test time, they normalize the cosine similarities between the embedding of the prompt generated so far and the experts' embeddings and combine expert parameters based on the resulting weights. The gramularity of their routing approach is step-level and the routing decisions are shared across layers, with expert aggregation at the parameter-level. The evaluation is performed on In-Distribution tasks, i.c. they use the test set of the same lasks the experts have been trained on.
4.12 CALM
Expert Training: Standard
Expert Data: Multiple
Routing Dataset: Multiple
Input Granularity: Task
Depth Granularity: N/A
Expert Selection: None
Expert Aggregation: Output
Generalization: Multiple
User Dataset: Full
Bansal et al. (2024) focus on composing knowledge from two models that can potentially have different
architectures and sizes. Given anchor model and augmenting model, the goal is to have final
15

---

Published in Transactions on Machine Learning Research (04/2025)
model that is good at the anchor task, augmenting task, and a "composition" task that corresponds to the composition of the anchor and augmenting tasks. To achieve this, CALM adds multiple cross-attention layers between the augmenting and anchor model which takes in the input activation from both models. Then, the output from this learned cross-attention layer is passed on to the anchor model. Both the anchor and augmentation models are frozen and the cross-altention layers are learned in an end-to-end manner on a mixture of anchor and augmenting task order improve performance on the composition task. They use PaLM2-XXS as the augmenting model and use PaLM2-XS PaLM2-S as the anchor model (Chowdhery et al., 2023). CALM is shown to be effective in experiments including adding low-resource support to an English model and improving the coding of the anchor model.
4.13 What the Weight?
Expert Training: Standard
Expert Data: Multiple
Routing Dataset: Multiple
Input Cranularity: Example
Depth Cranularity: Module
Expert Selection: Sparse
Expert Aggregation: Multiple
Generalization: Out-of-Distribution
User Dataset: Zero-Shot
Holtermann et al. (2024) do not propose a new method for MoErging but instead introduce a framework under which they can perform experiments to better understand the various components and how they impact zero-shot compositional generalization. They frame such generalization as having three steps; (1) selecting a subset of experts, (2) deciding weights for each expert, (3) combining the different experts based on their weight. They experiment with five different types of scoring functions to select and weigh experts: uniform, sentence similarity, tf-idf, domain priors, and entropy. After selecting the scores and the experts they perform two different types of aggregation, parameter-level and ensembling the outputs. Their large- scale study produces various new insigts, including: ensembling generally yields better results than parameter averaging, good performance can be attained even with simple routing strategies, and that the number of chosen experts is more important than the precise weights assigned to them.
4.14 Routoo
Expert Training: Standard
Expert Data: Private
Routing Dataset: Multiple
Input Granularity: Example
Depth Granularity: Model
Expert: Selection: Sparse
Expert Aggregation: None
Generalization: Out-of-Distributinn
User Dataset: Few-Shot
Mohammadshahi et al. (2024) describe a system that trains a router to perform model-level routing among generalist LLMs of varying sizes and architectures. fixed budget provided and the final objective is to maximize the overall performance across all querics while adhering to the budget constraints. Router training is done using a dataset of (query, response, evaluation score) triplets collected over many possible models. Mohammadshahi al. (2024) use labeled target-task examples (specifically from MMLUU) to synthetically generate the router training dataset with self-play for iterative refinement. However, we note that the method could in principle be used in zero-shot settings.
4.15 Ennggt--nsembling MoE
Expert Training: Standard
Expert Data: Private
Routing Dataset: Target
Input Granularity: Example
Depth Gramularity: Module
Expert Selection: Dense
Expert Aggregation:
Parameter
Generalization: Multiple
User Dataset: Zero-Shot
Tang et al. (2024c) argue that the interference when merging models should be dynamically resolved and hence design MoErging method that averages all parameters except MLP layers which may contain more task-spocific knowledge. They upcycle MLP layers into an MoE where cach MLP from each expert model is converted to a lask vector by subtracting the base model's parameters from the MLP layer's parameters. Routing is then performed between the expert MLP task vectors by multiplying the routing weights with the task vectors and then adding them back the base MLP weight. Routing is done at the example level by taking the mean of all the token-level routing weights. Expert training data access not required, but unlabelled test dataset used to learn the router by minimizing the routing distribution's 's entropy.
16

---

Published in Transactions on Machine Learning Research (04/2025)
4.16 PHATGOOSE
Expert Training: Custom
Expert Data: Private
Routing Dataset: None
Input Granularity: Step
Depth Granularity: Module
Expert Selection: Sparse
Expert Aggregation: Output
Generalization: Out-of-Distribution
User Dataset: Zero-Shot
Mugeeth et al. (2024a) focus zero-shot generalization to unscen tasks by reusing existing adapters that are trained using slightly modified training procedure. For each training task, they first train LoRA module and then they add sigmoid gate before each module which learns the importance of each token for this task. To compute this importance score they compute the sigmoid of the similarity between the gate and the per-token representations. Finally, they optimize the task loss for a given expert to learn these gates. Once LoRA and gales for all lasks are trained independently, then they creale an MoE-style model from these experts for performing zero-shot generalization to unseen tasks. Specifically, for each Lora module, they create router by stacking and normalizing all the gates from different experts. Then they normalize the token representation and route the token to the experts corresponding to the top-2 most similar gates. Results on improving the zero-shot generalization of T5.1.1 demonstrate that this approach outperforms other methods for learning post-hoc routing and can sometimes match the performance of explicit multitask routing.
4.17 LoraRetriever
Expert Training: Standard
Expert Data: Shared
Routing Dataset: Expert
Input Cranularity: Example
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation: Multiple
Generalization: Multiple
User Dataset: Full
Zhao et al. (2024b) train sentence embedding model to map from input query into embedding space that is then used to select expert model to route the query to. The embedding space is constructed by sampling a subset of each of the expert model's training task and computing their average embedding, The embedding model trained on a wide range of tasks in hopes of enabling generalization to unseen tasks and domains. Expert models are created as LoRA adapters to a base model, and are aggregated via merging or top-k output ensembling after routing performed.
4.18 LoRA-Flow
Expert Training: Standard
Expert Data: Private
Routing Dataset: Target
Input Granularity: Step
Depth Granularity: Module
Expert Selection: Dense
Expert Aggregation: Output
Generalization:
User Dataset: Few-Shot
Wang et al. (2024) propose LoRA-Flow, which introduces a fusion gate at each layer of the transformer that processes the input at that layer and generates weights to perform weighted average of the outputs from a set of pretrained LoRAs in the model. Some of these LoRAs are trained on multilingual language modeling, while others are task-specific and trained in English. These weights are generated for every token, making the routing token-level and layer-wise, with dense aggregation at the output level. Few-shot data from the downstream task of interest is used to learn this fusion gate, which comprises linear matrix and bias vector. Experiments were conducted on math and coding abilitics in multilingual setting, specifically MGSM (Shi et al., 2022) for math and HunanEval translated into different languages for code. Their method outperforms LoraHub (section 4.7), which learns weights per task and averages LoRA parameters rather than outputs.
4.19 PEMT
Expert Training: Custom
Expert Data: Private
Routing Dataset: Target
Input Granularity: Task
Depth Granularity: Module
Expert Selection: Dense
Expert Aggregation: Output
Generalization: Out-of-Distribution
User Dataset: Multiple
Lin et al. (2024) propose a method to train a parameter efficient adaptation for new task by utilizing adapters from other tasks. First, for each source task, they train both learnable soft prompt (Lester et al.,
17

---

Published in Transactions on Machine Learning Research (04/2025)
2021) (initialized with task description in Raffel et al. (2020)) and an adapter. Then, for a target task, they initialize soft prompt via attention-style mechanism using the embedded target task description a key and the source task prompts as keys and values. A task correlation matrix is constructed via a similar process, and a separate gating network is trained at each layer taking the correlation matrix as input. The gating network learns a weighting for computing an average of the source lask adapters' outputs. Finally, a new target-task adapter trained on downstream task data along with the gating network, soft prompt, and normalization parameters. This pipeline is shown to outperform other methods for recycling adapters such SPoT (Vu et al., 2021) and ATTEMPT (Asai et al., 2022).
4.20 Co-LLM
Expert Training: Custom
Expert Data: Private
Routing Dataset: Target
Input Granularity: Step
Depth Granularity: Model
Expert Selection: Sparse
Expert Aggregation: None
Generalization: In-Distribution
User Dataset: Full
Shen et al. (2024) trains a binary classifier on the top of a base model's last hidden state to determine when a base model should defer token generation to a frozen large model, facilitating collaboration between models. Given training data for a task, pseudo-labels are generated by evaluating both models and labeling instances where the large model predicts the correct next. token while the base model does not. This data is used to train the classifier's parameters and is further used initialization in the later stage when the classifier and base model are fine-tuned on the task. During inference, threshold is set the classifier using validation data that decides when to defer to the large expert model. This collaborative approach yields better results compared to fine-tuning the base model alone or using the frozen large model independently in instruction following, math, reasoning, and biomedical tasks.
4.21 Branch- Train-Miii--
Expert Training: Standard
Expert Data: Shared
Routing Dataset: Expert
Input Cranularity: Step
Depth Granularity: Module
Expert Selection: Sparse
Expert Aggregation: Output
Generalization: In-Distribution
User Dataset: Multiple
Sukhbaatar et al. (2024) fino-tune cach LLM on four different domains starting from sced LLM (Llama 7B (Touvron et al., 2023)) to create expert LMs for each domain. They propose combining the FFNs of each expert LM to form an MoE, as in Lepikhin et al. (2020); Du et al. (2022), and averaging other parameters from each expert LM. The resultant model is fine-tuned on a training mixture corresponding to all the domains. During inference, top-2 routing is used at each MoE layer. They evaluate on downstream tasks in zero-shot and few-shot settings corresponding to each domain and find that their method performs comparably to the best domain expert LM for that task. Their method also performs comparably to a computc-matched counterpart, where the seed model is scaled to be similar size to the final model by upeycling (Komatsuzaki et al., 2022) and trained using multitask data.
4.22 Dynamic Adapter Merging
Expert Training: Custom
Expert Data: Private
Routing Dataset: None
Input Granularity: Example
Depth Granularity: Model
Expert Selection: Dense
Expert Aggregation: Parameter
Generalization: In-Distribution
User Dataset: Zero-Shot
Dynamic Adapter Merging (DAM, Cheng et al., 2024) leverages domain-specific adapters of a base model to perform incremenall learning in the context of video question answering (VidQA). DAM first computes each domsin-specific training set's average embedding from the penultimate layer of the base model. The distances between a given query input's embedding and the datasct average embeddings are then normalized to create a routing distribution. Finally, the query is processed by merging the domain- specific adapters using per-adapter weights set according to the routing distribution. On standard VidQA benchmarks, DAM significantly outperforms continual learning on the domain-speeific datascts and nearly matches the performance of multitask training. However, the use of the base model embed the input roughly doubles the computational cost.
18

---

Published in Transactions on Machine Learning Research (04/2025)
4.23 Mixture of LoRA Experts (MoLE)
Expert Training: Standard
Expert Data: Private
Routing Dataset: Target
Input Granularity: Step
Depth Gramularity: Module
Expert Selection: Dense
Expert Aggregation: Output
Generalization: Multiple
User Dataset: Few-Shot
Wu et al. (2024) note that directly merging LoRA modules can degrade capabilities. They therefore aim to train routers to aggregate and reweight outputs from LoRAs at each layer where they have been introduced. Router training performed on downstream data with the rest of the model (base model and LoRA param- eters) fixed. In addition to a standard domain-specifie loss, MoLE include load balancing loss that aims to encourage the router lo assign weight to all LoRAs. During inference, MoLE considers the cases where all LoRA outputs are used and where some LoRAs manually removed. Experimental results include analysis of performing model, layer, or module-level routing that demonstrates that module-level gating networks result in the best performance.
4.24
Arrow 7
Expert Training: Standard
Expert Data: Private
Routing Dataset: None
Input Gramularity: Step
Depth Gramularity: Module
Expert: Selection: Sparse
Expert Aggregation: Parameter
Generalization: Out-of-Distribuu
User Dataset: Zero-Shot
Ostapenko et al. (2024) explore methods to build and reuse a library of expert LoRAs for zero-shot task generalization. The proposed solution builds a MoE-like architecture, where the different experts are dy- namically selected according to the input. To build router in zero-shot manner, the authors add linear router at each layer, where expert prototypes are initialized to the top singular vector of given LoRA expert. This then enables per-layer, per-step routing, using the top-4 experts at every selection step. The authors train experts on a 256-task subset of the FLAN dataset (Longpre et al., 2023), using Phi-2 (Microsoft Research, 2023) and Mistral-7B (Jiang et al., 2023) as backbones. They show performance gains both on in-distribution lasks, as well as on collection of 10 held-out lasks, ranging from common sense reasoning to python programming.
4.25 MeteoRA
Expert Training: Standard
Expert Data: Shared
Routing Dataset: Expert
Input Granularity: Step
Depth Gramularity: Module
Expert Selection: Sparse
Expert Aggregation: Output
Generalization: In-Distribution
User Dataset: Full
Xu et al. (2024) propose an efficient method to dynamically select between multiple LoRA adapters. each layer, a learned gating mechanism chooses a predetermined number of LoRAs to be activated for each token. The gating is learned by freezing the network and learning next token prediction over the same datasets used to train the experts. In addition to the architectural change, various engineering choices are made to ensure efficient parallelization of the gating choices, ultimately leading to substantial speedups. While MeteoRA is initially validated on in-distribution tasks with a labeled dataset, it could in principle be applied to ont-of-distribation tasks.
4.26 PWE MoE
Expert Training: Standard
Expert Data: Shared
Routing Dataset: Expert
Input Cranularity: N/A
Depth Cranularity: Module
Expert Selection: Dense
Expert Aggregation: Parameter
Generalization: In-Distribution
User Dataset: Full
Tang et al. (2024b) exlend WE MoE (covered in section 4.15) to settings where Pareto-oplimal performance
is desired on a set of tasks. Task importance is set according to a user-specified "preference vector" (whose entries are nonnegative and sum to 1) that designates which tasks are more or less important. As in WE MoE, specialized models are upcycled (Komatsuzaki et al., 2022) into an MoE-style model by merging non-feed-forwart network parameters via task vector arithmetic (Ilharco et al., 2022). Routers among the
19

---

Published in Transactions on Machine Learning Research (04/2025)
feed-forward networks are trained by sampling random preference vectors and optimizing standard losses that capture Pareto optimality over the expert tasks. Routing based on preference vector-specified task weighting is shown to outperform merging methods that use the preference vector to set model weights.
4.27 RouteLLM
Expert Training: Standard
Expert Data: Private
Routing Dataset: Multiple
Input Granularity: Example
Depth Gramularity: Model
Expert Selection: Sparse
Expert Aggregation: None
Ceneralization: Multiple
User Dataset: Zero-Shot
Ong et al. (2025) aim to reduce inference costs while maintaining performance by dynamically routing each input to either strong or a weak LLM. RouteLLM learns router that estimates the probability that the stronger model will outperform the weaker one on a specific metric for a given input and select the weaker model the probability is below a given threshold. The router is learned using a combination preference data from Chatbot Arena (Chiang et al., 2024), instruction-tunit tuning data such as Nectar (Zhu et al., 2023) and data with gold labels, such as MMLU (Hendrycks et al., 2020). RouteLLM is then evaluated on MMLU, MT-Bench (Zheng al., 2023) and GSMSK Cobbe et al. (2021). Given that MMLU is both used for learning the routing and for evaluation, some examples from the target evaluation tasks and other generic tasks. The paper shows that RouteLLM can learn to choose between GPT-4 and Mixtral effectively, lowering inference costs. In some settings, it was observed that the inclusion of some target-task examples in the calibration data for the router was important to learn an effective router.
described the Routing Dataset Multiple, to denote that it comprises
4.28 Twin-Merging
Expert Training: Standard
Expert Data: Shared
Routing Dataset: Target
Input Cranularity: Example
Depth Gramularity: Model
Expert Selection: Dense
Expert Aggregation: Parameter
Generalization: Multiple
User Dataset: Full
Twin- twin-Merging (Lu et al., 2024b) aims to address the gap in performance between a merged model and the original constituent models being merged. Specifically, Twin-Merging first constructs "shared" model by performing task arithmetic (Ilharco et al., 2022) to merge all of the expert models. Then, "exclusive knowledge"" experts are obtained by computing the rank-r reconstruction from the SVD of the difference between the shared model and each expert model. Finally, a router is trained on a small task-specific training set to generate a weighting over the exclusive knowledge exports. The exclusive knowledge experts are then merged using the router's weights for particular example. By performing adaptive merging, Twin-Merging is shown to create a single model that matches the individual models' performance.
4.29 RAMoLE
Expert Training: Standard
Expert Data: Shared
Routing Dataset: Expert
Input Granularity: Step
Depth Granularity: Module
Expert Selection: Dense
Expert Aggregation: Parameter
Generalization: Multiple
User Dataset: Zero-Shot
Zhao et al. (2024a) extend LoraRetriever (see section 4.17) by including a router that outputs a distribution over the retrieved LoRA modules. The router is architected as LoRA module that trained on a subset of the expert tasks (in the same way as the LoraRetriever- based retricval module). To ensure the router can handle oul-of-distributiin lasks, LoRA modules are randomly dropped out during router training. Routers are introduced at each location in the model where LoRA modules are introduced, producing per-token and per-module routing. Ultimately, performing a router-weit weightd average of modulc paramcters is shown to outperform top-1 routing or unweighted module merging.
5 Related Methods and Problem Settings
Although recycling expert models for decentralized model development has only emerged in recent years, many established research areas closely tied to this problem. Insights and solutions from these directions
20

---

Published in Transactions on Machine Learning Research (04/2025)
may help accelerate progress on MoErging. In this section, we therefore outline several directions related to MoErging, discuss their commonalities and distinctions, and highlight promising connections.
5.1 Multitask Learning
Multitask learning (Caruana, 1997) aims to solve multiple tasks simultancously, usually with a single model, to exploit commonalities across lasks. MoErging shares this high-level goal, but differs in that multitasking learning has mostly evolved in self-contained environments: one party decides on training data, trains the model, and deploys it for that party's own usc. This condition Icads to several koy differences. First, multitask learning usually leverages relatively few tasks - either those the model developer has access to or those directly relevant to the developer's intended use. MoErging methods, on the other hand, aim to receive contributions from more tasks of greater variety and may target a wider range of use cases. Second, when single party controls the entire data and model training pipelines, it is easier to implement sophisticated techniques. In contrast, for decentralized model development, innovations that deviate from the standard practice of sharing independently trained expert models may come at the cost of adoption.
Still, many previous works on multitasking learning tackle common challenges and can shed light on decen- tralized development. Multitask learning works often investigate the problem of task-delasss (Standley et al., 2020; Bingel & Søgaard, 2017; Achille et al., 2019; Vu ct al., 2020; Zamir et al., 2018; Mou et al., 2016), which is relevant to the routing mechanism. On the other hand, to better handle the commonalities and differences of multiple tasks, MTL has designed architectures that flexibly integrate task-specific components into the network (Misra et al., 2016; Ruder et al., 2017; Meyerson & Miikkulainen, 2017; Zaremoodi et al., 2018; Sun et al., 2019), which could be adopted to combine experts.
5.2 Mixture-of-Expert models
Mixture--ffexxpets (MoE) approaches train a set of experts that are sparsely activated during pre-training and inference (Jacobs et al., 1991; Jordan & Jacobs, 1994; Fedus et al., 2022; Shen et al., 2023; Shazeer et al., 2017). One of the main motivation for this approach is to scale model capacity while maintaining a fixed inference cost. In practice, experts are typically applied at the fully connected layers of the Transformer architecture (Lepikhin et al., 2020; Fedus et al., 2022). A sparse number of exports are typically activated per-token and per-layer by a learnable routing function. Switch Transformer (Fedus et al., 2022) only selects one top scoring expert and has demonstrated better scaling properties than previous work. Recent work applied mixture-offexpprts also to attention layers (Shen et al., 2023). Given the discrete decision on which experts to apply, gradient estimation becomes difficult which makes training more challenging. Techniques such adding load balancing losses have been proposed to ensure all experts are used (Fedus et al., 2022). Recent work formulated improvements to sparse back-propagation that show promise in training better sparse MoE models (Liu et al., 2023). Overall, mixture-of-experta approaches generally focus on end-to-end "from scratch" training of experts and the router instead of learning post-hoc routing from independently trained experts as in MoErging. Kang al. (2024) use the base LLM to generate synthetic data to create an MoE model by upcyeling where each expert is specialized to some domain. MoErging may also introduce challenges that are distinct to those encountered when training MoF. models from scratch for example, it might be suboptimal for routing in a MolEged model to be balanced across experts, which could make it more challenging to distribute the model over many accelerators.
The pool of experts in an MoE-style model can be trained jointly in multitask settings. Restricting the size of the expert pool to be smaller than the number of tasks can enable similar tasks to share experts while avoiding interference across tasks with negative transfer. Polytropon (Ponti et al., 2019) and MHR (Caccia et al., 2023) learn task-level router over experts, composing them into task-specific experts via weighted average in parameter space. Task-level routing has also been explored in standard MoEs for more efficient inference (Kudugunta et al., 2021). Other routing strategies have also been explored in multitask MoE models. SMEAR (Mugeeth et al., 2024b) performs example-level routing by averaging expert parameters, while Zadouri et al. (2023) learns a per-token routing stratogy. While MoErging mothods generally assume expert models are trained independently, multitask MoE methods can be considered an important bascline when expert data is shared (in which case multitask training is possible).
21

---

Published in Transactions on Machine Learning Research (04/2025)
MoE models have also been employed in contimual learning to prevent forgetting by preserving important experts from previous lasks while updating adding new experts for new tasks. For example, Chen et al. (2023) continuously trains an MoE' model by freezing old feed-forward network (FFN) experts and incorporating new ones with KL regularization, outperforming dense continual pretraining. Aljundi et al. (2017) trains lask-specific experts and uses autoencoders to identify the closest expert for new task and either fine-tunes or adds a new expert initialized from the closest experts weights, demonstrating effectiveness in image classification and video prediction problems. Yu et al. (2024) uses Low-Rank Adaptations (LoRAs) experts, employing router per task to weight each expert and using autoencoders on pretrained imput features to select the appropriate model for new task. This approach also freezes the top-k experts from previous tasks to prevent forgetting. Li et al. (2024) offers theoretical analysis into the benefits of MoEs in continual learning on a sequence of linear regression tasks. Doan et al. (2023) suggests using model ensembles to reduce forgetting and adds loss terms to maintain linear mode connectivity between weights across tasks. While most MoErging methods do not explicitly target a continual lcarning sctting, techniques for continual learning with MoEs could be applicable to MoErging methods (e.g. to continually update the router to account for a growing set of experts from the contributors).
5.3 Model Merging
Model merging aims to combine independently trained models that share an architecture into a single model that retains the individual models' capabilities. Merging can be performed via simple parameter averaging (McMahan et al., 2017; Stich, 2018) if the models fall into a linearly connected low-loss region in parameter space (Frankle et al., 2020; Wortsman et al., 2021). In the context of multiple expert models trained on different domains, (Mharco et al., 2022) proposed task vectors, denoted as the parameter shift. from the base model to the task-specific model and show that these can be combined to obtain a single generalist model. Several works go beyond uniformly averaging parameters, and explore various heuristics to estimate parameter importance. (Matena & Raffel, 2022; Tam et al., 2023) leverage the Fisher Importance Matrix to perform weighted merging- TeES-Merging (Yadav et al., 2024) resolve sign disagroemeent across experts and trim redundant values before merging. Ye et al. (2023a) uses a gating network trained on unlabeled data to predict task probabilities used to merge ViT layers and automatically select the appropriate classifier during inference. Merging can be considered a non-adaptive way of aggregating independently trained experts and is therefore a reasonable baseline for many MoErging methods.
Merging has also been applied to a continual learning setting where more expert models become available over time. Some efforts used such approaches to pretraining, slicing their data for efficient training (Li et al., 2022; Huh et al., 2024). the contribution of actual experts. While this line of work is similar to MoErging in terms of its focus on recycling a growing set of expert models, differences include that all models are being merged into one, and also that the new experts are already based on the previous model and are hence not completely independent.
Ychiya et al. (2023) trained each expert
different datasct to simulate
5.4 Federated Learning
Federating learning (FT.) is widespread framework for collaborative learning where locally updated models trained on private user data are shared with a maintainer to improve a base model (McMahan et al., 2017). Some work has explored the intersection of MoE models and federated learning as a way to mitigate the data heterogeneity issue, where non-IID datasets can considerably diminish the accuracy of the FL. model (Parsaccfard et al., 2021). FodMix (Reisser et al., 2021) and FodJETs (Dun et al., 2023) enable cach client to construct an MoE with shared router and a homogeneous local model. The router selects specific local models that are better adapted to a clients local data for output ensembling. However, they suffer from significant communication costs due to the model's distribution shifts.
In the personalized FL setting, PEL-MoE (Guo et al., 2020) customizes a MoE model by adapting the global model from the server and fine-tunes it to create a local expert model. These two models are then integrated with a local router into an MoE. During MoE training on the local data, the global expert remains frozen, while only the local expert and the router are updated to achieve personalization. More recently, PFedMoE (Yi et al., 2024) has introduced sophisticated non-linear router network to extract more knowledge from local
22

---

Published in Transactions on Machine Learning Research (04/2025)
data. This router network weights the final representations from each expert model, rather than their output labels like vanilla ensembling, in order to enhance the integration of global generalized and local personalized features. Work at the intersection of FL and MoE models is likely most relevant in cases where a user-tailored local model is desired and where user data must remain private.
6 Common Applications and Tools for MoErging
Development of an impactful machine learning model or a piece of software can happen in isolation. In con- trast, MoErging methods provide a framework for decentralized development that relies on contributors for success. Consequently, appropriate infrastructure, systems, and tools are critical to the success of MoErging as a mode of decentralized model development. To facilitate this form of development, several open-source tools and platforms have emerged. These resources empower researchers and practitioners to experiment with different MoErging techniques, share their expert modcls, and collaborate on building more capable and robust Al systems.
Hugging Face has become a central hub for sharing pre-trained models and PEFT adapters. Initiatives like the Open LLM Leaderboard and AdapterHub (Beck et al., 2021) facilitate the discovery and comparison of models across various lasks and domains. Git-theta (Kandpal el al., 2023b) offers a decentralized alvernative for tracking model weights, leveraging the Git version control software for collaborative development.
Several libraries have been developed to streamline the process of creating, combining and routing among expert models. Libraries like Hugging Face PEFT, Predibase's Lorax, Axolotl, and Unsloth can be used to efficiently create expert models either via full-finetuning or by parameter-efficintt finetuning. These experts can then be used in libraries like Arcee's MergeKit (Goddard et al., 2024), Flow-Merge, Predibase's Lorax, Mergoo, and airoboros, each of which supports different forms of merging and MoErging. These libraries provide convenient interfaces for loading, managing, and aggregating expert models with different. routing and aggregation strategies. Apart from these, ComfyUl UI offers a user-friendly graphical interface for designing and experimenting with complex merging pipelines, enabling users to visually connect and configure different expert modules to build performant aggregate systems.
The ongoing development of these tools and platforms highlights the increasing interest in MoErging and Merging for decentralized model development and ils potential to democratize AT research and development. As the field matures, we can expect to see even more sophisticated tools and applications emerge, further empowcring the community to collaboratively build and share more powerful and adaptable AI systems.
7 Takeaways and Open Problems
While there have been many papers developing MoErging methods that demonstrate improved performance or generalization, the field is still in its infancy. Notably, very few of the methods we surveyed are actually used in practice. We speculate that this could be due to a few factors: First, while many software tools that support MoErging exist (cataloged in section 6), many MoErging methods do not have user-friendly software inplementations available. As such, applying MoErging often requires technical expertise that many users lack. Second, as we have highlighted in this survey, it is often unclear which MoErging methods are applicable and/or optimal for a given use case. More rigorous comparisons could be made through benchmarks, competitions, and empirical surveys like those developed for model merging (Tam et al., 2024b;a; Tang et al., 2024a). In addition, assumptions made by many MoErging methods (such as a custom expert training procedure and/or sharing expert training data) make them inapplicable unless the standard practice of solely sharing expert parameters changes. Moreover, as the field progresses, a crucial direction for future work lies in developing robust theoretical frameworks for MoErging, Such theoretical underpinnings are essential to formally understand the properties of different MoErging approaches, guide the design of more effective methods, and establish a deeper understanding of the conditions under which MoErging can offer substantial benefits. Finally, the promise of MoErging as a framework for collaborative learning may be held back by the lack of platforms that facilitate the various stages of a MoErging pipeline. For example, while the Hugging Face model hub provides an casy and widely used way to share models and datasets, does not include extensive features for coordinating continual and communal model development.
23

---

Published in Transactions on Machine Learning Research (04/2025)
If MoErging does ultimately become more widespread, there will be new challenges to tackle. For example, how can we identify (and possibly remove) redundant expert models? Or decide whether or not to add a new model to the pool? Or identify if a given model has been contributed by a malicious user who aims to degrade the aggregate system? In addition, adoption of MoErging will motivate the development of platforms that provide a "closed loop" R for continual model development, where users can easily and cheaply fine-Lune expert models and then share them to continually improve the aggregate system. In such cases, there will be multiple rounds of MoErging, raising additional research questions around whether MoErging can be used for continual improvement (or even pre-training) of base model rather than one-off improvements single target task.
As the field of MoErging matures, it's likely that our survey and laxonomy will become out-of-date. In addi- tion, while we aimed to be comprehensive, is possible that we have missed certain methods or overlooked important design choices our taxonomy. However, we hope that our survey provides a solid foundation and launch-off point for rigorous and coordinated rescarch on MoErging.
Acknowledgments
We thank Feng Cheng, Qian Liu, Jingwei Xu, Ziyu Zhao, Shannon Shen, Yong Luo, Omar Sanseviero, Jonas Pfeiffer, Shizhe Diao, Alireza Mohammadshahi, Rachit Bansal, Zhisheng Lin, Carolin Holtermann, Tal Shnitzer, Mikhail Yurochkin, Joshua Belofsky, Keming Lu, Xun Wu, Joel Jang, Alexandra Chronopoulou, and lan Attwel for feedback on our survey.
References
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vee: Task embedding for meta-learning In Proceedings of the IEFF/CVF international conference on computer vision, pp. 6430-646 6439, 2019.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3366- 3375, 2017.
Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. Attempt: Parameter- efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6655 6672, 2022.
Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. Llm augmented 1lms: Expanding capabilities through composition. In The Twelfth International Conference on Learning Representations, 2024.
Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. Adapterhub playground: Simple and flexible few-shot learning with adapters. arXiv preprint arXiv:2108. 08103, 2021.
Joshua Belofsky. Teken-level adaptation lora adapters for downstream task generalization. In Proceedings of the 2023 6th Artificial Intelligence and Cloud Computing Conference, pp. 168 172, 2023.
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks arXic preprint orXiv: 702.7022.0000 08303, 2017.
Lucas Caccia, Edoardo Maria Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (ods.), Advances in Neural Information Processing Systems, volume 36, pp. 56916- 56931. Curran Associates, Inc., 2023. URL https::/procceeding neurips...... wwtpriippccppppprreeee .pff.
Rich Caruana. Multitask learning. Machine learning, 28:41 1997.
24

---

Published in Transactions on Machine Learning Research (04/2025)
Wuyang Chen, Yanqi Zhou, Nan Du, Yamping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Life- long language pretraining with distributtonnsspeeillieed experts. In International Conference on Machine Learning, pp. 5383 5395. PMLR, 2023.
Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Bertasius. DAM: Dynamic adapter merging for continual video qa learning. arXiv preprint arXi::240..0877 2024.
Wei-Tin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating 1lms by human preference, 2024. URL https: //trxv///xxvv.///a wwwxxiv.or//bbi
www.ooo////////eo.ooo.oooo aaaaab//22033 04132.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modcling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptcrsoup: Weight averaging to improve generalization of pretrained language models. In EACL (Findings), 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahmna, et al. Scaling instruction-finetuee language models. Journal of Machine Learning Research, 25(70):1 53, 2024.
Karl Cobbe, Vineet, Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL
Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and T. Zhang. cooturr-of-domai---ddapp Decoupling and injecting domain knowledge to pre-trained language models memories. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https: semantccscollar heng/CorpsID:: 259108831.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.068 06904, 2022.
Thang Doan, Seyed Iman Mirzadeh, and Mehrdad Farajtabar. Continual learning beyond a single model. In Conference on Lifelong Learning Agents, pp. 961-991. PMLR, 2023.
Shachar Don- Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen. ColD fusion: Col- laborative descent for distributed multitask finetuning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics ( (Volume 1: Long Papers), pp. 788 788-806, Toronto, Canada, July 2023. Association for Computational URL httts:
Linguistics. doi: 10.18653/vl1 11/2023.acl-longg
46.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmrenel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. CoRR, 2024.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture- of-experts. In International Conference on Machine Learning, pp. 5547-5569. PMLR, 2022.
Chen Dun, Mirian Hipolito Garcia, Guoqing Zheng, Ahmed Hassan Awadallah, Robert Sim, Anastasios Kyrillidis, and Dimitrios Dimitriadis. Fedjets: Efficient just-in-time personalization with federated mixture of experts, 2023.
Jon Durbin. airoboros: Customizable implementation of the self-instruct paper. https://github.cr jondurbin/airoboros,, 2024.
25

---

Published in Transactions on Machine Learning Research (04/2025)
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 39, 2022. URL htm.....m.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawelz. Arcee's mergekit: A toolkit for merging large language models. arviv preprint Xix::22003. 13257, 2024.
Binbin Guo, Yuan Mei, Danyang Xiao, Weigang Wu,, Ye Yin, and Hongli Chang. Pfi-moe: Personalized federated learning based on mixture of experts, 2020.
Shashank Gupla, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed H Awadallah, and Jianfeng Gao. Sparsely activated mixture-of-experts are robust multi-task learners. arXiv preprint Xrx:22204..0 07689, 2022.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parrameter-fffcient transfer learning. arXiv preprint arXiv:2110.0/42 043666 2021. URL https:// srxiv.org/pdee /2110. 04366. pdf.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ar preprint arXiv:2009.03.0 033000 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa. Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint ar (tx:::171..0000000 2017.
Carolin Holtermann, Markus Frohmann, Navid Rekabsaz, and Anne Lauscher. What the weight?! uni- fied framework for zero-shot knowledge composition. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 1138-1157, St. Julian's, Malta, March 2024. Association for Computational Linguisties. URL /ttps:://clandee wwwpchhoooooyooooooooooooo org/2024. findings-eacl.77
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parametor-effiiinn transfer lcarning for NLP. In Interna- tional Conference on Machine Learning, pp- 2790 2799, 2019. URL http://proceedingg mlr, wtt.gg.s.r....e//////m v97 / pdf.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL htttp:://ppeereviee http:///ppppeeeeeeeeeeeerrrrerre wwt..e/eoorrrrt.ddeeeeeeee.....
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. In First Conference on Language Modeling, 2024a.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task gencralization via dynamic lora composition, 2024b.
Minyoung Huh, Brian Cheung, Jeremy Bernstein, Phillip Isola, and Pulkit Agrawal. Training neural networks from scratch with parallel low-rank adapters. arXiv preprint arXic:2402. 16828, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprinl arXiv:2212.0 04009, 2022.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991. URL https:://atteraat/ www.aaarrrrraassseeeeeeeaaar 3.1.79.0 .d..f.
silverchair. www.rrrrraai.ccooooooo.oo 1991.
26

---

Published in Transactions on Machine Learning Research (04/2025)
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontac Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In International Conference on Machine Learning, pp. 14702 14729.. PMLR, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprinl ar xrXo:2330006885, 2023.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181 214, 1994.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowlodge. In International Conference on Machine Learning. PMLR, 2023a.
Nikhil Kandpal, Brian Lester, Mohammed Muqeeth, Anisha Mascarenhas, Monty Evans, Vishal Baskaran, Tenghao Huang, Haokun Liu, and Colin Raffel. Git-theta: A git extension for collaborative development of machine learning models. International Conference on Machine Learning, ICML, 2023b.
Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Ramoswar Panda, Rogerio Foris, and Alan Ritter. Sclf-moc: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406.. 12004 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXie preprint 2xXiv2001108366 2020.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of -f-eeeertttt-tt from dense checkpoints. arXiv preprint arXin:2212 2. 05055, 2022.
Sncha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh- Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pp. 3577-3599, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 111138/v122222 /ttp///...nnnnnnnmmmn.....a wttp////......iingg.oo... www.:iiininggene........... emnlp.304.
URL
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for paramoter-officien prompt tuning, 2021. URL https://aran org/ /ttp///ff2104.....//////1 /2104. 110691.pff..
Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B Shroff. Theory on mixture-of-experts in continual learning. arXiv preprint arXiv:2406.1000 16437, 2024.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah Smith, and Luke Zettle- moyer. Franch-train-merge. Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208. 03306, 2022.
Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al. 11m-101b: An open Ilm and how to train it with $100k budget. arXiv preprint arXiv:2309.0883 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. arXiv preprint ar 1.1::22211.091 2022.
27

---

Published in Transactions on Machine Learning Research (04/2025)
Zhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li, and Jianling Sun. Pemt: Multi-task correlation guided mixture-offexperts enables parameter-efficint transfer learning. In Findings of the Association for Com- putational Linguistics ACL 2024, pp. 6869 6883, 2024.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. Domain specialization as the key to make large language models disruptive: comprehensive survey, arXie preprint 18870 2023.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin
Raffel. Few-shot parameter-cffiient fine-tuning is better and cheaper than in-context learning, 2022. URL
https://arxis: http//axxvvvv...aaaaa/ argg/bs//2006
05638. 25660.
Jerry Liu. LlamaIndex, a data framework for your LLM applications. https://gitthub.com com/runnnlamaa/ llama_imnaa index, 2024.
Liyuan Liu, Jianfeng Gao, and Weizhu Chen. Sparse backpropagation for moe training, 2023. URL https:
//arxiv. wtg..bs///.........//./.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Lc, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. Xixiv preprint ar 2rx:v:2011. 13688, 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2023.
Kemiin Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 1964 1974, 2024a.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. Advances in Neural Information Processing Systems, 37:78905 78935, 2024b.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan.
Peft: State-of-the-art, parameler-efficeen fine-tuning methods. https: htaps:///tthbbbb com cow/huuggingfaccepptet.t. 2022.
Michael S Matena and Colin Raffel. Merging models with fiightee averaging. Advances in Neural Information Processing Systems, 35:17703 17716, 2022.
Maxine. Llama-2, mo lora. https://crunbly http:///rrmmbl...diuum.m.m.
2023.
R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint Xrxi::2309. 13638, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commanicttion-cffittenn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.
PMLR, 2017.
Elliot Meyerson and Risto Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. ArXio, abs/1711.00108,, 2017. URL https://apl. www.peiscsaattccccsssssseeee http:://ppp.ssmmtttttccoooooo 3285020.
wwwphholrrrrrr/ororooeeee org/CorpusID:
Microsoft Research. Phi-2: The Surprising Power of Small Language Models, 2023. URL https://www
microsoft. wwt.///nnsssrreeeeeed//boloo thtesssr----------iriii-i-- of small wwwssmmmaaaaageeeeeeeeeeaeaaaaee -anguagge-mmdllss/
28

---

Published in Transactions on Machine Learning Research (04/2025)
Ishan Misra, Abhinav Shrivastava, Abhinav Kumar Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. 2016 IEFF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3994 4003, 2016. URL https: wwmanticschollatttttt org/CorpusID::aa.. 1923223..
Alireza Mohammadshahi, Arshad Rafiq Shaikh, and Majid Yazdani. Routoo: Learning to route to large language models effectively. arXiv preprinl or Xrx:::241.1 139799 2024.
Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. How transferable are neural networks in nlp applications? In Conference on Empirical Methods in Natural Language Processing, 2016. URL
https://appl http://ppp..saanncccccccctssss semanticscholar. org/CorpusID: 11866664.
Mohammed Mugeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization. In Ruslan Salakhutdinor, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st Indernational Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. (8829-36846 PMLR, 21-27 27 Jul 2024a. URL https: //t:://prrooo http///roeeeen...e waaiingsmmmrrrr....... mml. Saalr.pres///e arress/v33//mme htts///mueehhaa..... hhmml
Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. Trans- actions on Machine Learning Research, 2024b. ISSN 2835-8856. URL https: https:://ppnrreviev.. www.orerwee/errrrm id=7I1991c54z. Featured Certification.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and lon Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieee wwwpaarwwwieeeeeeefeeeoeee net/formm?id 8sSqNntaMr.
Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular LLMs by building and reusing a library of LoRAs In Rus- lan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Leurning, vol- ume 235 of Proceedings of Machine Learning Research, pp. 38885-38904. PMLR, 21-27 Jul 2024. URL html.
https: pttps://proceeddie mlr. wttppp////////taeeeeeeeeeee
Saeedeh Parsaeefard, Sayed Ehsan Etesami, and Alberto Leon Garcia, Robust federated learning by mixture experts, 2021.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destruetive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pp. 487 503, April 2021. URL https:
//aclanthology. www.:/olooooogg//2....ooo wwcc-mainn.......aam..ammmmmmmmm 39.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. Modular deep learning, arXiv preprint arXiv:2302. 11529, 2023. URL https: // //tp////xvvv///////////////aaaaa .owg org/pdff /2022. 11529.pdd
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprinl ar XXii:2307.01953 2023.
Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Tvan Vulić, Roi Reichart, Thierry Poibeau, Flktternna Shutova, and Anna Korhonen. Modeling language variation and universals: A survey typological linguistics for natural language processing. Computational Linguistics, 45(3):559-601, 2019. URL https: //vatermark. silverchair... ctt.coooccaoaa
com/coli coll:olia a 00.000057..0 .pdf.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020. URL fttps://wwx 20- 20-074/20-00 074. pdf.
htte/pppers//olmme2//
29

---

Published in Transactions on Machine Learning Research (04/2025)
Nils Reimers and Iryna Gurevych. Sentenco-bert: Sentence embeddings using siamese bert-networks. arXiv preprinl arXiv: 10084, 2019.
Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling. Federated mixture of experts, 2021.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Sogaard. Latent multi-task architecture learning. In AAAT Conference on Artificial Intelligence, 2017. URL https://api sewasti cacholar.. org/CorpusID. : 115985550.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafoai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoli Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachol Bawden, Thomas Wang, Trishala Nocraj, Jos Rozen, Abhcosht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. Tn The Tenth International Conference on Learning Representations, 2022. URL https: //arx/aa.ivvv///a//aaa /gg/ $110.08200 pdf.
Noam Shazeer, Azalia Mirhoscini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference Learning Representations, 2017. URL https://openreevee: wed.
Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Modulelormer: Learning modular large language models from uncurated data. arXiv preprint arxi::2006.040 2023.
Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collabora- tively with multiple language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12974-12990, 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thouge reasoners. Xiviv preprint orXim:221000900 2022.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuckai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. In Pirst Conference Language Modeling, 2024.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-lask learning? In International conference on machine learning, pp. 9120 9132. PMLR, 2020.
Sebastian U. Stich. Local sgd converges fast and communicates little. arXio preprint 2018.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Was Wen Li, Wen-tau Yih, Jason E Weston, ct al. Branch-train-men Mixing expert 11ms into a mixture-of-experts Ilm. In First Conference on Language Modeling, 2024.
Ximeng Sun, Rameswar Panda, and Rogério Schmidt Feris. Adashare: Learning what to share for effi- cient deep multi-tase tast learning, ArXiv, abs II 208513386.
2019. URL h wwwanticscholar.oogo org/
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thougle can solve them. arXiv preprint arXiv:2210. 09261, 2022.
Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task subspaces. ar preprint arXiv:2312.0 04339, 2023.
30

---

Published in Transactions on Machine Learning Research (04/2025)
Derek Tam, Yash Kant, Brian Lester, Igor Gilitschenski, and Colin Raffe. Realistic evaluation of model merging for compasitional generalization. or Xrxie preprint, 2024a.
Derek Tam, Margaret Li, Prateek Yadav, Rickard Babii-Gabrielsson, and Jiacheng Zhu. Llm merging competition: Building lims efficiently through merging, 2024b. URL https///tnn-nerging github. io/.
Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Do, and Dacheng Tao. Fusionbench: A comprehensive bench- mark of deep model fusion. arXiv preprint arXiv:2406. 03280, 2024a.
Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, and Bo Du. Towards efficient pareto set approximation via mixture of experts based model fusion. arXiv preprint arXic:2406. (6.:94406009770, 2024b.
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Morging multi-task models via weighl-ensembling mixture of experts. In International Conference on Machine Learning, pp. 47778 47799. PMLR, 2024c.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advunces in neural informalion processing systems, 30, 2017.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across nlp tasks. arxiv preprint ar arXt::2055007700, 2020.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110. 07904, 2021.
Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12871 12882, 2024.
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint ar XrXiv:2211. 1. 1.05100, 2022.
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In International Conference on Machine Learning, pp. 11217 11227. PMLR, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korublith, et al. Model soups: averaging weights of multiple fino-tuned models improves accuracy without incrcasing inference time. In Interna- lional Conference on Machine Learning, pp. 23965 23998. PMLR, 2022.
Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. pi-tuning: Trans- ferring multimodal foundation models with optimal multi-task interpolation. In International Conference on Machine Learning, pp. 37713 37727. PMLR, 2023.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of loRA experts. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevee http://wwwrrre.eeeeeeoo//////a/a
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304 12244, 2023.
31

---

Published in Transactions on Machine Learning Research (04/2025)
Jingwei Xu, Junyu Lai, and Yumpeng Huang. Meteora: Multiple-tasks embedded lora for large language models. Xixiv preprint arXiv:2405. 13053, 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yonggi Huang, Yuning Zhang, and Wanli Ouyang. Merging vision transformers from different tasks and domains. arXiv preprint 2i2:22312112200 2023a.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprinl or XrX::23007. 10928, 2023b.
Liping Yi, Han Yu, Chao Ren, Heng Zhang, Gang Wang, Xiaoguang Liu, and Xiaoxiao Li. pfedmoe: Data- level personalization with mixture of experts for model-heetrrggeneous personalized federated learning, 2024.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture--o-cxxpors adapters. In Proceedings of the IEEE/CVP Conference Compuler Vision and Pallern Recognition, pp. 23219 23230, 2024.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint vrXiv:2309004444 2023.
Amir Zamir, Alexander Sax, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018. URL https://api. semanticscoolar.
wttpaiicccooaaaorooooooooooo org/ Cerg/orrpssID:
5046249.
Poorya Zaremoodi, Wray L. Buntine, and Gholamreza Haffari. Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation. In Annual Meeting of the Association for Computational Linguistics, 2018. URL https: semanticscholar org/ :otr///ID::1115555500 518877777
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei Wu. Retrieval-augmented mixture of lora experts for uploadable machine learning. arXiv preprint 2rXiv:2406. 16989, 2024a.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Lo- raretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. In Findings of the Association for Computational Linguistics ACL 2024, pp. 4447-4462, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as- a-judge with mt- betcbench and chatbot arena, 2023. URL https://arxivvor//aaaa
05685.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving 1lm helpfulness & harmlessness with rlaif, November 2023.
32


Reviews:
[{"review_id": "LxXGacJgV6", "reviewer": "Reviewer_MJx4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has the following advantages:\n- This paper provides a detailed method classification framework, such as expert training methods and data requirements, routing’s data, depth, and granularity, and different downstream applications.\n- This paper discusses the details of more than 20 related methods.\n- This paper discusses the relationship between MoE merging and multiple fields (multi-task learning, model merging, etc.).\n\nHowever, this paper still has the following weaknesses:\n- This paper does not clearly explain the challenges/difficulties in developing a new MoE-based model merging method, and what aspects need additional attention. In addition, an in-depth discussion of which settings/processes of machine learning can be changed by model merging can further strengthen the necessity of model merging techniques.\n- This paper overemphasizes the details of each method in Section 3, but lacks a macro comparison between methods. The author needs to add some systematic illustrations to more clearly explain the overall architecture/paradigm of a class of methods and how the design differences between methods are reflected.\n- The author needs to further discuss in which settings MoE merging may be effective, in which settings it may fail, and why it is effective/ineffective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses \"Model merging for collaborative learning\", an important topic that allows decentralized model development and the ability to reuse expert models in new tasks. This paper comprehensively discusses and compares existing MoE merging work from three levels: expert design, routing design, and application design, so that the community can compare these methods fairly. However, this article still lacks in terms of the challenges in developing MoE Merging methods and the unified framework presentation of existing methods. In general, this paper still needs to undergo a revision to improve the overall quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As discussed in the weaknesses section, the authors need to make further revisions. In addition, some additional revisions are as follows:\n- This paper does not have any figs, which makes it difficult to read. The authors can consider adding some figures to clearly compare different MoE Merging methods and different routing design methods, so that readers can more easily understand the differences and connections between various methods.\n- Some recent MoE-based model merging works [1-5] can be considered for citations and discussions. The authors need to further check other recent literature.\n  - [1] Oh, Changdae, et al. \"Adapting Foundation Models via Training-free Dynamic Weight Interpolation.\" Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning.\n  - [2] Shen, Li, et al. \"Efficient and effective weight-ensembling mixture of experts for multi-task model merging.\" arXiv preprint arXiv:2410.21804 (2024).\n  - [3] Tang, Anke, et al. \"Smile: Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models.\" arXiv preprint arXiv:2408.10174 (2024).\n  - [4] Kang, Junmo, et al. \"Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts.\" arXiv preprint arXiv:2406.12034 (2024).\n  - [5] Ye, Peng, et al. \"Merging Vision Transformers from Different Tasks and Domains.\" arXiv preprint arXiv:2312.16240 (2023).\n- The authors need to update the published version information of the paper instead of citing the Arxiv version, for example, Retrieval of Experts [1] was published in ICML 2023. Adaptersoup was published in EACL 2023. Authors need to check the publication information of other important literature and update it.\n  - [1] Exploring the Benefits of Training Expert Language Models over Instruction Tuning. ICML, 2023.\n  - [2] Adaptersoup: Weight averaging to improve generalization of pretrained language models. EACL, 2023.", "broader_impact_concerns": "This article is a survey and does not involve questions about the impact of work ethics.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ns59CWD0XC", "reviewer": "Reviewer_TRBH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper reviews and categorizes many methods, contextualizes them and provides open problems.\n* The work has an encyclopedic structure, describing each method in a single paragraph with a couple of references and a categorization table. It allows for a quick recall or fetching of a relevant reference.\n\nWeaknesses: \n* The brevity of the description for each method limits the paper's ability to make it self-contained work.\n* While reviewing a lot of methods, the paper does not necessarily guide with respect to the use of surveyed methods. This leaves the reader \nwho would like to use this survey for practically setting up their own pipeline with a sense of undertaste of wanting more. For instance, in Section 6, we can read \n\t* `the field is still in its infancy`,\n\t* `it is often unclear which MoErging methods are applicable and/or optimal for a given use case`,\n\t* `assumptions made by many MoErging methods (such as a custom expert training procedure and/or sharing expert training data) make them inapplicable`.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to review and categorize 29 MoErging Methods. There are also sections dedicated to broader context, tooling, and open problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LCU4aGAktV", "reviewer": "Reviewer_t29w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "##### Summary of strengths\n\n- The paper addresses a highly significant and timely topic in the field.\n- The paper proposes a taxonomy that organizes and classifies various MoErging methods into three broad categories: expert design, routing design, and application design, along with several detailed subcategories. This framework offers a clear framework for comparison and understanding.\n- The paper presents a thorough review of existing MoErging methods, covering numerous works in the field and related problem settings.\n\n##### Summary of weaknesses\n\n- The paper lacks visual illustrations and formalized equations.\n- It does not include experimental results or analyses that compare the strengths and weaknesses of different methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive survey of Model MoErging, which shares concepts with MoE (Mixture of Experts) and model merging but introduces distinct ideas. MoErging methods adaptively combine models to improve performance on a per-query or per-task basis. This approach shows great potential, and the paper, as a survey, establishes a solid foundation for future research in this emerging field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While no major revisions are necessary, I offer the following suggestions:\n\n- As mentioned, adding figures and math formulations would improve clarity.\n- If available, providing theoretical foundations for MoErging methods is beneficial.\n\n- It would be helpful for the authors to update the final version with recent publications, as this is an evolving field. Suggested papers include:\n\n  [1] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, ICLR 2024.\n\n  [2] Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing.\n\n  [3] RouterBench: A Benchmark for Multi-LLM Routing System.\n\n  [4] RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models, NeurIPS 2024.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths: 
- The survey is well-structured, providing a broad overview of MoErging techniques. 
- It categorizes methods effectively and discusses their relevance to related areas such as mixture-of-experts (MoE) models and multitask learning. 
- The paper introduces a useful taxonomy for organizing and comparing different model merging strategies. 

Weaknesses: 
- The paper lacks illustrative figures and equations that could improve conceptual clarity. 
- It does not provide an explicit comparison of methods in terms of performance, trade-offs, or practical effectiveness. 
- The discussion on open challenges and future directions could be expanded to offer more actionable insights.
- More details on applicable scenarios for different merging techniques would improve its usability for practitioners. 

Some reviewers recommended major revision due to the lack of visual aids and comparative discussions. Others acknowledged the paper’s significance but suggested improvements to ensure a more structured and accessible presentation. The authors have made progress in addressing concerns by incorporating additional explanations and figures, but further refinements are needed to enhance clarity and completeness. 

Given the importance of this survey and its potential to shape future research in model merging, I recommend minor revisions to strengthen the paper’s contribution. The authors should focus on adding visual illustrations, refining comparisons between methods, and enhancing the discussion on practical applicability. These changes will ensure the work meets the high standards of TMLR and maximizes its impact on the research community.
Decision: Accept (poster)