Collective Model Intelligence Requires Compatible Specialization

## Abstract

Abstract In this work, we explore the limitations of combining models by averaging intermediate features, referred to as model merging , and propose a new direction for achieving collective model intelligence through what we call compatible specialization . Current methods for model merging, such as parameter and feature averaging, struggle to effectively combine specialized models due to representational divergence during fine-tuning. As models specialize to their individual domains, their internal feature representations become increasingly incompatible, leading to poor performance when attempting to merge them for new tasks. We analyze this phenomenon using centered kernel alignment (CKA) and show that as models specialize, the similarity in their feature space structure diminishes, hindering their capacity for collective use. To address these challenges, we investigate routing-based merging strategies, which offer more flexible methods for combining specialized models by dynamically routing across different layers. This allows us to improve on existing methods by combining features from multiple layers rather than relying on fixed, layer-wise combinations. However, we find that these approaches still face limitations when layers within models are representationally incompatible. Our findings highlight the importance of designing new approaches for model merging that operate on well-defined input and output spaces, similar to how humans communicate
through language rather than intermediate neural activations. Code: github.com/jyopari/compatible_specialization Website: jyopari.github.io/compatible_specialization

## 1 Introduction

Machine learning has recently seen an explosion of models trained for diverse tasks and data and made readily available on platforms such as Hugging Face.
Given a new task, a question faced by practitioners is whether it is possible to reuse and combine information across existing models or whether a base foundation model should be fine-tuned for their application.
We hypothesize that if a new task can be solved by combining existing models – such a collective model may outperform a model obtained by fine-tuning a base foundation model. We refer to the process of leveraging and combining multiple models to solve a target task as model merging (Raffel, ; Ha & Tang, ; Ferreira et al., ).

The idea of collectively utilizing the intelligence of individual entities is commonplace in nature – organisms often come together to form collectives—bee colonies, whale pods, and human societies.
Coordination between individuals (or models) with potentially specialized roles (or functions)
can enhance the capabilities of the whole (Seeley, ; Sumpter, ).
Imagine we need to solve a problem related to disease modeling. Instead of training one person to be an expert in two fields, we may recruit two specialized experts – a mathematician and a biologist – who can work together to solve the given problem. Effective collaboration requires the two experts to share a common language. Regardless of their skills, if one speaks only Hindi and the other speaks only German, their collaboration will be limited. Thus, effective collaboration requires entities that are specialized and can communicate in a shared language – a phenomenon we term compatible specialization.

We investigate state-of-the-art methods for model merging from the lens of compatible specialization and find that current approaches cannot effectively combine specialized models. A prominent way to merge models is to pool features of the input predicted by different models – we refer to this paradigm as representation-based or feature averaging. This includes techniques such as parameter averaging (Wortsman et al., ; Ilharco et al., ; Matena & Raffel, ; Yadav et al., ; Yu et al., ) or computing a weighted average of the individual model features. (Ferreira et al., ; Fedus et al., ; Shazeer et al., ; Jiang et al., ; Yadav et al., ; Sukhbaatar et al., ; Tang et al., ). However, experiments show that feature averaging fails to achieve compatible specialization and is less effective than directly fine-tuning a base model for the target task.

To illustrate our perspective, consider this setup: we finetune a math model and a coding model from a common base foundation model, aiming to merge them for a new task that requires both math and coding skills—such as writing code to solve a math word problem. We now outline two key factors that prevent compatible specialization:

1. Representations diverge during fine-tuning.

To measure how ”compatible” the math and coding models are while fine-tuning, we use a representational divergence metric, $D$. Our experiments reveal a critical point, $t$, where the dynamics of merging these models shift. From the start of fine-tuning up to $t$, as the math and coding models become more specialized in their respective domains, $D$ increases, yet merging still improves performance on tasks that require both math and coding skills, such as writing code to solve a math word problem. However, beyond $t$, further increases in $D$ lead to diminishing returns: merging yields fewer gains, and performance eventually degrades. This critical point $t$ marks the threshold where specialization begins to interfere with compatibility. Thus, merging the math and coding models is effective only when they haven’t become too specialized.

2. Layers become representationally incompatible.

Traditionally, merging is done by fine-tuning models from the same base model, where features from the same depth in each model are combined (Zheng et al., ; Jiang et al., ; Lange et al., ). We refer to layers at the same depth across different models as corresponding layers. However, a more flexible approach would allow for the combination of features from layers at different depths across multiple models, as layers at different depths may represent distinct functional computations. Unfortunately, the aforementioned trade-off between specialization and mergeability is mirrored here. As layers become more specialized, especially when they are positioned further apart in depth, representational divergence increases, limiting their compatibility for merging. Therefore, we cannot achieve compatible specialization across layers.

In essence, for model merging to be effective, models must not only be specialized but also able to communicate their knowledge in a way that facilitates collaboration rather than relying on representational alignment across layers. We argue that representations across finetuned models will remain fundamentally incompatible, even if they originate from a common base model. Therefore, model merging should not be approached as simply combining intermediate representations. Instead, it should focus on enabling models to exchange information, akin to how humans collaborate through language rather than combining their brain activity. This shift in perspective highlights the critical need to develop methods that support compatible specialization. While this paper does not propose a specific solution for achieving compatible specialization, it aims to identify this need and demonstrate that current model merging techniques fall short of it. We offer insights into how future work can move toward realizing true compatible specialization.

Figure: Figure 1: In the left sub-figure, we illustrate how fine-tuning a base model separately on two different tasks initially improves merging performance, but after a critical point, further fine-tuning leads to a decrease in performance. In the right sub-figure, we examine how merging features across different layers in the two models affects merging performance. The diagonal teal color represents cases where layers of the same index/position are merged. As we attempt to merge layers that are progressively further apart in the network, performance begins to plateau. We hypothesize that both phenomena stem from a lack of compatible specialization, where models need to maintain compatibility to be effective in collective use.
Refer to caption: x1.png

## 2 Preliminaries

In this section, we present the main tools we will use. In sub-section 2.1, we describe the general Mixture of Experts formulation and how we employ it for model merging. In sub-section 2.2, we outline the merging methods we compare. In sub-section 2.3, we discuss the tasks used to evaluate the merging methods. Finally, in sub-section 2.4, we explain how we analyze model representations and measure similarity.

### 2.1 Mixture of Experts For Merging

In our work, we investigate model merging based on the Mixture of Experts (MoE) framework (Yadav et al., ). Instead of typically merging models into a single compressed one, we maintain the original models and introduce a router that dynamically weights token features outputted by different experts. Typically, the MLP layer in a transformer (Vaswani, ) is treated as the expert in an MoE (Fedus et al., ).

Let $\{\operatorname{E}_{i}\}_{i=1}^{N}$ denote $N$ experts. For a sequence of $T$ tokens in $\mathbb{R}^{d}$, represented by ${\bm{X}}\in\mathbb{R}^{T\times d}$, the router $R(x):\mathbb{R}^{d}\to\Delta_{N}$ assigns a distribution over the experts for each token $x_{t}\in\mathbb{R}^{d}$.

The output of the MoE layer for token $x_{t}$ is a weighted sum of expert outputs:

$$ $y_{t}=\sum_{i=1}^{N}r_{t,i}\operatorname{E}_{i}(x_{t}),$ $$

where $r_{t,i}$ is the weight assigned by the router to expert $\operatorname{E}_{i}$ (^1^11Note that the router can be sparse and only utilize a subset of experts as well). Thus, for the entire sequence ${\bm{X}}$, the output is $\{y_{t}\}_{t=1}^{T}$.

In our work, the routing function $R(x_{t})$ is a linear transformation. Given an input token $x_{t}\in\mathbb{R}^{d}$, the router is parameterized by a weight matrix ${\bm{W}}\in\mathbb{R}^{N\times d}$. The router output $r_{t}$ is computed as:

$$ $r_{t}=\text{softmax}({\bm{W}}x_{t}),$ $$

We now describe how we utilize the MoE formulation for model merging.
During fine-tuning, we train only the MLP layers to create specialized models. The MLP layers become our “experts”. Additionally, we train the router exclusively on the adaptation dataset. We capture this in Figure 2. Performance is evaluated through auto-regressive next token prediction on a held-out validation set, and we report results using cross-entropy loss (CE Loss).

Figure: Figure 2: We illustrate the general routing based merging pipeline as follows. First, the MLP layers are finetuned from a base model on specialized datasets. Once we obtain a set of specialized models, we construct a Mixture of Experts (MoE) where the experts are the finetuned MLP layers from the various models. Finally, on a novel adaptation dataset, we train only the router.
Refer to caption: x2.png

### 2.2 Different Merging Methods

While merging methods typically are a form of parameter averaging, we find that MoE routing style merging performs consistently better and shift our focus on them. We now describe the different routing schemes we investigate.

- •
Activation Interpolation Instead of interpolating the experts via a router, we have a static coefficient $\alpha\in[0,1]$.
So for layer $l$, $x^{(l)}=\alpha\operatorname{E}_{A}^{(l)}(x^{(l-1)})+(1-\alpha)\operatorname{E}
_{B}^{(l)}(x^{(l-1)})$.
- •
Single Router We place a single trainable router after the first attention layer to produce the routing coefficients $r_{t}$ that are used for weighted averaging of the MLP outputs across all layers.
- •
Full Layer Routing Instead of just having a single router determining the weighting between MLP outputs, we have a separate routing function per layer, more similar to a standard MoE.
- •
Full-Layer Routing with Base Model In our experiments we obtain specialized models by fine-tuning MLP layers from a common base model. Instead of only routing between the finetuned models, we also include the base model as well as another model that can be merged. This tests how routing scales with access to more models.
- •
Multi-Layer Routing We extend MoE merging by allowing the router at layer $l$ to route to experts from different layers. This allows for reuse of experts and more complex routing paths.

### 2.3 Tasks

We now describe the different tasks we evaluate on.

- •
In-Domain We fine-tune a pretrained GPT-2 (Radford et al., ) model using the nanoGPT codebase on two different HuggingFace coding datasets(^2^22nampdn-ai/tiny-codes and TokenBender/code_instructions_122k_alpaca_style). In all fine-tuning, we only update the MLP layers and freeze the attention layers to allow for easy MoE-style routing for merging. We evaluate merging adaptation on two additional coding datasets(^3^33nickrosh/Evol-Instruct-Code-80k-v1 and open-phi/programming_books_llama.).
- •
Cross-Domain We investigate how well merging can be used for cross-domain adaptation. To this end, we fine-tune a pretrained GPT-2 model on a math dataset(^4^44microsoft/orca-math-word-problems-200k.) and a coding dataset(^5^55nampdn-ai/tiny-codes.). We measure merging adaptation on a dataset(^6^66reasoning-machines/gsm-hard.) that requires both math and coding reasoning.

We report results in validation Cross-Entropy (CE) Loss.

### 2.4 Centered Kernel Alignment

We will be utilizing the centered kernel alignment (CKA; Kornblith et al., ) metric to compare the representations of two models. CKA is a well established metric that has been used in numerous analysis works (Raghu et al., ; Lange et al., ; Phang et al., ). It is important to note that the CKA metric is invariant to isotropic scaling, biases, and orthogonal transformations to the representations.

Given two sets of representations, $\mathbb{X}\in\mathbb{R}^{n\times d_{1}}$ and $\mathbb{Y}\in\mathbb{R}^{n\times d_{2}}$.

$$ $\mathbb{H}=\mathbb{I}-\frac{1}{n}\mathbf{1}\mathbf{1}^{\top},\quad\mathbb{ \overline{K}}=\mathbb{H}\mathbb{K}\mathbb{H},\quad\mathbb{\overline{L}}= \mathbb{H}\mathbb{L}\mathbb{H}$ $$

$$ $\displaystyle\text{HSIC}(\mathbf{K},\mathbf{L})$ $\displaystyle=\frac{1}{(n-1)^{2}}\operatorname{tr}(\overline{\mathbf{K}}\, \overline{\mathbf{L}})$ $\displaystyle\text{CKA}(\mathbf{X},\mathbf{Y})$ $\displaystyle=\frac{\text{HSIC}(\mathbf{K},\mathbf{L})}{\sqrt{\text{HSIC}( \mathbf{K},\mathbf{K})\cdot\text{HSIC}(\mathbf{L},\mathbf{L})}}$ $$

where $\mathbb{H}$ is a centering matrix, and $\mathbb{K}_{i,j}=K(x_{i},x_{j})$ and $\mathbb{L}_{i,j}=L(y_{i},y_{j})$ are kernel matrices generated by the kernel functions $K,L$. We use the dot product as the kernel function in our experiments. $\text{HSIC}(\cdot,\cdot)$ is the empirical estimator for the Hilbert-Schmidt Independence Criterion (Gretton et al., ), which was originally developed to measure the statistical independence of random variables.

## 3 Representational Similarity Degrades over Time

As models become increasingly specialized, their internal representations tend to shift towards being more task-specific and less generalizable. We show this is Figure 3, where despite the task specific models become more capable at math and coding respectively, the merging performance on the adaptation task that requires math and coding abilities steadily degrades after initially improving. This posses a trade off where we need to balance specialization with representational compatibility. This is also clearly shown in the middle and right sub-figures in Figure 3 where we can see a “U” shape curve when plotting representational similarity vs merging performance. Therefore, from the start of fine-tuning, as the models acquire specializations, the merging performance improves. However, it appears after a critical step, the representations diverge to the point where further specialization does not contribute to merging performance.

It is import to note that we measure the representational similarity on the adaptation dataset that requires math and coding skills as well as the pretraining dataset OpenWebText (Gokaslan & Cohen, ). Having diverging representations on the pretraining datasets implies that the models have less compatibility. To mitigate this we try injecting the pretraining data into the fine-tuning process to enforce more compatibility, but we did not improve merging performance (see, Appendix A.4). This warrants the need to establish representational structure and compatibility which can mitigate the existence of the aforementioned trade off.

Figure: Figure 3: (Left) Validation cross-entropy loss (CE Loss) for math and coding models during finetuning, as well as the merged models. The math and coding models exhibit steady decreases in validation loss as they specialize on their respective tasks. In contrast, the validation loss of the merged model via activation interpolation on a cross-domain task requiring both math and coding decrease quickly and increase gradually after a critical point. (Middle) Merging loss plotted against CKA similarity computed on data from the adaptation dataset. (Right) Merging loss plotted against CKA similarity computed on data from the pretraining dataset.
Refer to caption: x3.png

In summary, representational compatibility appears to play a crucial role in the success of merging specialized models. As models specialize, their internal representations diverge, making alignment across models over time increasingly difficult. This divergence is particularly evident when merging models that are highly specialized. To improve the effectiveness of model merging in feature space, strategies that promote representational similarity, either through modifications in pretraining or fine-tuning process as well as architectural are essential. In the following section we explore more complex merging methods to see if our current merging practices are limiting performance.

## 4 Merging with More Degrees of Freedom Improves Adaptation Performance

### 4.1 Routing Is More Effective than Static Interpolation

Figure: Figure 4: Performance comparison of various model merging techniques for In-Domain and Cross-Domain tasks. The plot shows the progression of different merging methods, from simple interpolation strategies, (SLERP, LERP, activation interpolation) see A.1, to more complex ones involving routers (Single Router, Full Router, Routing with Base Model). The trend demonstrates that increasing the complexity and capacity for model merging results in performance gains, as reflected by the lower adaptation loss.
Refer to caption: x4.png

Figure: Figure 5: Comparison three routing strategies in model merging: Standard, 2-Layer, and 3-Layer Routing (see Figure 6). Evaluated on two in-domain tasks and one cross-domain task, results show that increased routing complexity reduces CE loss across all tasks. 2-Layer Routing achieves notable gains over standard routing, with 3-Layer Routing offering further, minor improvements.
Refer to caption: x5.png

We provide a characterization of different merging methods based on their degrees of freedom which describes how large their search space is. For example, when we do a simple interpolation of parameters where $\theta_{\text{merge}}=\alpha\theta_{A}+(1-\alpha)\theta_{B}$, we are searching along a one dimensional subspace. Searching the right subspace has shown to be important (Wortsman et al., ). Pushing this direction, we focus our attention on routing methods, where each router can explore an interpolation between two or more experts conditioned on the current input or token. By introducing routers we can dynamically combine information between layers from different models, thus increasing our search space for merging.

As shown in Figure 4, in all tasks as the complexity of the merging method increases in degrees of freedom, we observe an improvement in the adaptation performance. We see that merging scales with more routers and models. To push current methods we will describe in the following section how we increased the search space for routing based merging.

Figure: Figure 6: Visual representation of different routing strategies for model merging. From left to right: (1) Standard Routing performs layer-wise merging between corresponding layers from models A and B using a weighted average. (2) Routing to 2 Layers expands the merging process by incorporating not only the corresponding layers but also the next layer, allowing the router to combine outputs from the current layer and the layer above. (3) Routing to 3 Layers extends this further by merging the current layer, the layer above, and two layers above, enabling more complex
Refer to caption: x6.png

### 4.2 Multi-Layer Routing improves Merging Performance

Based on the aforementioned trend we further increase the search space by allowing the router at layer $l$ of the MoE to route to expert MLPs at different layers than $l$ across different models. Typically in an MoE model, experts will come from the same layer $l$ in the specialized models. This allows experts to be reused across layers and consequently it broadens the search space. We visually describe what this means in Figure 6.

Figure: Figure 7: CKA representational similarity analysis for MLP layers in a cross-domain math and coding task, comparing layer outputs across models. The top row shows comparisons based on the adaptation dataset, while the bottom row shows comparisons based on the pretraining dataset. The first two columns depict intra-model comparisons, illustrating the self-similarity of representations within the same model across different layers. These plots show that adjacent layers exhibit higher representational similarity, whereas layers farther apart have significantly lower similarity. The third column shows inter-model comparisons, reflecting the similarity of corresponding layers between the math and coding models. Layers in distant positions, demonstrate lower representational alignment.
Refer to caption: x7.png

However, there is a plateau in performance across all tasks as shown in Figure 5 when increasing the number of layers the router can route to. This implies that unless there exists a better way to efficiently decompose the model, the current method of MoE routing style merging is limited. Ideally, we would want to be able to create a more complex routing scheme that uses different blocks across different locations across different models, instead of being locally limited. In the following section we investigate this phenomenon through a representational lens.

### 4.3 The Challenges of Layer Wise Representational Incompatibility

When the performance in Figure 5 plateaus, we find that is is correlated with representational similarity between layers within a model and across models. We specifically analyzed this in the cross domain experiment as shown in Figure 7. We pass a batch from the adaptation dataset and the pretraining dataset into both the math and coding finetuned models, and measure the representational similarity between different layers. It is apparent that within a model (Intra), layers adjacent to each other produce the most similar representations. In addition, even across the finetuned models (Inter) there is highest representational similarity around similar relative positions in the network. This suggests that layers with high representational dissimilarity can not have their outputs combined in a straightforward way.

Consequently, we are limited to routing experts that come from similar relative positions in the network. To mitigate this one would have to change the architecture, or how the models are pretrained or fintuned to ensure representationally compatibility across different layers in the network.

## 5 Current Limitations and Perspectives on Future Directions

While we were able to extend merging performance via more complex routing schemes, we were limited by representational incompatibility. Furthermore, when we see how merging in general compares to directly fine-tuning, as shown in Table 1, we can see across our tasks fine-tuning the base model on the adaptation task consistently outperforms routing based merging which is our most competent merging method. Therefore, our current methods for merging are not effective enough for this to be adopted in practice. To this end, we propose future directions to improve model merging that we believe can lead to a decentralized collective intelligence (Raffel, ).

**Table 1: Loss comparison of merging and fine-tuning methods on the cross-domain task.**
| Method | Cross-Domain | In-Domain 1 | In-Domain 2 |
| --- | --- | --- | --- |
| Merging | 1.17 | 1.10 | 1.36 |
| Fine-tuning | 1.04 | 0.91 | 1.01 |

### 5.1 How Should we Address Representational Compatibility?

We have seen successful examples where we are able to combine models to solve a novel task. Take for example stitching vision language models such as LLaVA, (Liu et al., ; Li et al., ), where they learned a simple adapter to map vision tokens to language tokens. Adapters are a powerful tool that does allow to combine models if there representations have some similar underlying structure that differs by a simple transformation (Yang et al., ; Sung et al., ; Yang et al., ). However in most of these settings there are two models and the overall computational graph is hand designed, making it easy to inject adapters and tune them. In the paradigm where we have a collection of models and at test time we want to be able to route them dynamically, it would require having an adapter for every possible combination. This clearly will not scale well. Hence, having models directly being representationally compatible both internally and across models appears critical if we want feature space merging. There have been works to improve layer wise compatibility within models (Jiang et al., ), yet we still have no clear way to ensure representations are compatible across models as they specialize during the fine-tuning process. To this end we discuss an alternative direction to address these concerns.

### 5.2 Routing Models in their Input and Output Spaces

Feature space based merging has no clear way to enforce structure to obtain compatibility across models or layers. When we look at other forms of collective intelligence like open source code which is an analogy referred in (Raffel, ), we care about the inputs and outputs of a function. When access to a collection of libraries available on GitHub, when we solve a new task we do not try to figure out how to stitch various internals of different libraries which would be akin to the idea of feature space based model merging. Therefore, we propose to have specialized models that operate on a common space, such as language.

### 5.3 Problems with Routing

The current router formulation used in MoE models are fixed in the number of experts they can route to, and it requires a smooth monotonic interpolation of the loss w.r.t. expert outputs for it to learn the optimal weighting.

When a programmer decides which library to use in their code, they typically read the description of the library API and then determine if and how they should integrate it in their pipeline. We need routers to behave in a similar manner where they can actively search and retrieve models based on the task at hand, which will likely call for optimizing with Reinforcement Learning (RL). In addition, models need to be well characterized to describe their functionality to the router (Lee et al., ). This can manifest as humans giving a description of their specialized model and examples of inputs / ouputs when uploading it to a collective hub.

## Acknowledgments

We would like to extend our deepest gratitude to Minyoung Huh, who was invaluable throughout the entire project, providing many of the key ideas while he was a student at MIT. We are also grateful to Han Guo, Brian Cheung, Idan Shenfeld, Seungwook Han, Leshem Choshen, Shivam Duggal, Akarsh Kumar, Yoon Kim, Jeremy Bernstein, Tongzhou Wang, Lucas Hennigen, Adrian Munoz, Moritz Reuss, Anurag Ajay, Zhang-Wei Hong, and the members of the Improbable AI Lab for their useful feedback on the idea and manuscript. The research was sponsored by the Army Research Office and was accomplished under ARO MURI Grant Number W911NF-23-1-0277. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.

## Author Contributions

- •
Jyothish Pari Led the project, conceived the idea, designed and implemented the experiments, and wrote the manuscript.
- •
Samy Jelassi Provided guidance on the experimental design and organization of the paper.
- •
Pulkit Agrawal Oversaw the project, assisted with positioning the paper, and contributed to the writing and presentation of the results.

## Appendix A Appendix

### A.1 Weight Interpolation Methods

Linear Interpolation (LERP) Given two models’ weights $\theta_{A},\theta_{B}$, we search along the one dimensional subspace: $\alpha\theta_{A}+(1-\alpha)\theta_{B}$, $\alpha\in[0,1]$.

Spherical Interpolation (SLERP) One potential issue with linear interpolation is that if the normalized dot product between two flattened weights is close to $-1$, then a linear interpolation can result in a low norm weight. To mitigate this effect, we perform spherical interpolation of the weights, where we search along the arc. Let $\mathbf{v}_{0},\mathbf{v}_{1}$ be two flattened weight vectors, we switch weight notation from $\theta$ to $\mathbf{v}$ avoid confusion when referring to angles between weights.

$$ $\text{slerp}(\alpha,\mathbf{v}_{0},\mathbf{v}_{1})=\frac{\sin((1-t)\theta)}{ \sin(\theta)}\mathbf{v}_{0}+\frac{\sin(t\theta)}{\sin(\theta)}\mathbf{v}_{1}$ $$

where
$\theta=\cos^{-1}(\mathbf{v}_{0}^{\top}\mathbf{v}_{1})$.
If the vectors are nearly colinear (i.e., the dot product is close to 1), linear interpolation (LERP) is used:

### A.2 Architecture Details

**Table 2: Model sizes, architectures, and optimization hyper-parameters for fine-tuning GPT2**
| params | dimension | $n$ heads | $n$ layers | learning rate | batch size | $n$ tokens |
| --- | --- | --- | --- | --- | --- | --- |
| 124M | 768 | 12 | 12 | $8.0\times 10^{-4}$ | 64 | 160k |

We use a learning rate of $8\times 10^{-5}$ for the fine-tuning results in Table 1

### A.3 In Domain Layer-Wise Similarity

See Figure 8 for the layer wise representational analysis between two coding models across two different coding adaptation tasks.

Figure: Figure 8: In the in-domain tasks, we plot the CKA representational distance between different MLP layer outputs for the same batch of inputs. The first two columns show the self-similarity of representations within the same model at different layers, illustrating how adjacent layers are more representationally similar. The right column shows the similarity between the representations of corresponding layers from two different models.
Refer to caption: x8.png

### A.4 Routing Ablations

**Table 3: Cross Domain Performance with different ablations. DataMix Routing refers to routing a math and coding policy but the math and coding models were co-finetuned on the pretraining data. 2 Layer MLP refers to a router being a 2 layer MLP to test the effects of having more routing capcity.**
|  | Cross-Domain CE Loss |
| --- | --- |
| DataMix Routing | 1.35 |
| 2 Layer MLP Router | 1.28 |
| Standard Routing | 1.30 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - The issue of merging task-specialized pretrained models has been explored before; therefore, it will be interesting to compare and contrast the presented _compatible specialization_ with:
    - $$L^2-SP$$: this method represents a preliminary fine-tuning-based approach, where regularization is added to retain pretrained knowledge: https://arxiv.org/abs/1802.01483;
    - The question of whether two models are compatible, with each other, has already been studied exhaustively, through the lens of _linear mode connectivity_ (LMC). For instance, see: https://arxiv.org/abs/2110.06296.
-  Section 2.4 introduces CKA, yet it seems that a bunch of equations were just thrown to the reader without the needed context. I suggest revising this section by somewhat explaining step-by-step what CKA is doing.
- What is the difference between _activation interpolation_ (see Section 2.2) and averaging weights with varying coefficients?
soundness: 2
strengths: - Insightful Identification of Compatibility Issues: The paper highlights that representational divergence during fine-tuning can hinder model merging efforts, which is a fundamental issue when merging any kind of machine learning models; hence the paper is well motivated.
- The paper is generally well-written and offers useful analogies for the reader to ground his or her intuitions correctly.
    - The reader can smoothly go over the paper, as academic language develops every concept with eloquence.
- Experiments are conducted on both, in-domain and out-of-distribution settings, which are typical scenarios to assess knowledge transferability between models.
- There is no discussion on time and space complexity caveats.
summary: In this paper, the central argument arises as the authors highlight how fine-tuning models may hinder their parameter-wise combinations (i.e., model merging). The authors present empirical evidence (via a CKA similarity measurement) on how specializing models, via transfer learning, cause their representation to diverge; hence, feature averaging algorithms will tend to be suboptimal, for merging models in this setting.

The authors claim that it is crucial for models to communicate their (layer-wise) learned knowledge; they go on to state that feature averaging and representational alignment methods do not address this problem. As an alternative, they propose a _mixture of experts_ (MoE) mechanism, focusing entirely on adequate routing. Overall, given two (pretrained) models, their method proposes to treat certain layers as experts and route them, instead of combining them with any interpolation coefficient. This method is, allegedly, flexible to support merging different architectures.

Experiments show that while the proposed routing-based merging approach does improve upon basic feature averaging methods, it still faces substantial limitations. Specifically, the authors observe that as models become more specialized through fine-tuning on distinct tasks (e.g., math vs. coding), the merged model’s performance on a cross-domain adaptation task initially improves but subsequently declines as representational divergence increases. The experiments also demonstrate that routing methods with more degrees of freedom—such as multi-layer and cross-layer routing—yield better performance than single-layer static interpolation, particularly when the models share similar internal structures. However, even these sophisticated routing techniques struggle when the internal layer representations of the models are highly dissimilar, ultimately reaching a performance plateau.
weaknesses: - Arguing that models require **compatible specialization** is central to the purposes of this paper. Yet the concept remains ill-defined after the methodology section, which brings certain issues such as:
    - no direct/intuitive connection between routing mechanisms and a neural model's _ability to communicate its knowledge_;
    - lack of literature review on past methods that could, at least, somehow contribute to enhancing compatible specialization;
- While Figure 6 showcases how layer incompatibility increases between more distant layers, no concrete actions were taken to improve the experimental decisions; in particular, only conjectures were made to handle similar relative positions between models, without explicitly modifying their method.
- There are no experimental ablations against past model merging techniques, rather than just comparing the routing with feature averaging.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. **Practical Implications of Compatible Specialization**: Beyond the discussions provided in the paper, how do the authors envision compatible specialization being achieved in practice? Are there potential architectures or training methods that might enhance inter-model compatibility?
2. **Routing Complexity Trade-offs**: The results in Figure 5 suggest diminishing returns as routing complexity increases. Could the authors elaborate on potential architectural modifications or efficiency improvements to manage the trade-offs of increased routing complexity?
### Minor Suggestions
- **Line 343 (Figure 4 caption)**: The caption appears incomplete. Consider rephrasing to finish the sentence describing the complexity of routing to different layers.
- **Lin 411**: It is written " we find that is is correlated". There is one too many "is" here.
- **Consider Additional Citations**: Adding citations to studies that critique the use of CKA in representational analysis would provide a balanced view and strengthen the experimental conclusions.
soundness: 2
strengths: - **Originality**: This paper introduces a fresh perspective on model merging by emphasizing the importance of compatible specialization rather than direct feature merging. The application of centered kernel alignment (CKA) in this context is a novel approach that strengthens the paper’s analysis and arguments.
  
- **Clarity**: The paper is well-organized, with clear sections and helpful diagrams (e.g., Figures 1 and 4) that effectively illustrate the merging processes and highlight the limitations of current methods.

- **Significance**: By addressing the challenges of model merging and outlining future directions, this work provides valuable insights for advancing collective model intelligence. This is increasingly significant as fine-tuned, domain-specific models continue to proliferate. The paper’s emphasis on both computational efficiency and compatibility offers a meaningful contribution to the field.
summary: This paper investigates the challenges of merging specialized neural models, focusing on issues of representational similarity and compatibility. It highlights that common merging techniques, such as feature averaging, often struggle to produce effective merged models as the individual models become more specialized. This specialization leads to divergent feature representations, which limit the models' ability to combine effectively. The authors propose exploring dynamic routing-based strategies, where features are merged across multiple layers instead of using static, layer-by-layer averaging. Their empirical analysis, including the use of centered kernel alignment (CKA) metrics, underscores the limitations of current merging methods and emphasizes the need for new approaches that prioritize compatible specialization. This work signals the importance of inter-model compatibility in the development of effective model merging techniques.
weaknesses: - **Quality**: The paper’s experimental scope is limited, and expanding it to include a wider range of tasks, such as those in SuperGLUE [[1](https://arxiv.org/abs/1905.00537)], would strengthen its findings. Additionally, comparisons with other benchmark model merging of Mixture of Experts (MoE) methods, such as those presented in recent works [[2](https://arxiv.org/pdf/2406.09770), [3](https://arxiv.org/abs/2402.00433), [4](https://arxiv.org/pdf/2402.05859), and [5](https://arxiv.org/pdf/2306.03745)], would help contextualize the contributions and insights of this paper within the broader field.

- **Experimental Validation of CKA Insights**: The paper uses CKA analysis to examine representational similarity (e.g., in Figures 2 and 6). However, recent works have raised concerns about the reliability of CKA in drawing conclusions based on its similarity values, suggesting that it can sometimes produce misleading results in neural network similarity analyses [[6](https://arxiv.org/pdf/2210.16156), [7](https://arxiv.org/pdf/2405.01012), [8](https://proceedings.neurips.cc/paper_files/paper/2022/hash/2f0435cffef91068ced08d7c7d8e643e-Abstract-Conference.html)]. Given that CKA’s benefits depend on observing specific conditions (discussed in aforementioned works), further investigation into its validity in this context, or the inclusion of additional similarity metrics, would lend greater robustness to the findings.

- **Findings of Section 4.3**: Section 4.3 begins with the statement:
  > “When the performance in Figure 5 plateaus, we find that this is correlated with representational similarity between layers within a model and across models.”

  However, the analysis in Figure 6 does not conclusively support a correlation between the performance plateaus observed in Figure 5 and representational similarity among adjacent layers. As Figure 6 shows only a single snapshot (likely for the "Routing with 3 Layers" setting, please clarify if it is not the case as it is not mentioned in the text), observing CKA changes from "Standard Routing" to "Routing with 2 Layers" to "Routing with 3 Layers" would be necessary to conclude a correlation. If CKA values indeed increase between nearby layers across these configurations, then a correlation might be established. The current results as it stands only reconfirm previous observations that adjacent layers exhibit high CKA similarity, as shown in earlier studies [[9](https://arxiv.org/pdf/2010.15327) and [10](https://arxiv.org/abs/2202.07184)].

- **Scope of the Vision for Collective Intelligence**: Although the paper’s proposal for collective intelligence in Section 5.2 (Routing Models in Input and Output Spaces) is intriguing, its practical impact is limited by a lack of concrete steps or preliminary experiments to advance this vision.

---

### References

1. [Wang, A., et al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://arxiv.org/abs/1905.00537)

2. [Tang, A., et al. Merging Multi-Task Models via Weight-Ensembling Mixture of Experts](https://arxiv.org/pdf/2406.09770)

3. [Jiang, A. Q., et al. Mixtral of Experts](https://arxiv.org/abs/2402.00433)

4. [Yang, L., et al. Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models](https://arxiv.org/pdf/2402.05859)

5. [Zheng, Y., et al. Layer-Wise Representation Fusion for Compositional Generalization](https://arxiv.org/pdf/2306.03745)

6. [D’Ascoli, S., et al. The False Promise of the Similarity Hypothesis in Neural Networks](https://arxiv.org/pdf/2210.16156)

7. [Kim, M., et al. Pitfalls of Measuring Neural Network Similarity with Centered Kernel Alignment](https://arxiv.org/pdf/2405.01012)

8. [Doimo, D., et al. Interpreting CKA as Similarity in Representations: The Importance of Calibration](https://proceedings.neurips.cc/paper_files/paper/2022/hash/2f0435cffef91068ced08d7c7d8e643e-Abstract-Conference.html)

9. [Raghu, M., et al. Do Vision Transformers See Like Convolutional Neural Networks?](https://arxiv.org/pdf/2010.15327)

10. [Kornblith, S., et al. Similarity of Neural Network Representations Revisited](https://arxiv.org/abs/2202.07184)

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: I'm curious that what would the CKA pattern look like for an MoE-merged model compared to an interpolation-based merged model? 

Specifically:
How do CKA values between the MoE model and the original models (A or B) differ from those seen in interpolation-based merging?
Further analysis on these patterns could reveal additional insights into the advantages of MoE merging over parameter interpolation, especially in cross-domain contexts.
soundness: 3
strengths: The proposed MoE-based approach introduces a fresh perspective to the challenge of model merging by routing between domain-specific models. This strategy could address issues with prior methods that rely on simple parameter interpolation, particularly when models are poorly aligned in terms of CKA.
summary: Previous works use parameter interpolation for merging models from two domains, which may fail if the CKA (Centered Kernel Alignment) between the two models is too large. This paper proposes using a Mixture of Experts (MoE) approach that routes between two models as a new model merging strategy.
weaknesses: 1. The paper only uses cross-entropy (CE) loss to evaluate the merged model, which may be insufficient to capture the model’s true performance, particularly in language modeling tasks. CE loss alone may not fully reflect the merged model’s language modeling and generation quality. Following guidelines from prior work ([1]), it would be beneficial to include additional metrics that measure both the intrinsic quality of the model (e.g., perplexity and accuracy) and the quality of generated text (e.g., diversity and coherence).

2. Since the MoE approach increases the model’s parameter count (potentially doubling it compared to individual models), the observed reduction in loss may be due in part to this increase in model size rather than the efficacy of the MoE merging itself. Interpolation of parameters, by contrast, maintains the original model size, which could lead to an uneven comparison. It would be more informative to compare models of similar sizes to isolate the effectiveness of the MoE merging strategy.

3. While the paper claims that a larger CKA divergence between models degrades the performance of interpolation-based merging, it lacks experimental results to support this claim. Testing model pairs with varying CKA levels (e.g., comparing interpolation results between a GPT-math and GPT-code model versus two similar GPT-code models) would strengthen the case for MoE merging as a preferable alternative. For example, additional experiments can be conduct to verify that when $CKA(GPT_{math}, GPT_{code}) > CKA(GPT_{code1}, GPT_{code2})$, then $Loss(\alpha GPT_{math} + (1 - \alpha)GPT_{code}) > Loss(\alpha GPT_{code1} + (1 - \alpha)GPT_{code2})$. 


[1] Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L., & Collier, N. (2022). A contrastive framework for neural text generation. *Advances in Neural Information Processing Systems*, *35*, 21548-21561.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - Why are results reported in validation cross-entropy loss?
- In response to "Hence, having models directly being representationally compatible both internally and across models appears critical if we want feature space merging. There have been works to improve layer wise compatibility within models (Jiang et al., 2024b), yet we still have no clear way to ensure representations are compatible across models as they specialize during the finetuning process.", what are the trade-offs? Would this compatibility perhaps be at the cost of individual specialisation performance?
- Why are math and coding chosen? One may argue that both tasks have more in common than other tasks.
- Can you list the number of parameters in addition to performance in Table 1? Am I correct in stating that fine-tuning method has less than half of the parameters of the merging method?
soundness: 3
strengths: - The analyses conducted in the paper involving multiple tasks and representation similarity is insightful, motivates future work.
summary: This paper studies model merging focusing on compatible specialization. The paper points out that as models specialise, the similarity of their internal representations, as measured by centered kernel alignment (CKA), diminishes, and therefore, make it harder to merge with other specialised models. The paper investigates model merging via dynamic routing across different layers but reports limitations to such methods as well.
weaknesses: - The experimental design of the study is somewhat weak:
-- The only routing function R studied in this work is a linear transformation which may explain the plateau reported in Fig.5.
-- The paper reports a plateau in Fig. 5 using only 3 points (standard, 2 layers, and 3 layers) but it is unconvincing to report a plateau using only 3 points.
-- Results are reported in validation cross-entropy loss, which is a disadvantage, as that may not necessarily translate to merging of expertise; one or more well defined benchmarks should be chosen that require multiple specializations, to allow more meaningful and comparable results.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. CKA(Centered Kernel Alignment) does not appear to be a commonly used metric for measuring feature distance.  How does it differ from the more commonly used metrics, such as cosine similarity and PCA-based methods?
2. As training progresses, the disparities between expert models become more pronounced. However, this not only poses the drawback of increasingly difficult communication among experts but also offers the benefit of further enhancing the experts' specialized capabilities. How does the author perceive this balance?
soundness: 2
strengths: Pros:
1. Investigating the impact of similarity among expert models in MoE on merging performance is a valuable direction.
2. The concept of cross-layer routing is intriguing and holds potential for developing new methodologies.
summary: This paper explores certain phenomena related to feature merging within the MoE (Mixture of Experts) architecture. The authors have conducted validation experiments on the GPT-2 model using mathematical and code datasets, arriving at two primary conclusions: 1) As the expert models are trained, the decreasing similarity among expert models affects the performance of merging in MoE architecture. 2) Granting the router more freedom (enabling it to route more MLPs) within the MoE architecture improves the performance of the MoE model.
weaknesses: 1. Novelty: The conclusions drawn are not particularly surprising, as similar and more in-depth discussions in MoE research [1,2,3] already exist, especially concerning the form of the router. Additionally, their discussions often relate to practical models or applications, whereas this paper merely performs validation, leaving the content somewhat lacking. I strongly encourage the authors to advance their research further, particularly with the cross-layer router.
2. Scope of the paper: The experiments conducted by the authors are based solely on the MoE architecture for GPT, focusing on feature merging. There is a significant gap between this and another major area of model merging: parameter merging. Moreover, feature merging encompasses many methods not related to MoE. Whether the conclusions drawn from MoE can be generalized to methods across the entire field of model merging is debatable. I suggest the authors clearly specify their focus on MoE in the abstract and introduction to avoid potential misunderstandings.
3. Experimental settings: The use of a relatively small GPT-2 model trained on mathematical and code tasks with SFT, and using validation loss as a metric, is not a common setting. Given the small scale, code and mathematical tasks are particularly challenging for the GPT-2 model, especially with SFT training, which raises concerns about potentially adverse effects on the experimental results. I recommend the authors consider using more traditional, simpler tasks or a larger-scale model.
Typos:
There is a conspicuous error in Figure 3.

[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
[2] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
[3] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Meta-review: Reviewers appreciated the papers emphasis on the importance of compatible specialization rather than direct feature merging. Reviewers raised concerns about the scope and generalizability of the experiments (e.g., using only small-scale GPT-2 models on math and coding tasks), the reliance solely on validation cross-entropy loss as a performance metric, and the absence of comparisons with a broader range of model merging techniques. They also questioned the use of CKA as the sole similarity measure and pointed out that the findings, while interesting, lack concrete steps toward practical implementations or more comprehensive experimental evaluations. Overall, the reviewers see value in the direction but remain unconvinced that the current work fully justifies its claims or provides robust evidence.

Decision: Reject