Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Authors: Pierre Ablin
Angelos Katharopoulos
Skyler Seto
David Grangier
Venue: ICML 2025 poster (2025)
Keywords: Pre-training
Specialization
Domain Adaptation
small models
Primary area: general_machine_learning->scalable_algorithms
TL;DR: We propose a new architecture that, given a specialist dataset,  is able to quickly instantiate a small specialist model that is good on that dataset.

Abstract: Machine learning models are routinely trained on a mixture of different data domains. 
Different domain weights yield very different downstream performances.
We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. 
Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. 
We learn the linear combination coefficients as a function of the input domain weights.
To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights.
We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly.
Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a size constraint.

Soup-of-Experts:
Pretraining Specialist Models via Parameters Averaging
Pierre Ablin' 1 Angelos Katharopoulos'
Skyler Seto David Grangier'
Abstract
or be small company's internal document database. It is, therefore, impossible to train good-quality specialist model on such data alone.
Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Expers novel architecture that can instantiate model at test time for any domain weights with mini- mal computational cost and without re-traing re-training the model. Our architecture consists of bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear com- bination coefficients domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Exper appealing when one needs to ship many different specialist models quickly under a size constraint.
To obtain a small model that performs well on the special- ization data, we leverage a large, generic pretraining dataset. That pre-training set contains data from several domains. A powerful method obtain a good specialist model is importance sampling: it adjusts the mixture weights of the pretraining distribution to resemble the scarce specialist dataset. This method has been shown to outperform generic pre-training (Grangier et al., 2024b), but it has a major drawback: it requires pre-training a full model for each spe- cialization dataset available. This makes training cost scale linearly with the number of specialized downstream tasks, which can be intractable as model size and data scales.
function of the input
The goal of this paper is to answer the following question: How can we leverage a large pre-training set to obtain specialized models that can be instantiated quickly when the specialization data is revealed?
are particularly
We formalize this question by considering the two phases of serving specialist models.
Pretraining We have multiple pre-trate pre-rriniig domains and use them to train model. At this point, we do not know the specific data and are unaware of what specific tasks we will need to address later on.
1. Introduction
Large Language Models (LLMs) work well on diverse tasks because they have many parameters and are trained on gen- eralist datasets (Brown et al., 2020; Bommasani et al., 2021). However, they are costly to train and to serve, both in terms of memory and inference cost.
Specialization phase We receive specific dataset, and using the pre-trained model, we need to quickly instantiate small model that works well on this specific dataset.
Specialist language models hold fewer parameters; they are, therefore, cheaper to store, send, and use at inference. However, they must give up the generality of LLMs and specialize in a few specific topics.
In Table 1, we summarize the different costs and constraints associated with these two phases and provide qualitative review of the strengths and weaknesses of several strategies.
In this landscape of different models, we introduce the Soup- of- of-Exxerts, which is designed be able to instantiate a small specialist model in flash.
In cases where there is an abundance of specialization data, training a small model on those data yields good specialist. However, many settings, the specialization data is scarce:
Our main idea is to learn to instantiate models with any mixture of domain weights by taking linear combination of jointly optimized base models, called experts. We are inspired by the works of model merging (Wortsman et al., 2022; Arpit et al., 2022; Rame et al., 2022; 2023; 2024). The gist of model merging is that two model parameters θ and O that are obtained by fine-tuning the same model
for instance, it may come from narrow topic of interest
Apple.
Correspondence
to:
Pierre Ablin
<p.ablin @ aom>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Soup of Experts
Model
Spec. size
Pretrain. size
Pretrain. cost
Spec. Cost
Spec. Latency
Spec. Loss
Large generic model
Large
Large
Large
Null
Large
Small
Mixture of Experts
Large
Large
Small
Null
Small
Med
Small generic model
Small
Small
Small
Null
Small
Large
Domain Experts
Small
Large
Med
Small
Small
Med
CRISP Soup-of-Experts
Small
Null
Null
Large
Small
Small
Small
Large
Small
Small
Small
Small
Table 1. The different quantitics that matter during the phases of serving a specialized model. Spec. size is the number of parameters the specialized model. Pretrain. size is the total number parameters of the pretrained model. Pretrain. cost the cost of pretraining the model. Spec, cost is the cost to obtain specialized model when the specialized data is made available. Spec. latency is the cost of performing inference with that model. Spee. loss is the loss on the specialized dataset. With these constraints in mind, we compare different models. The goal of this work model size. Large generic model is generalist model, with many parameters, that requires a long training. A mixture of Experts (Fedus al., 2022a; Krajewski al., 2024) is small model with added parameters that marginally impact the latency. Since both the LLM and the MoE have many parameters, they are discarded from our study. Small generic model is one small model trained on a generalist distribution. Domain experts (Gross et al., 2017) train one small model per pre-training domain. CRISP (Grangier et al., 2024b) trains one model once the specialized data is available using a data mixture that imitates the specialized data distribution. Our proposed method, the Soup-of-Experts, trains one model with many parameters and can quickly instantiate small model that is good on the specialized data. The results in this table are qualitative.
propose the best possible model under the constraint of having small specialized
on different domains A and B can be merged by averaging, yielding new model O" 1 A θB), sometimes called model soup (Wortsman et al., 2022), to obtain good performances on both datasets. An important lesson from model merging is that some models' parameters can be linearly combined and yield good models.
Algorithm 1 Sampling from mix(h) Σ D;D;
Input: Domains Di,..., DR. domain weights h E R batch size B
for b = I..., do
Sample an index i ~ Categorical())
Sample vh, uniformly at random from domain D; end for
A caveat of model merging is that the merged models can only be fine-tuned versions of the same base model: for merging to work, the two models must not be too far apart in the parameters space. Our method, pre-trains multiple experts that can, by design, be linearly combined to yield a single specialized model. The linear coefficients of the combination are learned as function of the pre-training domain weights. Figure gives an overview of the architecture and the training pipeline.
Output: Mini batch [x1,...,
2.1. Sampling from the pre-training set
The pre-training set is composed of domains Di,..., DR X where X is the sample space (in the case of LLMs, this is the space of text sequences). Each domain contains many samples, usually enough to train model without repeating data or overfitting. We can query samples from each of these domains, therefore we can sample from a weighted mixture of domains: for some domain weights h E R* we define the sampling law mix(h) => h,D, such that
Paper overview In Section 2, we explain the details of the Soup-of-Experts, its training pipeline, and how it can be used to instantiate a specialist model quickly. In Section 3, we demonstrate the promises of this approach in a standard language model pre-training setup, where we train small 10M models on Redpajamav2 (Weber et al., 2024) and specialize them on 16 domains from the Pile (Gao et al., 2020). We conduct several analyses to clarify the roles of model size, number of experts, training distribution, and specialized dataset size. Finally, in Section 4, we position our work within the literature.
P(a|mix (h) = Σ P.)
(1)
This law mixes the datasets D; with proportions h; where the domain weights are non-negative and sum to one. We can efficiently query samples from mix(h) for any domain weights h, by picking domain at random following the categorical law induced by h. and then sampling an element at random from the corresponding domain D, The corre- sponding law is illustrated in Figure 2, and this strategy is described in Algorithm 1.
2. Methods
Figure 1 gives overview of the proposed architecture, its interplay with data, and its training pipeline. We first explain the training data setup.
Classical generic pre-training uses fixed pre-training domain
2

---

Soup of Experts
Model
S
Domain weights h
D1
D3 m
Sample domain
D2
C1
E
weights h ~ π
Da
MLP
On x
En
h1 h2 h3 h4
X
h h2
h2
Loss @ xt)
Pretraining domains
Figure 2. Data mixture sampling Given several pretraining do- mains Di..... DE. an domain weights hi,..., lu. we can train a model on the mixture mix(h) = Ei= A,D., using the sampling procedure described in Algorithm Domain weights have a criti- cal impact on the downstrear performance.
Sample data mix(h)
Data D1
D2
Dk
Figure 1. The -xpeerr---xtssssseee and its training pipeline. The Soup-of-Experts consists of shared parameters S, n experts param- eters Ei.... En. and an MLP that acts as a routing mechanism. At each optimization step. we sample domain weights from a meta-distribution т. These domain weights have two purposes: they are passed through an ML.P to give a vector of coefficients a that instantiates model by combining the experts" weights, and they are used to sample mini-batch of data following the domain weights law. We then backpropagate through the corresponding loss to update the parameters of the Soup-of-Experts
find domain weights hape such that mix(Aspe) = Depe and then ii) train the model by minimizing L(O, hepe). This saaprtanc-samlling---he
method called CRISP gives much better specialists than generic pre-training since trains the model on a distribution that has lots of data and yet close to the targeted specific distribution.
One caveat of this approach is that it requires retraining model from scratch anytime one wants to obtain a spe- cialized model. While this cost might be justified in some critical applications, we study alternative avenues to obtain specialized models at a much smaller cost: this is the pur- pose of the new architecture that we propose in this paper, the
weights Ageneric which define a generic dataset
D Dgeneric- mix mix((hgeneric).
These weights are defined to train large generalist models that perform well on average. Finding weights for a good average behaviour to train large models difficult (Xie et al., 2023). For smaller models, even a good mix(hheneric) would yield a model far from strong specialists, i.e., giving model good at everything but excellent at nothing.
2.3. Soup-of-Experts
The goal of the -f-p-erettt is amortize the training of models multiple different domain weights. It de- fines method that, given training domain weights h R quickly instantiates model θ that depends on those domain weights and that yields a low loss h)),
2.2. Training with mixtures of pre-training domains
To do so, we enhance the base model with n experts Ei...., E,, E R?, which for ease of notation we stack into matrix E = [Ei...., En] E Rux,, We linearly combine the weights with shared parameters S E R., For given set of expert coefficients R", we instantiate small model as
We let θ E E2" the parameters of a model to be trained on the pre-training set. We define (() the loss function for sample E X (the next token prediction loss in this paper, since we focus on language modeling). The standard LLM pretraining consists of running Adam (Kingma, 2014) to approximately minimize the generic loss
n
= Comommo (S, a, = S + Σ Eg
(2)
j=1
Lgenerie ( ( EEz Decnasic [)();a))
Our main idea is to learn coefficients a as a function of the domain weights h. To be more precise, we want learn parameters S, E, and a function : R$ R" such that, for any domain weights h E R* the instantiated model θ = Combine S, E, d(h)) performs well on the dataset mix(h), i.e., leads to a low loss L(O, h). In practice, we use a two-layer MLP for & parameterized by parameters w. denoted as Φh.. Although one can think of many different
Alternatively, we can train a model on any given mixture with domain weights h by running Adam on the loss
L(O,h) h) = Ez-mix(A) [()
Grangier et al. (2024b) showed that a powerful technique to obtain good small model on a specific set Dape is to i)
3

---

Soup of Experts
Algorithm 2 Pre-training loop for Soup-of-Expent to minimize the loss function [(S, E, w) in Equation 3.
reflect the distribution of specific tasks that one wishes to address during the specialization phase. In our experiments, we favor sparse domain weights and use meta-distributions и that first sample < k domains and then take uniform random domain weights over these 8 domains.
Input: Initial parameters Z = (S, E, w). domain weights sampling law т, domains D1,..., Dx. optimizer optim, optimizer state s.
for I 0, .,...,
do Sample random domain weights h
2.5. Computationnal cost
Sample mix(h) using Algorithm 1
The Cuup-of-Eperrte leads to some computational overhead compared to standard pre-training, which we explicit here. We compare training a generic model of size d with standard pre-training and training a Soup-of- -ff-xxttts with n experts of size d. The routing MLP is small compared to the model size; hence, we omit the cost of training it in the subsequent analysis. We let b be the mini-batch size.
Compute gradients g of the parameters by backpropa- gation through the loss & (CCombine(S, E, & (h), Update parameters and optimizer state: Z. + optim(g, 8)
end for
Output: Learned parameters S, E,W w
The training loop of models consists of A) computing gra- dients and B) updating the parameters with Adam. For the generic model, the cost of A) scales roughly as db (Ka- plan et al., 2020), we let C be the proportionality con- stant. This computes 2))( with a cost C'bb. For the Houp-of-Experts with respect to the experts E; and shared parameters S. Let θ S + Σ Ea the merged model. Then, the chain rule gives the gradients VsL- = Vol(O, x) and Vg,L= a, Vel(O, z). In other words, the gradients w.r.t. the experts are simply obtained by rescaling the gradient wrt the merged parameters, which gives n b floating point operations. Hence, the cost of A) is C d- Soup-of-Expee Crucially, the cost of backprop through the Soup-of- -o-eexertsxsssseeeee is only mildly affected by n, as the gradients of the experts are obtained trivially from those of the merged model.
ways to define a mapping from domain weights to model weights, we chose the parameterization in Equation 2 as it allows us to easily scale the number of total parameters (by increasing the number of experts n), and we know from the model merging literature that, perhaps surprisingly, different model parameters can be linearly combined to yield one good model (Wortsman et al., 2022).
we need to compute the gradients
The Soup-of-Experts is an asymmetrical model in the sense that it has many trained parameters (the base model and the added expert's parameters), but it instantiates smaller, stand-alone smaller models for inference.
b for the
We now explain how to leverage a large pre-training set in order to train a Soup-of-Experts.
2.4. Training Soups of Experts with meta-distributions
In order to train the Soup-of ofup-of-Experts to achieve good per- formance on a diversity of domain weights, we use meta- distribution R,, that is, sampling law over domain weights.
For B), the generic pretraining needs to update the two Adam EMA parameters, and then update the parameters. In total, the flops count is 14d. For the Soup-of-Expers we need to use Adam for all the parameters, so the cost of B) is 14n- d.
For instance, one can define π as the uniform distribution over histograms, by sampling hi,..., h& i.i.d. uniformly in [0, 1] and defining the domain weights as h; = h4/ Σ hy.
Overall, one training iteration for generic pretraining costs CGen = Cdb + 14d, while it is CsoB = Cdb + 15nd for the SoE. Discarding the cost of Adam for the generic pretraining, the relative increase is CSOE/ OSoe/CGen Hence, we see that it all depends on whether n Cb/15; 15; the value of the constant C' depends on the architecture and other factors. In a large batch size b setting, the added cost of using a -fppeeessss is negligible.
We then train the Soup-of of-Experts by minimizing the aver- age error of the model over this meta-distribution, which is the objective function
15n
Ch
L(S, E,,,, w) = =E [[[ommbine E, (w))) [)]
(3)
We minimize this function using Adam, where at each step, we sample domain weights h π, instantiate the corre- sponding model, sample a mini-batch from mix(h), and do an optimization step on ((Combine(S, E, ((h)). æ). The full algorithm is described in Algorithm 2, and Figure illustrates this training pipeline.
The cost of training a Soup-of-Expert of size with n experts close to that of training a single model of size d, and therefore it is much lower than that of training single large n x d model. For instance, in the subsequent experiments, we train Soup-of- -epettts---eessss--e with 128 experts of size 110M in about a day using grid of 8GPUs. Training a model of size 128 x 110M = 13B would take orders of
The choice of meta distrriisti ग has a critical role on the Coup-of-Experts obtained after training. Ideally, it should
4

---

Specialization in a Soup of Experts
Algorithm 3 (Grangier al., 2024b) Estimating specialist domain weights that are good for specialized dataset Dape
Estimate domain weights
S
Combine
experts
Input: Specialist dataset Dape, embedded domain cen-
D spe
X
E1
troids C1,..., Ck- Init: hi...., hy = O.
MLP
for in D Depe do
h1 h2
h
Find closest centroid: i = arg min |[Brr(x)) cell Increment h; = h; + 1 D spe
Cx
E
end for
Figure 3. Quickly instantiating small model from pre- trained Soup-of-Expers Given a specialist dataset with a few samples, we compute the domain weights using Algorithm 3. The domain weights are then passed through the Soup-of-Experts' MLP to get the coefficients that are then used to merge the experts. This process is quick since the MLP is small, and it requires no training.
Output: Domain weights h hi....,
magnitude more compute, typically weeks with 128GPUs (see (Touvron et al., 2023)).
2.6. Parameter efficient representation with low-rank expert
Ez [ E, x)). Since a is low dimen- sional (in practice we never use more than n 128 ex- perts), this minimization is quick, and unless Dspe has very few samples, there is no risk of overfitting. However, this method requires to backpropagate through the network, which more costly than the previous method.
Thus far, the experts have the same size as the original model. In order to diminish the total number of parameters, we can use low-rank representation for the experts in the spirit of Lora (Hu al., 2021): in each expert Ej. each square matrix W E _axb is materialized as W ABT where A E Raxr and B E Rexr, where r < a, is the rank. Interestingly, even though each expert's matrices are low rank, the instantiation Σ a Eú has rank up to n X r, which might be full rank if we have enough experts. Although promising, we report in our experiments that this method is less parameter-efficien than using fewer dense experts. Still, in heavily resource-come constrre- settings, low-rank experts can benefit over generic model.
As with any other model, the instantiated specialist model can then be fine-tuned on the specialization data to increase its performance if the computational budget allows it.
3. Experiments
We first detail the experimental setup: datasets, models, metrics, and hyperparameters.
Pretraining domains We pre-train language model on Redpajama2 (Weber et al., 2024), a widely used curated web-crawl dataset. We obtain the pre-training domains Di,... Dx with the same clustering method as Grangier et al. (2024b): we embed each document using sentence- bert (Devlin, 2018), and then use the k-means algorithm on these embeddings to split the dataset into pre-training domains. We use a hierarchical k-means, where we first cluster the dataset into k 64 domains and then cluster each of these domains into 64 smaller domains, yielding in total A 4096 domains. We also collect the k correspond- ing centroids c.. in the embedding space, in order to use Algorithm 3 to obtain specialist domain weights.
2.7. Instantiating a flash
After pre-training, the Soup-of-Experts has the flexibility to quickly provide a model that is good for any data dis- tribution domain weights h, simply by forming the param- eters θ- = Combine(S, S, E, (h)). This instantiation only requires& forward pass through a small MLP, and merging parameters; it does not require any training.
We describe two ways to specialize Soup-of-Experts into a model that performs well on a target specific dataset Dspe- The fastest way is to obtain domain weights hape from Depe so that Dape ~ mix(hspe). To do so, we use the nearest- neighbor method of (Grangier et al., 2024b), which de- scribed in Algorithm 3 for completeness. We then instanti- ate the parameters Combine(S, E, (hhspe)). This method is summarized in Figure 3. Since it is simple and fast, this is the method we use in all our experiments
Specialization domains We consider 16 datasets from the PILE (Gao et al., 2020) as target specialization sets: arxiv, dm fin_mathematics enron emails, europarl, freelaw, github, hackernews, nih exporter, openwebtext, 1g19, phil papers, pubmed, stackexchange, ubuntu, uspto, and wikipedia.
For each of these datasets, we compute the corresponding specialist domain weights using Algorithm 3.
A better method, which also more expensive, is to learn a coefficient vector a with gradient descent, by minimizing the function of a only vo(a:) =
We evaluate different methods on each of the specialization
5

---

Soup of Experts
datasets individually, and we report averaged losses over these domains. We defer individual domain results to the appendix.
Soup of Experts
Domain Experts
Generic Pretraining
CRISP
Avg. Specialized Loss
Generic Loss
We highlight that these specialist domains and specialist do- main weights are never used or seen during the pre-training phase for all methods except for CRISP.
3.2
Models We consider standard GPT-2 type transformer ar- chitectures, which we train with the next-token- next-token-predictionn loss. Apart from scaling experiment, we consider base model size of 10M parameters. Architecture and training hyperparameters are specified in Appendix A.
3.0
2.8
20
0
20
Time (h)
Time (h)
Metrics In this work, we measure the ability of a model on specialization dataset with its next-token prediction loss on that domain: we focus solely on language model- ing. This loss predicts well the downstream performance of models with more complex metrics like reasoning, question- answering or translation ability (Gonen et al., 2022; Du et al., 2024; Gadre et al., 2024).
Figure 4. Training curves of the different methods. The average specialized loss is the average of the loss of the models over 16 domains from the Pile. The generic loss is the loss of the models on the standard pre-training distribution of RedPajamav2. The x-axis is the training time. This number is roughly proportion- nal to number of tokens processed, since in this setting, the cost of instantiating the Soup-of-Experts is small in front of that of back backpropgg through the network. The domain experts and CRISP have train many models, so they are not competitive in this setup. The Soup-of- of Expertt performs almost similarly to generic pre-training on the generie loss, which mcans that holds the general knowledge in the pre-training set, while CRISP and Domain Experts are not good generalists (Domain Experts are even out of the figure limits on the right figure). The Soup-of- -ep--ttff----tss-- gives the best specialists, as seen on the left figure.
Training hyperparameters for the Soup-of- Expeetss Un- less specified otherwise, we train the
with 128 experts. With base model size of 110M these ofup-of-Experts therefore hold a total of (128 + 1) x 110M = 14B parameters, that can be linearly combined into small 110M models. Apart from the corresponding analysis, we use a meta-distribution π with a support size of 5 = 4 (see Section 2.4).
Infrastructure We train each model on 8 A100 GPUs.
3.2. Main results
3.1. Baselines
We report the training curves on the pre-training set as well as the average loss on the specialization domains in Figure 4. The specialized loss is obtained by computing the loss on each specialization domain for the corresponding specialist domain; each domain uses a different model (except for the generic pretraining method, which uses the same model for each specialization set).
All the methods we compare in this work instantiate, at specialization time, a model with the same architecture and number of parameters. As explained in the introduction (Table 1). consider the following models:
Generic Pretraining We train one generic model on the standard pre-training distribution. At specialization time, the model stays the same and is evaluated on the specializa- tion set.
The x-axis corresponds time, which in this case close to being proportionnal of the computational cost required to train the model (indeed, the cost of instantiating the experts is small in front of that of backpropagating through the network, see Section 2.5; we get throughput with the SoE that is 77% of that of the generic pretraining).
Domain experts (Gross al., 2017) We train one model on each pretraining domain D., At specialization time, we select the model that yields the smallest loss on the specialization set. This technique does not scale with the number of domains. We only train à = 64 domain experts, as it would be infeasible to train 4096 with our budget.
For the Soup-of- Expertssss and the generic pre-trained mod- els, the training time is unambiguous. For the two other baselines, which train multiple models, we report the total training time taken by all the models.
CRISP (Grangier et al., 2024b) We train one model per specialization set on the mixture mix(hspe). At specializa- tion time, we use the corresponding model. This method does not scale with the number of specialization domains; it requires one pre-training run per specialization domain.
We observe that the Soup-of-Expertt achieves the best per- formance among all methods on the specialized domains, and is only slightly worse than generic pre- pre-training on the pre-traiming loss (while generic pre- tee-training explicitly mini- mizes this loss).
6

---

Soup of Experts
Soup of Experts
Tokens saved with SoE
10"
0.32
Generic Pretraining
Avg. Acc.
0.31
Avg. Spec. Loss
2.8
10
0.30
Generic Pretraiming
Soup of Experts
2.4
10"
20
40
Time
2.2
uspto
exporter
ubuntu
phil
free freelaw pg19
arxiv pubmed
Figure 5. Soup-of-Experts improve downstream tasks. We train Coup-of-Experts with 335M parameters and 32 experts. We report the average accuracy of these models on MMLU, ARC-Easy and ARC-Challenge as a function of the training time using 8GPUs. Soup-of- of-Experts is always more efficient, and training long time yields higher plateau.
10
10"
Fine-tuning Tokens
Figure 6. The gains of Soup-of-Experts during pretraining are maintained during fine-tuning and sometimes lead to large savings. On each of the 16 domains from the PILE, we fine-tune the corresponding instantiated with limited number of fine-tuning tokens. We stop fine-tuning at the point where validation loss starts increasing. Left: Average loss over domains. We see that the Soup of of-Experts maintains its vdnanta regardless the number of available fine-tuning tokens. Right: The number of fine-tuning tokens one needs to fine-tune generic model to reach the same validation loss as the base, not fine-tuned, Soup-of- Experts. For example, uspto, one needs 10M tokens fine-tune the generic model reach the same loss the Soup-of of Experts instantiated uspto out of the box after pre mve-training.
-eep-rr---xpetsss--eee and generic model,
The Soup-of-Experts and the generic pretraining are the only scalable methods with respect to the number of pretraining domains and number of specialization domains. Indeed, we consider 16 specialization domains here. Had we consid- ered more domains, the CRISP method would have taken more and more pre-trainn pre-training time. Similarly, increasing the number of domains would increase the computational cost of the domain experts method lot.
Improvements on downstream tasks In order to verify that the improvements in terms of loss transfer to downstream tasks, we train a larger Soup-of-Expeeee (335M parameters) with fewer experts training. We then evaluate it on question answering tasks: MMLU, ARC-Easy and ARC-Challenge. To instantiate Soup-of- Experts on these tasks, we take a held-out part of these datasets, and use the questions as the specific set, for which we compute the histogram using Algorithm 3. We report the average accuracy on those tasks Figure 5
Soup of Experts
Generic Pretraining
32). We compare it to generic pre-
Avg. Spec. Loss Generic Loss
3.4
Avg. Spee. Loss
3.05
3.2
3.00
3.0
2.95
2.8
10 100 1k 10k samples estimate domain weights
Complementarity to fine tuning For each of the pile do- mains, we instantiate the corresponding Soup-of-Experts We then fine-tune this model and the baseline model with different numbers of available fine-tuning tokens. We report the validation losses in Figure 6, as well as the number of to- kens the generic pretraining method needs to use to recover a performance similar that of the Coup-of-Experte
35M 55M
110M 35M 55M 110M
Model size
Model size
Figure 7. Left: Impact of model scale We train Soup-of-Experse and generic models with different instantiated model sizes. We observe that Coup-of-Experts maintain their advantage across the three scales considered here. Right: Impact of the number of samples in the specialization set In order instantiate spe- cialist model from a Soup-of-Experts. we need to estimate the domain weights corresponding the specialized set, explained in Figure 3. We investigate the impact of the scarcity of data in the specialist set the estimation of the domain weights, which then impacts the instantiated Soup-of-Experts. We see that on average, only three samples are enough to recover model that as good as the generic pretrained model, and 100-1000 samples enough to instantiate the optimal Soup-of-Fxee --pp-rrs---..s....
3.3. Analysis
We quantify the impact of several hyper-parameters on the behavior of the Soup-of-Experts
Model scale We train generic pretrained models and Soup- of-Experts with different instantiated model sizes. We report those results in Figure 7, left.
Domain weights estimation The specialized domain weights, computed with Algorithm 3, are estimated as fre- quencies. When we have little data available the spe-
cialization set, the estimated domain weights become noisy.
7

---

Soup of Experts
We study how data scarcity impacts the performance of the Soup-of-Experts using a limited number of samples input to Algorithm 3. We report the results in Figure 7, right.
distillation (Gu et al., 2024; Riviere et al., 2024) and model compression (Li et al., 2024; Wan al., 2024). Specializa- tion and sparsification are also important, complementary strategies for efficient inference.
distrdiittrnuttion sampling law We study the impact of the sampling law т on the Soup-of-Experns performance. We train several Soup-of-Expert with different support sizes s, as explained in Section 2.4. We report the impact of on the loss on the specialist datasets, the generic dataset, and random sparse domains in Figure 8.
Specialization trades generality for inference efficiency: a small model trained on data close to the targeted domain can be strong on this domain. Since many tasks only pro- vide little in-domain data, fine-tuning a generalist model for few steps is common strategy. Data selection is complementary approach that resamples the pretraining set to emphasize the most relevant parts for the target domain, allowing a specialist model to be trained from scratch. Data selection based on gradient alignment (Fan et al., 2023; 2024; Grangier et al., 2024a; Wang et al., 2024) and im- portance sampling (Grangier et al., 2024b) are the main strategies for task- pretraining.
Low rank experts As discussed in Section 2.6, we inves- tigate the use of low-rank experts. The advantage of this method is that, at a fixed total number of parameters count, we can increase the number of experts and hence the ability of the Soup-of-Expers to have a fine-green grained representation of the diversity of training domains. Sadly, as we report in Figure 9, this method is less parameter efficient than having dense experts. We posit that this is an optimization issue. Indeed, a Soup-of-Experts with experts with a high rank is able, in principle, to learn weights that are very similar to the dense one. Yet, in practice, it is harder to train.
Sparsification improves inference efficiency with mixture of experts (MoE). These models avoid using all parameters for all inputs. They jointly learn expert modules providing specialized weights for different data slices, and routing models deciding which weights to use (Shazeer et al., 2017; Fedus et al., 2022b; Jiang et al., 2024; Dai al., 2024; Abnar et al., 2025). MoEs focus on the computational cost of inference but not on its memory cost (Pan et al., 2024): their routing decision are performed once the input is available, which means that they still require access to all the weights in memory. Model pruning (Xia et al., 2022: 2023; Ma et al., 2023) focuses on task-dependent sparsification as a fine tuning step, which allows memory saving. However, such pruning strategies are difficult since their pretraining has no incentive to discover sparse structures.
Comparison to model merging We compare the Soup-of- Experts with standard model merging. To do so, we train both a ofup-ot-Experts with k=64 domains and generic model for 64000 iterations. Then, we fine-tune the generic model for T steps on each of the k domains, yielding models 01...., Og which are specialists for each domain. We then take k' < k domains among the & at random, and either a) instantiate the SoE with weights uniform over the k' domains and 0 over the other domains, or b) merge the specialized models O, where covers the k' domains. We use a linear combination of models to merge. We then report the average loss of the corresponding model a) or b) on the k' chosen domains. For the model merging, we always pick the number of fine-tuning steps T in [1000, 2000, .... 10000] that yields the smallest loss for each individual experiment.
This work proposes an alternative. Like an MoE we train a large number of parameters but still achieve efficient in- ference. Unlike MoEs, we specialize the model to a given test domain and provide a small, stand-alone model. Unlike fine-tuning or task-aware pretraining, the specialized model does not require knowing the test domain at training time: specialization is not the result of optimization as the spe- cialized parameters results from a closed form merging of the pretrained parameters. An interesting future avenue of research is to combine MoEs and Soup-of- -f--exerts, where the base architecture in the Soup-of-Expers is itself MoE. It would increase the performance of the Soup-of-Expent without sacrificing its latency.
When there is only one domain, fine-tuning is the best method, since it gives a specialist. If we want good models on more than 4 domains, Soup-of-Experts becomes advan- tageous. Importantly, the SoE did not have to be trained on each domain individually: in total, it has done 64000 steps, while the model merging required 64000 + 64 x 10000 steps.
4. Related Work
The simultaneous growth of training set sizes, computational budgets and parameter count has yield large language mod- els (LLMs) strong at addressing a variety of tasks (Brown et al., 2020; Jiang et al., 2023; Dubey et al., 2024).
This work takes inspiration from litterature on the merging of fine-tuned models, aka task-arithmetic. This litterature ob- serves that models fine-tuned from a common ancestor can be linearly combined without retraining (Wortsman et al., 2022; Ilharco et al., 2022; Huang et al., 2023; Ortiz-Jimenez et al., 2023; Tam et al., 2024). It has also been proposed to further fine-tune merge models (Choshen et al., 2022; Rame et al., 2023) Our work extends this merging strategy beyond
The effectiveness of these generalist models comes a high inference and training cost. To improve inference effec- tiveness, significant effort has been invested in knowledge
8

---

Soup of Experts
Avg. Spec. Loss
Generic Loss
3.0
3.1
Test support size
2.9
2
21
2
-23
24
2
-2
-2
Loss
3.0
Loss
2.8
2.9
2.7
2.6
2
21
2
2
24
2
2
2
2
20
2
2
23
2
2
2
2
2
21
2
23
24
26
27
2
Support size
Support size æ
Support size for æ
Figure 8. Role of the support size for the meta-distributton The training meta-distribution draws random domain weights by first sampling random domains, and takes domain weights uniformly at random on those domains. We investigate the impact s. the support size. A small value of means that the Soup-of-Experts does not much interaction between domains, while large means that Soup-of-
rarely sees sparse domains, which are critical for specialized downstream tasks. Left middle: average specialized and generic loss when varying the support size of the meta dettadiitribution. As expected, taking support size of 1 is bad, since the Soup-of of of-Exxerts cannot learn links between domains. The generic loss gets better as the support size increases, which is expected since the generic distribution is spread across all domains. The specialized loss is best for 2, which is explained by the sparsity of the specialized set domain weights. Right: average loss on random mixtures of fixed support size. We see that the training conditions are reflected in testing: the best Soup-of-Expers test domains of support size is the Soup-of-Expers trained with domains.
Number of Experts
Experts type
16
64
128
Dense
Low rank
3.0
Avg. Specialized Loss
Generic Loss
3.10
2.9
Soup Experts Model Merging
2.98
Loss
3.05
32 64
2.96
Number of domains used
3.00
Figure 10. Comparison to model merging Soup-of Soup-of-Experts al- lows to mix more efficiently high number of domains that model merging, while requiring no fine-tuning from base model.
10 Total number of parameters shared between all the experts
10
10
10
Figure 9. Low-rank experts We investigate the possibility low-rank experts as detailed Section 2.6. Low rank experts allow increasing the number of experts parameters, by diminishing the rank of experts. We train multiple Soup-of-Experts with low rank experts, varying the number of experts in {16, 32, 64, 128} and rank in {32, 64, 128, 256, 512), yielding models that have between 300M and 12B parameters in total between the experts. We observe that, at a fixed parameters count, the number experts for low rank experts does not matter. We also train standard Soup-of-Experts with dense experts, we see a different trend. We find that low-rank experts are worse than dense experts.
different number of experts than domains, while we use MLP to map histograms to experts. Finally, we introduce shared parameters addition to the experts, which allows us to amortize between tasks. We discuss the diffcrences with this work in more detail in Appendix B.
fixed total number of
Conclusion
We have introduced a novel asymmetrical architecture, the Soup-of-Experty It holds a large set of expert parameters that encodes a family of small, stand-alone models obtained by linear combination of the parameters. We propose a learning algorithm so that the coefficients of the linear pro- jection are function of the domain weights from which the input is sampled. A pre-traine trainariddd Soup-of- -f--exertsssss-- can, there- fore, instantiate instantly model tailored to any mixture of domain weights. We demonstrated the benefits of this approach on standard datasets, even when these datasets and the corresponding domain weights are unavailable when the soup is trained.
fine-tuning and incorporates into the pretraining phase.
(Dimitriadis al., 2023) also propose architecture that dynamically mixes weights according to the input task, but there are key differences with our work. First, they consider tasks that share the data but differ in their losses, while we consider tasks that have different data distributions with the same loss. Second, their framework does not allow having

---

Soup of Experts
Impact Statement
Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K. Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experss language models. CoRR, abs/2401.06066, 2024. URL pttttp::/aaxxvvorrt abs/2400 060666666666....
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Acknowledgment
Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXh::1810004005 2018.
The authors are indebted to Alaaeldin El Nouby and Marco Cuturi for their insightful comments and suggestions. The authors heavily relied on codebase that was kick-started by Awni Hannun. The authors thank Arno Blaas for his careful proofreadingg Finally, the authors warmly thank Sophie Lepers, Arnau Lepers, and Nicolas Berkouk for their judicious suggestions in the design and organization of the figures.
Dimitriadis, N., Frossard, P., and Fleuret, F. Pareto manifold learning: Tackling multiple tasks via ensembles of single- task models. In International Conference Machine Learning, pp. 8015 PMLR, 2023.
Du, Z., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss per- spective. arXiv preprint arXiv:2403. 15796, 2024.
References
Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint ar vrXv:25511.23700 2025.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21786 2024.
Fan, S., Pagliardini. M., and Jaggi, M. Doge: Do- main reweighting with generalization estimation. arXiv preprint wrXiv:2310.15393 2023.
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting per- formance in domain generalization. Advances in Neural Information Processing Systems, 35:8265 8277, 2022.
Fan, S., Grangier, D., and Ablin, P. Dynamic gradi- ent alignment for online data mixing. arXiv preprint arXiv:2410.0244 2024.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07256. 2021.
Fedus, W., Dean, J., and Zoph, B. A review of sparse expert models in deep learning, arXiv preprint arXiv:2209.01667. 2022a.
Fedus, W. Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022b.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan. A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss. A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https://prcceeddngs neurips. oc
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXhv:2403.08540, 2024.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXi::2101.00027, 2020.
f scch/ppee_ffllees/pppe////////// 1457 ( pdf.
Paper.
Gonen, H., Iyer, S., Blevins, T., Smith, N. A., and Zettle- moyer, L.. Demystifying prompts in language models via perplexity estimation. arXiv preprint 2rXiv:2212.04007, 2022.
Choshen, L., Venezian, E., Slonim, N., and Katz, Y. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.088 030444 2022.
10

---

Soup of Experts
Grangier, D., Ablin, P., and Hannun, A. Adaptive training distributions with scalable online bilevel optimization. Transactions Machine Learning Research (TMLR), 2024a. URL htt.tttt///// http::://pppeeeeeeeeeeee wttpeeereeviee.ete/eeoeeee ner/morum? (D=JP1GVyF511
Li, S., Ning, X., Wang, L., Liu, T., Shi, X., Yan, S., Dai, G., Yang. H., and Wang, Y. Evaluating quantized large lan- guage models. In International Conference Machine Learning (ICML), 2024.
Ma, X., Fang, G., and Wang, X. LLM-Pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Grangier, D., Fan, S., Seto, S., and Ablin, P. Task-adaptive pretrained language models via chustered-importance sampling. arXiv preprint arXiv:2410.03735, 2024b.
Ortiz-Jimenez, G., Favero, A., and Frossard. P. Task arith- metic in the tangent space; Improved editing of pre- trained models. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 66727-66754 Curran Associates, Inc., 2023.
Gross, S., Ranzato, M., and Szlam, A. Hard mixtures of experts for large scale weakly supervised vision. Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6865-6873, 2017.
Gu, Y., Dong, L.. Wei, F., and Huang, M. MiniLLM: Knowledge distillation of large language models. In International Conference Learning Representations, 2024. URL http::::///penreeee..ee.o forum? id=5h0qf71B72.
Pan, B., Shen, Y., Liu, H., Mishra, M. Zhang, G., Oliva, A., Raffel, C., and Panda, R. Dense training, sparse infer- ence: Rethinking training of mixture-of-experts language models. arXiv preprint arXiv:2404.05567 2024.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821 2022.
Huang, C., Liu, Q., Lin, B. Y., Pang. T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint vrXiv:2307.13269 2023.
Rame, A., Ahuja, K., Zhang, J., Cord, M., Bottou, L., and Lopez-Paz, D. Model ratatouille: Recycling di- verse models for out-of-distributin generalization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 of Proceedings of Machine Learn- ing Research, pp. 28656-28679. PMIR, 23-29 Jul 2023. URL https::p// /ttttpp::/pppcceee http///poceeenng.peepe mlr mtt.prsss///pppp////mmm html.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing mod- els with task arithmetic. arXiv preprint arXiv:22/2.04005 2022.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.0663 2023.
Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M.. Soulier, L., and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine- tuned on diverse rewards. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint 140:22401.04088, 2024.
Riviere, M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.000 001188 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models.
arXiv preprint arXiv:2001.0836/ 2020.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: /2t. 2014.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural net- works: The pparsely-gated mixture-of-experts layer, 2017. URL https:
Krajewski, J., Ludziejewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebicra, K., Król, K., Odrzygóždż, T., Sankowski, P., et al. Scaling laws for fine-gre ane-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024.
arwiv.iiiivv.//vv http://://///a/p/////////a////// httaab////11.. 06538.
Tam, D., Kant, Y., Lester, B., Gilitschenski, I., and Raffel, C. Realistic evaluation model merging for compositional generalization. arXiv preprint arXiv:2409. 18314, 2024.

---

Soup of Experts
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P. Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Wan, Z., Wang, X., Liu, C., Alam, S., Zheng. Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q., Chowdhury, M., and Zhang, M. Efficient large language models: A survey. Transactions on Machine Learning Research (TMLR), 2024.
Wang, J. T., Wu. T., Song, D., Mittal, P., and Jia, R. GREATS: Online selection of high-quality data for Ilm training in every iteration. In Advances in Neural Infor- mation Processing Systems, 2024.
Weber, M. Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., et al. Redpajama: an open dataset for training large lan- guage models. arXiv preprint arXiv:2411.12372 2024.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Goneijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing in- ference time. Chaudhuri, K., Jegelka, S., Song, I.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. http///poreeenn...... m.nss/
PMLR, 17-23 Jul
2022. URL StLhttps:: httptutttttt////oeeeeeee v1 62 //t/ sman22a. html.
Xia, M., Zhong, Z., and Chen, D. Structured pruning learns compact and accurate models. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume Long Papers), pp. 1513- 1528, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10. 10.18653/w122022..--ongg 107. URL https:/ http///aaaanoooooooor////0 acl-long 1077
Xia, M., Gao, T., Zeng, Z., and Chen, Sheared Ilama: Accelerating language model pre-training via structured pruming, arXiv preprint arXiv:2310.0669. 2023.
Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P. Le, Q. Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training, arXiv preprint J0449 2023.
12

---

Soup of Experts
A. Training hyper-parameters
Model size
Vocab. size
Embedding size
Hidden MLP dim
layers
num heads
110M
32K
768
3072
12
12
55M
32K
512
2048
12
8
35M
32K
512
2048
6
8
Table Model architectures. We use GPT-2 style transformers, All experiments except the scaling one use the 110M model. We use shared embedding matrices for input and output.
Table 2 details the model architectures used the experiments.
Hyperparameter
Value
Batch size
128
Sequence length
1024
Learning rate
3e-4 for generic pre-training, domain experts and CRISP; le-4 for SoEs
Warmup steps
2000
Adam $1
0.9
Adam B2
0.999
Gradient clipping
0.1
Table 3. Training hyperparameters.
We report the training hyperparameters in Table After a search of learning rate {le-4, 3c-4, le-3), we found that the best learning rate for the Srup-of-Experts
le-4, while it was 3e-4 for the other models.
We use different number of iterations for the different analyses: for the main experiments (Figure 4, Figure 6). we train the Soup-of-Experts and the generic pre-training model for 1024К iterations (134B tokens), while we train the domain experts and CRISP for 128K iterations (17B tokens), since we need to train multiple versions of those models.
For the model size analysis (Figure 7), we train for 1024K iterations (134B tokens). For the support size experiment (Figure 8), and for the low-rank experts experiment (Figure 9), we train for 128K iterations (134B tokens).
B. Comparison with (Dimitriadis et al., 2023)
Using the notation of our paper, it is possible to reframe the model of (Dimitriadis et al., 2023) in the following way.
They consider a distribution of tasks II,..., L that are loss functions from TRP x X' to R. They consider a matrix of task interaction W E R and n experts EI...., En. Given a meta-distributton æ over the simplex of dimension n, they consider the following loss function, which should be optimized with respect the experts parameters:
L(E) = ENND
EROT
Σ [(Wn).L.
Σ hi (eE
(4)
where D is the distribution of the inputs In.nn our view, the best way make this formulation as close the one presented in this paper is consider that D is the product space of the pretraining domains D D2 DR. that has elements (Z....., Ik) and to consider the loss functions L;(0; (fI...., E(0: æz), and to take W 1g. Then, Equation 4 becomes
i(E)
ER
DE el Σ h2E3))
(5)
k
= ER
T
h Fa;
(6)
This formulation highlights the key differences with our work:
13

---

Soup of Experts
There are shared parameters in the experts, while have shared parameters in the experts that allow to amortize computations
The input domain weights are used directly to mix the experts, while we use a small MLP to map the domain weights to the experts. This allows us have a different number of experts than domains.
Finally, in the original formulation of Equation 4, the expectation is taken over domains, and the algorithm proposed by Dimitriadis et al. (2023) minimizes the loss over this expectation. our setting where D is a product space, it means that the algorithm of Dimitriadis et al. (2023) needs to query samples from each domain in order to do one step of optimization, while our training loop only queries samples from one mixture of domains a time.
14

---

Soup of Experts
C. Detailed per-specific-oomain results
We report the detailed results on the 16 PILE domains in Figure Figuee 12, Figure 13, and Figure 14.
15

---

Soup of Experts
Soup Experts
Generic Pretraining
Domain Experts
CRISP
arxiv
maths
emails
parl
3.5
4.5
2.2
3.0
4.0
2.0
3.0
2.5
50
0
50
50
50
Tokens (B)
Tokens (B)
Tokens (B)
Tokens (B)
freelaw
github
hackernews
exporter
3.0
3.0
4.00
3.00
3.75
mamag
2.5
2.5
2.75
3.50
50
0
50
50
0
50
Tokens (B)
Tokens (B)
Tokens (B)
Tokens (B)
openwebtext
pg19
phil
pubmed
4.0
4.0
4.0
3.00
2.75
3.5
3.5
3.5
2.50
50
0
50
50
50
Tokens (B) stackexchange
Tokens (B) ubuntu
Tokens (B)
Tokens (B)
uspto
wikipedia
3.50
3.5
3.50
3.5
3.25
3.0
3.25
3.0
3.00
3.00
2.5
50
0
50
50
0
50
Tokens (B)
Tokens (B)
Tokens (B)
Tokens (B)
Figure 11. Training curves of the different methods. Detailed results from Figure 4 on the 16 PILE domains.
16

---

Soup of Experts
Soup of Experts
Generie Pretraining
Soup of Experts without any finc-tuning
arxiv
maths
emails
parl
2.5
2.5
1.5
2.0
2.0
1.0
2
10
10
10
10
10
Fine-tuning Tokens freelaw
Fine-tuning Tokens github
Fine-tuning Tokens hackernews
Fine-tuning Tokens exporter
2.8
2.4
3.25
2.0
2.6
2.2
3.00
1.5
2.4
10
10*
10
10
10
10
10
10
Fine-tuning Tokens openwebtext
Finc-tuning Tokens pg 19
Finc-tuning Tokens phil
Finc-tuning Tokens pubmed
3.1
3.25
2.4
3.2
3.0
3.00
2.2
2.9
3.0
2.75
2.0
10
10
10
10
10
10
10
10
Fine-tuning Tokens stackexchange
Fine-tuning Tokens ubuntu
Funiintuning Tokens uspto
Fine-tuning Tokens wikipedia
3.0
2.6
2.8
2.5
2.4
2.6
10
10
10
10
10
10
10
10
Fine-tuning Tokens
Fine-tuning Tokens
Finc-tuning Tokens
Finc-tuning Tokens
Figure 12. Fine tuning results. Detailed results from Figure 6 on the 16 PILE domains. The dashed horizontal line indicates the loss of the Soup-of-Exxe Experts without fine- munituning.
17

---

Soup of Experts
Soup Experts
Generie Pretraining
arxiv
maths
emails
parl
3.00
2.8
2.00
3.90
2.95
2.7
1.95
3.85
1.90
2.90
3.80
10
10'
10
10a
10
10
10
10
freelaw
github
hackernews
exporter
2.7
3.50
2.9
2.45
2.6
2.40
3.45
2.8
2.5
2.35
10'
10"
10
10
10
10
10 pubmed
10
openwebtext
Pg19 19
pg
phil
3.40
3.25
3.4
2.60
3.35
3.20
2.55
3.3
3.30
3.15
10
10²
10
10
10
10'
10
stackexchange
ubuntu
uspto
wikipedia
3.05
3.20
2.95
2.65
2.60
3.00
3.15
2.90
2.55
2.95
3.10
2.85
2.50
10
10
10
10
10
10
10
10
samples to estimate domain weights
# samples to estimate domain weights
# samples to estimate domain weights
samples to estimate domain weights
Figure 13. Impact of the number of samples in the specific set on the instantiated Soup-of-Experts. Detailed results from Figure 7,
right, on the 16 PILE domains. The dashed horizontal line indicates loss the Coup-of-Experts without fine-tuning.
18

---

Soup of Experts
arxiv
maths
emails
parl
3.00
2.8
2.50
3.9
2.95
2.7
2.25
2.90
3.8
2.00
2.85
3.7
2.6
2
2 24
2
22
2
2 24
2
22
2 2
2
2
2
23
2
2
2
freelaw
github
hackernews
exporter
2.8
3.6
3.0
3.0
2.6
3.5
2.5
2.8
2.4
3.4
2021
2
24
2
26
2
2
2
2
12
2
22
23
2622262"
20
2
24
2
22
openwebtext
Pg19
phil
pubmed
3.5
2.8
3.3
3.5
3.4
3.4
3.2
2.6
3.3
3.3
2
23
22
2
2
23
2
2
22
20
2 22
2
2
23
24
2
26
22
stackexchange
ubuntu
uspto
wikipedia
3.4
3.0
2.8
3.1
3.3
2.9
3.2
2.6
3.0
2.8
3.1
2.9
2º
25
20 22
2 21
20
22
23 2212
20
23
23
22
Support size for
Support size for
Support size for
Support size for
Figure 14. Impact of the support size of the meta-distribution. Detailed results from Figure 8, on the 16 PILE domains.
19


Reviews:
[{"review_id": "P7TOyDxBwo", "reviewer": "Reviewer_4deQ", "summary": "The authors introduce \"Soup-of-Experts\" a method for pretraining language models so that they can quickly instantiate small, specialized models for different domain distributions. The architecture consists of a shared base model, multiple expert parameter sets, and a small learned routing function (an MLP) that maps domain weights (i.e., mixtures of training domains) to a set of coefficients used to linearly combine the experts at inference. Since each specialist is simply a linear combination of trained parameters (plus the shared parameters), it can be instantiated in a single forward pass of the routing MLP, with no further training or fine-tuning needed. Experiments on several language modeling tasks (drawn from RedPajama for pretraining and from the Pile for specialization) show that the proposed approach can produce strong specialist models that significantly outperform a single generic model while only slightly sacrificing performance on a broad domain. Soup-of-Experts also outperforms other baselines (Domain Experts, CRISP) in terms of specialization performance and computational efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nFirst, we thank you for your work, and for your remarks that will help us improve our work. We are happy to hear that *\"the claims to be supported by sound evidence\"*, that *\"the methods and evaluation criteria are well aligned with the stated goal\"*, and that *\"Experiments are methodologically consistent\"*.\n\nWe now address your concerns.\n\n- *\"Low-rank experts did not yield expected improvements; there is a need for further investigation.\"*\n\nIndeed, we could not make this idea work, and honestly report it in the paper. We think making it work would enable another avenue to scale soup-of-experts, even though we could scale them without Lora (see https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf where we train 350M models, with 32 experts for the SoE).\n\n- *\"Potential scalability challenges when significantly increasing the number of experts beyond tested scenarios.\"*\n\nWe need to select a tradeoff between the number of experts and model size, at a fixed memory size. We have been able to train 350M models with 32 experts, see https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf\n\n- *\"Limited theoretical analysis or no formal guarantees regarding stability.\"* \n\nIndeed, we do not conduct theoretical analysis, as we are unsure what type of results we would want to prove; note that papers in a similar flavor like [1,2,3] also do conduct any theoretical analysis\n\n- *\"It would have been helpful to explicitly clarify how sensitive the results are to different embedding methods used for domain clustering.\"* and *\"Also, I am curious to know how sensitive is the method's performance to the choice of embedding method used during the clustering of pretraining domains?\"*\n\nThanks for the suggestion. We will clarify in the text that we use this type of embeddings since it is the one which offers the best performance are reported in [4], fig.5. Due to lack of time in the rebuttal period, we could not obtain training runs using different embeddings, but we will make sure to add this ablation to the final version.\n\n- *\"Consider providing additional runtime benchmarks or memory usage comparisons explicitly against baselines.\"*\n\n\nWe note that fig.4 gives a time comparison between methods, just like the newly added larger scale experiment https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf .\nThe baseline and Soup-of-Experts training use the same hardware, that is one node of 8 A100 GPUs, and both have a GPU utilization of over 95%. We wil include these datapoints in the final paper.\n\n\n- *\"It would be helpful if the authors clarified the effect of the MLP capacity on final performance. Does a larger routing network yield more fine-grained domain combinations?\"* \n\nWe have run an ablation where we change the width of the routing network. The MLP has input dimension 4096, output dimension 128, and we either take a hidden dimension of 4 * input_dimension or 4 * output_dimension. We could not see any difference between those two choices; the training curves looked similar. We will make sure to add a more thorough ablation in the final version of the paper.\n\n\nWe hope that our response clarifies the questions you have raised, and we thank you again for your thorough review. \n\n[1]: Ha, David, Andrew Dai, and Quoc V. Le. \"Hypernetworks.\" arXiv preprint arXiv:1609.09106 (2016).\n\n[2]: Krajewski, Jakub, et al. \"Scaling laws for fine-grained mixture of experts.\" arXiv preprint arXiv:2402.07871 (2024).\n\n[3]: Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" International conference on machine learning. PMLR, 2022.\n\n[4]:Grangier, David, et al. \"Task-adaptive pretrained language models via clustered-importance sampling.\" arXiv preprint arXiv:2410.03735 (2024).", "claims_and_evidence": "I consider the claims to be supported by sound evidence. Details are provided below:\n\nClaim (1) - SoE can quickly produce specialized models targeted at specific domains by simply merging expert parameters with learned coefficients (no fine-tuning needed) >> The authors show that for 16 domains from Pile, SoE achieves lower perplexity/loss than a comparable single, generically pre-trained model.\n\nClaim (2) - SoE is scalable >> The authors compare SoE to two baselines that train separate models for each domain. These baselines incur linear overhead (one pretraining done per domain), whilst SoE has a single pretraining phase for all possible domain mixture, then re-combines parameters at specialization.\n\nClaim (3) - SoE almost preserved general performance >> Figure 4 confirms that SoE's overall perplexity on the broad (generic) dataset is only slightly worse than that of a purely generic model, suggesting that SoE retains broad-domain knowledge while also being adaptable.", "methods_and_evaluation_criteria": "From a reviewer’s standpoint, the methods and evaluation criteria are well aligned with the stated goal of obtaining strong specialized models from a single pretraining run. The choice of next-token prediction loss to measure performance is standard for language modeling tasks.", "theoretical_claims": "* The paper does not provide a formal proof of correctness beyond standard machine-learning assumptions about parameter interpolation, so there are no complex proofs to verify.  \n\n* The theoretical argument is mostly intuitive, referencing earlier works on “model soups” / model merging and showing that experts with a shared initialization can be combined (especially when they are simultaneously trained).", "experimental_designs_or_analyses": "Experiments are methodologically consistent (fair), meaning that each approach is trained with similar computational budgets (in tokens) and is then evaluated under the same metrics (next-token prediction loss). The authors report both the specialized loss and the generic training loss to show the trade-off between specialization and generality.", "relation_to_broader_scientific_literature": "The authors position SoE relative to large language models (generic but expensive) and specialized smaller models (efficient but less general). They frame SoE as a novel extension of “model soup” ideas and parametric interpolation methods but used during pretraining with learned mapping from domain weights to combination coefficients.", "essential_references_not_discussed": "The references appear sufficient to contextualize the core idea, but I did not carefully check for other similar references.", "other_strengths_and_weaknesses": "Strengths:\n- Clear motivation and problem formulation, well-written submission.\n- Novel combination of existing ideas (model merging + MoE-inspired expert routing).\n- Empirical validation across datasets.\n- Extensive ablation studies providing insights into method robustness (in the supplementary).\n- Practical significance due to low computational overhead at specialization time.\n\nWeaknesses:\n- Low-rank experts did not yield expected improvements; there is a need for further investigation.\n- Potential scalability challenges when significantly increasing the number of experts beyond tested scenarios.\n- Limited theoretical analysis or no formal guarantees regarding stability.", "other_comments_or_suggestions": "It would have been helpful to explicitly clarify how sensitive the results are to different embedding methods used for domain clustering. \n\nConsider providing additional runtime benchmarks or memory usage comparisons explicitly against baselines.", "questions_for_authors": "It would be helpful if the authors clarified the effect of the MLP capacity on final performance. Does a larger routing network yield more fine-grained domain combinations?\n\nAlso, I am curious to know how sensitive is the method's performance to the choice of embedding method used during the clustering of pretraining domains?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncrLsDUIcT", "reviewer": "Reviewer_No4u", "summary": "The authors propose Soup-of-Experts, which can quickly create a specialized model for a given mixture of data domain weights at test time. Soup-of-Experts jointly pretrains and learns a function to compute weights for a linear combination of expert model weights for specialization.\n\n## update after rebuttal\nI updated the review by increasing the score from 2 to 3 after the authors addressed the majority of my concerns. I still see scaling as a drawback due to the large number of experts that need to be updated jointly. This also makes the method more inflexible since all of the experts need to be trained at once during pretraining.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \nThanks a lot for your review and for your questions and remarks; they will help us improve the paper. We are please to see that you found that the main claim is well supported, that \"the proposed method makes sense\", and that the experiments are well designed.\n\nWe now answer your questions and the points you have raised:\n\n- \"*There are a number of similar approaches in the domain adaptation realm in the fine-tuning setting which are not discussed, such as rewarded soups [1] and personalized soups [2]. It would be useful for the authors to provide comparisons to these approaches,[...]*\"\n\nThanks for raising this point. While we already mentioned [1], we will discuss [2] and [3] in the paper. Thanks to your remark, we have implemented a comparison to standard model merging. To do so, we train both a SoE and a generic model during 64000 iterations, and for the SoE we use k=64 domains. Then, for each k domain, we fine-tune the generic model for T steps on that domain, yielding models $\\theta_1,\\dots \\theta_k$ which are specialists for each domain. \n\nWe then take $d$ domains among the $k$ at random, and either a) instantiate the SoE with weights uniform over the $d$ domains, 0 otherwise, or b) merge the specialized models $\\theta_i$ where i covers the $d$ domains. We use a linear combination of models to merge. We then report the average loss of the corresponding model a) or b) on the $d$ chosen domains. For the model merging, we always pick the number of fine-tuning steps T in [1000, 2000, ..., 10000] that yields the smallest loss for each individual experiment. The results are here: https://anonymous.4open.science/r/rebuttal_soup-E8F9/soe_vs_merge.pdf \n\n When there is only $1$ domain, fine tuning is the best method, since it has trained a specialist. As soon as we need to merge >= 4 specialists, Soup-of-Experts become advantageous. Importantly, the SoE did not have to be trained on each domain individually: in total, it has done 64000 iterations, while the model merging required 64000 + 64 * 10000 steps, so about 10 times more compute ! We will add this insightful experiment to the paper.\n\n- \"*This approach seems very computationally expensive, since we first need a large number of domain experts which then need further pre-training to be adapted*\"\n\nWe will clarify the text, the computational cost of the method is not high as long as we have enough memory to train the models. For instance, the x-axis in fig4 is the number of hours to train the models, the fact that the SoE is better means that it reaches better losses in less time; it is therefore not more expensive.\n\n- \"*The authors do not compare to or discuss approaches for souping fine-tuned models for domain adaptation.*\" \n\nSee above.\n\n- \"*Experiments are limited to small models (33M - 110M parameters)*\"\n\nWe have now trained a 350M model with 32 experts for 33B tokens,, see https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf . The method scales: it still is about twice as fast as generic pretraining to reach a certain level of specialized loss. We will add this to the paper.\n\n- *\"When is this method advantageous over using a single pretrained model of the same total number of parameters as is contained in the experts? What is the total training time comparison between this larger model when compared to training the experts and Soup-of-Experts?\"*\n\nThis is an interesting question. To give a concrete order of magnitude, in the experiments on the paper we use 110M models with 128 experts, yielding in total 13B parameters. There are many differences between the SoE and a 13B model. First, the SoE will most likely have a much higher perplexity, since it instantiates a model 100 times smaller. However, as a trade-off, it will have a much higher throughput since it is a 100 times smaller model, and because of that the training costs are also orders of magnitude different. For instance, training a 13B model may take weeks with 128 GPUs (see table 2 in [3]), while our training takes ~1 day with 8 gpus. Overall, these models are very different. We will clarify this in the text. \n\n- *\"How does this method compare to existing approaches combining fine-tuned models from multiple domains?\"*\n\nSee response above. Indeed, Soup-of-Experts is a pretraining method, it should be seen as complementary to usual model merging.\n\n- \"*Is this approach feasible for larger models? The large number of experts seems like a computational challenge if the models are scaled.*\"\n\nAs said above, we could train larger 350M models on one node of 8 Gpus. We could train larger models with more computational ressources, but this is beyond our constraints.\n\nWe hope that we have convincingly answered your questions, and we thank you again for your review !\n\n[3]: Touvron, Hugo, et al. \"Llama 2: Open foundation and fine-tuned chat models.\" arXiv preprint arXiv:2307.09288 (2023).", "claims_and_evidence": "The main claim, that soup-of-experts can be quickly used to obtain a pre-trained specialized model, seems somewhat well-supported by experiments comparing to other pretraining methods, as it performs slightly better than CRISP in the specialized loss while having much better performance in the generalized loss. It also has better specialization and similar general loss to pretraining a single similar-sized model.", "methods_and_evaluation_criteria": "The proposed method makes sense as a way to adapt model souping to pretraining. The benchmark datasets used have a variety of specialization domains and are useful for evaluation.", "theoretical_claims": "There were no proofs", "experimental_designs_or_analyses": "The experiments seem well-designed as a way to compare the models' general and domain adaptation capabilities. However, there are some additional baseline comparisons that would be useful (see Questions 1-2).", "relation_to_broader_scientific_literature": "While model soups have been used previously with fine-tuned models and existing pretrained models, this paper instead focuses on test-time domain adaptation using linear souping by jointly pretraining expert models and a function on the domain weights.", "essential_references_not_discussed": "There are a number of similar approaches in the domain adaptation realm in the fine-tuning setting which are not discussed, such as rewarded soups [1] and personalized soups [2]. It would be useful for the authors to provide comparisons to these approaches, since they only provide comparisons to pretraining-based approaches and it is not clear if pretraining results in gains over similar works combining fine-tuned models.\n\nIt would also be useful to discuss previous methods souping pretrained models without requiring joint pretraining from scratch, such as [3]. \n\n[1] Rame et. al. Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards\n[2] Jang et. al. Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging\n[3] Imfeld et. al. Transformer Fusion with Optimal Transport.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of jointly pretraining the experts and linear combination weights seems novel and interesting, compared to existing approaches that do one or the other\n2. The experiments support the claim that this approach increases domain adaptation performance over existing pretraining-based methods\n\nWeaknesses:\n1. This approach seems very computationally expensive, since we first need a large number of domain experts which then need further pre-training to be adapted\n2. The authors do not compare to or discuss approaches for souping fine-tuned models for domain adaptation.\n3. Experiments are limited to small models (33M - 110M parameters)", "other_comments_or_suggestions": "Typo in 2.5 section header", "questions_for_authors": "1. When is this method advantageous over using a single pretrained model of the same total number of parameters as is contained in the experts? What is the total training time comparison between this larger model when compared to training the experts and Soup-of-Experts?\n2. How does this method compare to existing approaches combining fine-tuned models from multiple domains? Soup-of-Experts seems inherently less flexible since the experts must be pretrained from scratch rather than leveraging existing fine-tuned models. \n3. Is this approach feasible for larger models? The large number of experts seems like a computational challenge if the models are scaled.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NGWEuhOR7z", "reviewer": "Reviewer_qpFf", "summary": "This paper studies the problem of training specialized models efficiently for a new domain. This is done by design from the pretraining stage by training a model in a style akin to mixture-of-experts. The data is sampled across domains in a mini-batch and each expert corresponding to each domain is updated, together with the general model that is always updated. This results in one model for each domain and a general model. For specialization, the data from the new domain is categorized using nearest neighbors based on semantic similarity into a distribution. This distribution is them used to perform a weighted merging of models via averaging to obtain the specialist model.\n\nThe paper uses a model of 110M parameters based on the GPT-2 architecture and pretrains on RedPajama2 128 different experts with data from 4096 domains that are obtained using k-means on semantic similarity. The loss of the model is shown to be worse on the generic data than that of the regular model. However, when using data from 16 domains of the Pile data set, fine-tuning the obtained specialized model shows better loss than the generic model.\n\nThe paper is overall well-written. The task setup that is being studied is very relevant and important.\n\nIn my opinion, the major drawback of the paper is that is does not show evaluation on downstream data sets and only shows the loss. Even the paper's abstract mentions that the downstream performance is the main motivation for this setup. Additionaly, perplexity would be another interesting metric to see.\n\nAnother important drawback of this approach is that in this experiment, we need to keep n=128 experts trained, which will require a lot of space. Further, in order to efficiently do the model updates, all these experts need to be stored in memory, which is prohibitive for large models.\n\n## update after rebuttal\n\nThank you for your response.\n\nThe additional experiments are indeed valuable and show positive results. It would be good to learn more about them and also present multiple downstream evaluations on more than 3 data sets.\n\nRegardless, I am willing to improve my overall assessment in light on these two new results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nWe thank you for your review and for your insightful comments, which will help us improve the paper. We are happy to hear that \"The paper is overall well-written\" and that  \"The task setup that is being studied is very relevant and important.\"\n\nWe now address your concerns.\n\n- \"*the models trained are very small for generative LLMs (110M parameters). I suggest running an experiment on a larger model.*\"\n\nWe have now trained larger 350M models, using 32 experts for the Soup-of-Experts. The training curves are available here: https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf. The behavior is essentially similar to that in fig.4 in the paper. Due to lack of resources, we could not train larger models.\n\n- *\"the major drawback of the paper is that is does not show evaluation on downstream data sets and only shows the loss.\"* \n\nThis is a good point. We have score the 350M models that we have trained on three datasets: MMLU, Arc-easy and Arc-challenge. We report the corresponding average accuracy as a function of time in this figure: https://anonymous.4open.science/r/rebuttal_soup-E8F9/accs.pdf\n\nWe observe that the SoE consistently outperforms generic pre-training. We will add these results to the paper. We believe that this result indeed makes the paper more convincing.\n\n- *\"Additionaly, perplexity would be another interesting metric to see.\"*\n\nIn all our experiments, we train and report with the next-token prediction loss; the perplexity can therefore be obtained by taking the exp of our curves in fig. 4, 5, 6, 7. It will not change the ordering of methods.\n\n- *\"Another important drawback of this approach is that in this experiment, we need to keep n=128 experts trained, which will require a lot of space. Further, in order to efficiently do the model updates, all these experts need to be stored in memory, which is prohibitive for large models.\"*\n\nIndeed, we will clarify in 2.5 that the method requires storing all the expert's weights, which might be problematic for large model sizes. However, in the case where memory is not an issue, Soup-of-Experts achieves significantly better performances than standard pretraining at a fixed computational budget, as reported in Fig. 4 and https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf.\n\n- \"*Claims are not fully supported by the experiments as the downstream fine-tuning metrics and perplexity are lacking.*\"\n\nSee above, this is now fully fixed.\n\n- \"*The major weakness in my view of this paper is that is lacks evaluation on downstream fine-tuned models with tasks in the specalist domain distribution. Even though the loss looks better, further validation on fine-tuning is critical to prove the effectiveness of the approach.*\"\n\nAs seen above, we have validated that these gains in loss also transfer to downstream tasks.\n\n\n- *\"Another model trained on this setup will add additional strength and show more generality.\"*\n\nAs explained above, we have now trained larger 350M models. We think that this indeed improves the generality of our paper. We acknowledge that we focus solely on the decoder-only LLM architecture, which is one of the most widely used architecture nowadays.\n\n- *\"The Pile data sets may not be very different to the data that is present in the RedPajama2 set, as that contains common crawl snapshots, and the pile data sets are most likely present inside these snapshots (e.g. wikipedia).\"*\n\nIndeed, this is a judicious and important remark. We used the Pile as it has a variety of downstream domains that overlap more (wikipedia) or less (dm_mathematics) with the pretraining domain, in order to cover a variety of practical cases. We will clarify this in the text.\n\n\n- \"*Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models - Li et al 2022.*\"\n\nThanks for the relevant reference that we missed, we will discuss it in the paper.\n\n- \"*Page 5: sentence-bert (Devlin, 2018) is the wrong citation for sentence-bert. It would be good to know which specific model version was actually used for similarity.*\"\n\nThanks for spotting this ! Indeed, the model we use is a fine-tuned version of MPNET [1], obtained at https://huggingface.co/sentence-transformers/all-mpnet-base-v2. We will properly acknowledge this in the paper.\n\n\nWe thank you again for your work reviewing our paper and for your comments which help us improve the paper. We hope that your concerns have been alleviated. \n\n\n[1] Song, Kaitao, et al. \"Mpnet: Masked and permuted pre-training for language understanding.\" Advances in neural information processing systems 33 (2020): 16857-16867.", "claims_and_evidence": "Claims are not fully supported by the experiments as the downstream fine-tuning metrics and perplexity are lacking.", "methods_and_evaluation_criteria": "The major weakness in my view of this paper is that is lacks evaluation on downstream fine-tuned models with tasks in the specalist domain distribution. Even though the loss looks better, further validation on fine-tuning is critical to prove the effectiveness of the approach.\n\nFurther, the models trained are very small for generative LLMs (110M parameters). I suggest running an experiment on a larger model. An alternative would be to try this on BERT/RoBERTa models. Another model trained on this setup will add additional strength and show more generality.", "theoretical_claims": "The main contribution of this paper is the new architecture for the pre-trained model. This is described well and in good amount of detail.", "experimental_designs_or_analyses": "The Pile data sets may not be very different to the data that is present in the RedPajama2 set, as that contains common crawl snapshots, and the pile data sets are most likely present inside these snapshots (e.g. wikipedia).", "relation_to_broader_scientific_literature": "See summary section", "essential_references_not_discussed": "Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models - Li et al 2022.\n- this paper explores a similar setup", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "Page 5: sentence-bert (Devlin, 2018) is the wrong citation for sentence-bert. It would be good to know which specific model version was actually used for similarity.", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PcCRfmsIHW", "reviewer": "Reviewer_wvpV", "summary": "This paper introduces Soup-of-Experts, a method that trains a group of expert models to construct a specialized model for a given target domain. The specialized model is obtained by linearly combining the expert models in parameter space. The architecture consists of a set of expert models, a shared model, and an MLP that generates blending weights based on input domain weights. During testing, the domain weights for each target domain are estimated either through an off-the-shelf method or a learning-based approach, leveraging training domain information for comparison. The proposed method demonstrates superior performance across various domains while maintaining a compact specialized model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe thank you for your review and for helping us improve the paper. We are happy to read that \"The experiment design and analysis are appropriate.\", that \"this paper delivers good results\", and that \"the proposed method is easy to understand\".\n\nWe now try to address your concerns.\n\n- *\" The explanation in Section 2.5 regarding the negligible overhead cost for training the group of experts is unclear. [...] Additionally, both the forward computation and backpropagation scale linearly with the number of experts.\"*\n\nThis is an important remark, thanks ! We will greatly clarify this part with the following analysis: We compare training a model of size d with generic pre-training and training a Soup of Experts with n experts of size d. We let b be the mini-batch size.  The training loop of models consists of A) computing gradients and B) updating the parameters with Adam.\nFor the generic model, the cost of A) scales roughly as $db$ [1], we let $C$ be the proportionality constant. This computes $\\nabla\\_\\Theta \\ell(\\Theta, x)$, with a cost $C\\cdot d\\cdot b$.\nFor the SoE, we need to compute the gradients w.r.t. the experts $E\\_i$ and shared parameters $S$.  Let $\\Theta = S + \\sum \\alpha\\_i E\\_i$ the merged model. Then, the chain rule gives the gradients $\\nabla_S L = \\nabla\\_\\Theta \\ell(\\Theta, x)$ and $\\nabla\\_{E\\_i}L = \\alpha_i \\nabla\\_\\Theta \\ell(\\Theta, x)$. In other words, the gradients w.r.t. the experts are simply obtained by rescaling the gradient wrt the merged parameters, which gives $n\\cdot b$ floating point operations. Hence, the cost of A) is $C\\cdot d\\cdot b + n\\cdot b$ for the SoE. Crucially, **the cost of backprop through the SoE is only midly affected by n**, as the gradients of the experts are obtained trivially from those of the merged model; the cost is **not** $C\\cdot b\\cdot n\\cdot d$, which would be the same cost as that of training a much larger $nd$ full model.\n\nFor B, the generic pretraining needs to update the two adam EMA parameters, and then update the parameters. \nIn total, the flops count is 14d. For the SoE, we need to use Adam for all the parameters, so the cost of B) is 14$n\\cdot d$.\n\nOverall, one training iteration for generic pretraining costs $C_{gen} = Cdb + 14d$, while it is $C_{soe} = Cdb + 15nd$ for the SoE. Let us discard the cost of Adam for the generic pretraining, the relative increase is $C_{soe} / C_{gen} = 1 + \\frac{15n}{Cb}$. Hence we see that it all depends on whether $n \\gg Cb / 15$; the practical value of the constant $C$ depends on the architecture and other factors.\nIn a large batch size $b$ setting, the added cost is negligible.\n\nWe insist that in fig 4 and in https://anonymous.4open.science/r/rebuttal_soup-E8F9/reb_soe_time.pdf we report time as x-axis, where the additionnal (small) overhead is already factored in.\n\n- \"*Furthermore, it is unclear which “standard pre-training” is being referenced for comparison\"*\n\nWe will clarify that standard pre-training trains one single model of size $d$, which in section 2.5. is compared to the cost of training one soup of experts with $n$ experts of the same size $d$.\n\n-\"*the experimental results indicate that selecting only two domains (s = 2) and sampling uniformly from them yields the best performance. This seems counterintuitive[...]*\"\n\nWe want to clarify a possible misunderstanding: as explained in 2.4, we train using only $s$ active domains at a time, but these domains are sampled at random at each iteration. For instance, if we had only 3 domains and use $s=2$, sampling from $\\pi$ might give these 4 samples: `[0, 0.2, 0.8], [0, 0.6, 0.4], [0.1, 0.9, 0], [0.7, 0, 0.3]`, where there are only 2 non-zero domain at each time, but their index change.\nHence, this strategy allows us to explore the space of all pairs of domains, not just one fixed pair of domain. In practice, we use 256000 iterations, so the model has seen data coming from each domain multiple times. We hope that this clarifies the \"counterintuitive\" observation that the SoE accelerates training. We will add this explanation in the text to make it clearer.\n\n- \"*the model includes a shared branch [...]. Given the sparse domain sampling strategy, it is unclear whether this setup might lead to conflicts across different training iterations.*\"\n\nIndeed, as you mention, there is a tension between domains, and the shared branch has to accomodate between all domains, just like a standard LLM has to share its capacity across different domains. However, we did not observe any particular instabilities during training, the training hyperparameters (gradient clipping, adam parameters) are standard, and the curves in fig.4 are very smooth. \n\nWe hope that our answers have addressed your concerns, and thank you again for your helpful review !\n\n[1]: Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).", "claims_and_evidence": "•  The explanation in Section 2.5 regarding the negligible overhead cost for training the group of experts is unclear. The authors state, “The forward pass through the experts also yields a negligible cost as long as the experts all fit in memory since it only requires adding several parameters.” However, this claim is confusing because the parameter space is duplicated n times, along with the saved feature maps. Additionally, both the forward computation and backpropagation scale linearly with the number of experts. Furthermore, it is unclear which “standard pre-training” is being referenced for comparison. The description in lines 306–309 suggests that the generic pretraining involves only a single model, making the comparison unclear.\n\n•  The paper's primary motivation is to incorporate domain information during training to learn specialized experts. This approach suggests that capturing a diverse distribution of domains at the meta-level should be beneficial. However, the experimental results indicate that selecting only two domains (s = 2) and sampling uniformly from them yields the best performance. This seems counterintuitive, as s = 2 appears overly sparse given the total of 4096 training domains. Such a small sample may not adequately capture the complex relationships among domains. Moreover, iterating over all 4096 domains would require at least 2048 iterations. Yet, the authors claim that this approach can reduce both training cost and time, which seems unintuitive and requires further clarification.\n\n•  Additionally, the model includes a shared branch, \\mathcal{S}, which is common across all training domains and experts. Given the sparse domain sampling strategy, it is unclear whether this setup might lead to conflicts across different training iterations. Since \\mathcal{S} is optimized toward the selected two domains at each iteration, substantial domain gaps between different domain pairs across iterations could lead to instability in training. The paper should address whether this issue affects model convergence and performance.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiment design and analysis are appropriate.", "relation_to_broader_scientific_literature": "This paper is closely related to major learning paradigms in language models, including dense architectures and Mixture-of-Experts (MoE) structures. Additionally, the proposed method builds on the concept of model soup from previous literature, which averages models in parameter space to achieve a better tradeoff between in-distribution and out-of-distribution performance (Wortsman et al., 2022). The primary objective of this paper is to mitigate distribution shift and adapt the model more effectively to target domains.", "essential_references_not_discussed": "The references are thoroughly discussed.", "other_strengths_and_weaknesses": "This paper delivers good results on how to reduce the resources and computation needed when the model is expected to be specialized to a particular domain. \n\nThe proposed method is also easy to understand and does not contain complex settings/modules. It will benefit the future research in the area, especially larges models become the focus,s but it is also important to focus more on the specialized scenarios.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ArCkkC3Mk", "reviewer": "Reviewer_pSxt", "summary": "The paper proposes a method for combining a set of pretrained models by learning linear coefficients that are used to merge corresponding layer weights, thereby producing domain-specific architectures. In essence, the approach learns weighting coefficients similar to a router in mixture-of-experts models that dynamically determine how to combine the expert parameters based on the input domain. Experimental results demonstrate that this strategy yields promising performance improvements on specialized tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe thank you for your work reviewing our paper and for your comments. We are glad to read that \"The experimental designs are well conceived\", and we propose \"a novel method that is both theoretically grounded and practically relevant\".\n\nWe now try to address your concerns.\n\n- \"*the authors focus solely on next-token prediction as the evaluation metric.*\"\n\nWe have now trained bigger 350M models with 32 experts, and evaluated the corresponding models on MMLU, arc_easy and arc_challenge. The reported accuracy is here (https://anonymous.4open.science/r/rebuttal_soup-E8F9/accs.pdf); we see that the gains obtained in loss also transfer to accuracy.\n\n- \"*the method is not directly compared to other merging techniques.*\"\n\nSee answer to rev.No4u where we compare SoEs to model merging.\n\n- \"*The authors do not adequately compare their approach to existing expert merging techniques (e.g., methods detailed in [0] and [1])*\"\n\nWe will add the following discussion to the paper.\n[0] propose a variant of mixture of experts, where instead of mixing activations, they mix expert weights. However, the routing mechanism is still that of an MoE, based on input samples. On the contrary, our routing mechanism is based on input domains. For instance, one cannot use [0] to instantiate a small model, while it is one of the main features of SoEs.\n\n[1] propose a routing mechanism to merge a bank of Lora weights. Similarly to the previous work, this cannot instantiate a frozen model of small size in a flash, while SoEs can.\n\n- \"*Several key papers that provide additional context to the proposed method were not discussed in the paper. These include*\"\n\nThanks for these useful references, we will discuss the most relevant in the paper.\n\n- \"*The experiments are conducted on relatively small architectures, and given the computational resources available, it would be beneficial to evaluate the approach on larger models (e.g., at least 1B-parameter models) to better assess scalability.*\"\n\nAs discussed above, we have now trained 350M models with 32 experts. \n\n- \"*It is unclear whether the number of tasks used in the expert merging experiments aligns with those used in the Soup-of-Experts experiments, leading to ambiguity about the direct comparability of the approaches.*\"\n\nWe have compared the two approaches (see above). We want to highlight that the both methods are not used in the same way; SoE is a pretraining technique, while merging is used on fine-tuned models. \n\n- \"*model soup is also a specific algorithm [...] It would be beneficial for the authors to clarify these differences when discussing the model soup concept.*\"\n\nIndeed, we will clarify in the text that we simply mean model averaging when speaking of model soups (i.e. we only consider *uniform* soups using the Model soups' paper terminology).\n\n- \"*The captions are overly long*\"\n\nWe will simplify the captions.\n\n- \"*How does your method compare to hypernetwork-based and other mixture-of-experts*\" \n\nAs explained above and in table1, SoEs and MoEs serve different purposes: MoEs cannot instantiate a small model, and they are not aware of the input domain distribution. We will clarify.\n\n- \"*How does the individual performance of a model instantiated via the Soup-of-Experts framework compare to that of a task-specific fine-tuned model on in-distribution data?*\"\n\nWe believe this is the experiment in fig.5, where we compare SoEs to fine-tuned models. \n\n- \"*Is there a universal guideline for selecting the prior used for the dataset weights sampler?*\"\n\nIn our experiments, we used ad-hoc priors that are completely blind to the downstream tasks that are eventually going to be addressed. We still obtain very good performances compared to regular pre-training. Incorporating knowledge about the downstream distribution of tasks is an exciting research direction. \n\n- \"*Given that the number of dataset-specific experts in the baseline (e.g., 64) is much lower than the number of domains used in the meta-soup (4096), doesn’t this disparity affect performance?*\"\n\nWe also tried training models with 64 domains, but the performances where worse than with 4096 domains. The routing mechanism learns to combine those domains in a meaningful way that makes sense for the experts. \n\n- *\"While training 4096 individual models may be computationally prohibitive, the difference in exposure could naturally lead to a higher generalization capacity. Could you provide more explanation on this issue?\"*\n\nWe believe that these models would be over-specific, and have no general knowledge at all. We conducted that experiment with 64 individual models in fig.4, and those models are all very poor generalists. Such a method would therefore perform poorly compared to SoE.\n\nWe thank you again for your detailed review which help us improve the paper ! We hope that our comments have resolved your concerns.", "claims_and_evidence": "The paper makes several key claims, and overall, many of these are backed by experimental evidence. However, a few claims could benefit from additional substantiation.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are well-suited to the problem at hand. The central idea is to train a mixture-of-experts model on multiple datasets by optimizing a router that learns to weight the experts based on the input domain. At inference time, selected expert weights are linearly merged to quickly instantiate a specialist model tailored to a specific domain. The authors evaluate their approach on standard language modeling tasks using the next-token prediction loss", "theoretical_claims": "The proofs are mathematically detailed and build upon well-established properties of Gaussian distributions and clustering. Overall, they appear to be correct under the stated assumptions.", "experimental_designs_or_analyses": "The experimental designs are well conceived and align with the goals of rapid model specialization under a size constraint. However, the authors focus solely on next-token prediction as the evaluation metric, which limits the depth of investigation into the proposed method’s broader performance on downstream tasks. Additionally, the method is not directly compared to other merging techniques. For instance, there is no experimental comparison with alternative merging approaches such as hypernetwork-based methods that dynamically select specialized weights even though the concept appears similar. This leaves an open question as to how the proposed method performs relative to other state-of-the-art merging strategies.", "relation_to_broader_scientific_literature": "The paper synthesizes ideas from model merging, MoE architectures, and importance sampling into a unified framework. It extends prior findings by addressing the computational and scalability challenges inherent in training and serving specialized models, thereby contributing a novel method that is both theoretically grounded and practically relevant. However, the paper could benefit from a more detailed investigation into how its approach compares to hypernetwork-based methods which also enable rapid adaptation by dynamically selecting specialized weights and a more thorough discussion of alternative merging strategies.", "essential_references_not_discussed": "The paper builds on ideas from model merging, MoE architectures, and importance sampling. However, some essential related works could have been discussed more explicitly to contextualize its contributions:\nSeveral key papers that provide additional context to the proposed method were not discussed in the paper. These include:\n\n- **[0] Merging Experts into One: Improving Computational Efficiency of Mixture of Experts:** This work explores methods to merge multiple experts into a single model to enhance computational efficiency.\n- **[1] Mixture of LoRA Experts:** This paper investigates leveraging LoRA-based adaptations within a mixture-of-experts framework with learnable gate.\n- **[2] Mixture of In-Context Experts Enhance LLMs’ Long Context Awareness:** This approach uses a mixture of experts to improve the long-context processing capabilities of large language models.\n- **[3] REMOE: Fully Differentiable Mixture-of-Experts with ReLU Routing:** REMOE proposes a fully differentiable routing mechanism using ReLU, which addresses training stability and efficiency issues.\n- **[4] Mixture-of-Experts with Expert Choice Routing:** This work introduces an alternative routing strategy where experts are chosen based on an expert-choice mechanism rather than traditional softmax gating.\n- **[5] HyperTuning: Toward Adapting Large Language Models without Back-propagation:** HyperTuning explores rapid adaptation methods for large language models without relying on full back-propagation.\n- **[6] HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation:** HINT leverages hypernetwork-based techniques for instruction tuning, enabling efficient zero- and few-shot generalization.\n\nDiscussing these papers would enrich the context of the proposed method by comparing alternative strategies for expert merging, dynamic weight adaptation, and efficient model specialization.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The paper presents an effective strategy to reduce the challenge of generating domain expert models. By learning weights for merging experts into a single specialized model, it provides a computationally efficient alternative to training separate models for each domain.\n- The proposed method is practically sound—it is scalable and designed for rapid model specialization, which is supported by comprehensive experimental results and ablation studies.\n\n**Weaknesses:**\n\n- The experimental evaluation is narrow, as it is limited to next-token prediction metrics. This focus may not fully capture the performance improvements on a broader range of downstream tasks.\n- The authors do not adequately compare their approach to existing expert merging techniques (e.g., methods detailed in [0] and [1]). They present their method as a first without discussing how it outperforms other strategies, including hypernetwork-based approaches.\n- The experiments are conducted on relatively small architectures, and given the computational resources available, it would be beneficial to evaluate the approach on larger models (e.g., at least 1B-parameter models) to better assess scalability.\n- It is unclear whether the number of tasks used in the expert merging experiments aligns with those used in the Soup-of-Experts experiments, leading to ambiguity about the direct comparability of the approaches.", "other_comments_or_suggestions": "See above sections:\n\n- In lines 047–051 (right column), the authors refer to “model soup” as a general term for model averaging in weight space. However, model soup is also a specific algorithm that starts with the best model, then averages with the second best if performance improves, uses that average as a new baseline for averaging with the third model, and so on—skipping any model that does not yield an improvement at a given step. It would be beneficial for the authors to clarify these differences when discussing the model soup concept.\n\n- It would be valuable to compare the proposed method with existing expert merging approaches and evaluate its scalability on larger models.\n\n- The captions are overly long, which makes them hard to read. It would improve clarity if detailed descriptions were moved to the discussion sections of the corresponding experiments instead of being included entirely in the captions.", "questions_for_authors": "1. How does your method compare to hypernetwork-based and other mixture-of-experts merging approaches in terms of performance, generalization, and in-distribution accuracy?\n\n2. How does the individual performance of a model instantiated via the Soup-of-Experts framework compare to that of a task-specific fine-tuned model on in-distribution data?\n\n3. Is there a universal guideline for selecting the prior used for the dataset weights sampler, especially when the eventual application domain of the model is unknown?\n\n4. Given that the number of dataset-specific experts in the baseline (e.g., 64) is much lower than the number of domains used in the meta-soup (4096), doesn’t this disparity affect performance? While training 4096 individual models may be computationally prohibitive, the difference in exposure could naturally lead to a higher generalization capacity. Could you provide more explanation on this issue?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary of the Paper:**
The paper introduces "Soup-of-Experts" (SoE), a novel architecture designed to quickly instantiate small, specialized models for different domain distributions. The architecture consists of a shared base model, multiple expert parameter sets, and a small learned routing function that maps domain weights to a set of coefficients used to linearly combine the experts at inference. The method is evaluated on several language modeling tasks, demonstrating its ability to produce strong specialist models that outperform a single generic model while maintaining computational efficiency.

**Strengths and Weaknesses Highlighted by Reviewers:**
**Strengths:**
  - Clear motivation and problem formulation.
  - Novel combination of existing ideas (model merging and MoE-inspired expert routing).
  - Empirical validation across multiple datasets.
  - Extensive ablation studies providing insights into method robustness.
  - Practical significance due to low computational overhead at specialization time.

**Weaknesses:**
  - Limited theoretical analysis or formal guarantees regarding stability.
  - Potential scalability challenges when significantly increasing the number of experts.
  - Initial experiments were limited to relatively small models (33M - 110M parameters), later extended to 350m models

**Rebuttal and Discussion with Reviewers:**
The authors provided additional experiments with larger models (350M parameters) and demonstrated that the gains in loss also transfer to downstream tasks. They addressed concerns about scalability by showing that the method scales well with larger models and provided comparisons to model merging techniques. The authors clarified the computational cost and the impact of the routing network's capacity on performance. Reviewers acknowledged the authors' responses and updated their assessments positively.

**Recommendation for (Weak) Acceptance:**
We recommend a weak acceptance for this paper. The proposed framework, particularly its joint pretraining the experts and linear combination weights, is a small but novel departure from existing mixture of experts / expert souping methods. The authors have provided substantial empirical evidence to support their claims, and the additional experiments conducted during the rebuttal phase have strengthened the paper. While there are some concerns about scalability and theoretical analysis, the approach has sufficent potential upside to make it an interesting proof-of-point contribution to the conference program.
Decision: Accept (poster)