ICML 2025

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Pierre Ablin 1 Angelos Katharopoulos 1 Skyler Seto 1 David Grangier 1
Abstract
Machine learning models are routinely trained on
a mixture of different data domains. Different
domain weights yield very different downstream
performances. We propose the Soup-of-Experts,
a novel architecture that can instantiate a model
at test time for any domain weights with mini-
mal computational cost and without re-training
the model. Our architecture consists of a bank of
expert parameters, which are linearly combined
to instantiate one model. We learn the linear com-
bination coefficients as a function of the input
domain weights. To train this architecture, we
sample random domain weights, instantiate the
corresponding model, and backprop through one
batch of data sampled with these domain weights.
We demonstrate how our approach obtains small
specialized models on several language modeling
tasks quickly. Soup-of-Experts are particularly
appealing when one needs to ship many different
specialist models quickly under a size constraint.
1. Introduction
Large Language Models (LLMs) work well on diverse tasks
because they have many parameters and are trained on gen-
eralist datasets (Brown et al., 2020; Bommasani et al., 2021).
However, they are costly to train and to serve, both in terms
of memory and inference cost.
Specialist language models hold fewer parameters; they
are, therefore, cheaper to store, send, and use at inference.
However, they must give up the generality of LLMs and
specialize in a few specific topics.
In cases where there is an abundance of specialization data,
training a small model on those data yields a good specialist.
However, in many settings, the specialization data is scarce:
for instance, it may come from a narrow topic of interest
1Apple.
Correspondence
to:
Pierre
Ablin
<p ablin@apple.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PM...