FlexOlmo: Open Language Models for Flexible Data Use
Authors: name: Weijia Shi, name: Akshita Bhagia, name: Kevin Farhat, name: Niklas Muennighoff, name: Pete Walsh, name: Jacob Morrison, name: Dustin Schwenk, name: Shayne Longpre, name: Jake Poznanski, name: Allyson Ettinger, name: Daogao Liu, name: Margaret Li, name: Dirk Groeneveld, name: Mike Lewis, name: Wen-tau Yih, name: Luca Soldaini, name: Kyle Lo, name: Noah A. Smith, name: Luke Zettlemoyer, name: Pang Wei Koh, name: Hannaneh Hajishirzi, name: Ali Farhadi, name: Sewon Min

We introduce FlexOlmo, a new class of language models (LMs) that supports (1)
distributed training without data sharing, where different model parameters are
independently trained on closed datasets, and (2) data-flexible inference,
where these parameters along with their associated data can be flexibly
included or excluded from model inferences with no further training. FlexOlmo
employs a mixture-of-experts (MoE) architecture where each expert is trained
independently on closed datasets and later integrated through a new
domain-informed routing without any joint training. FlexOlmo is trained on
FlexMix, a corpus we curate comprising publicly available datasets alongside
seven domain-specific sets, representing realistic approximations of closed
sets. We evaluate models with up to 37 billion parameters (20 billion active)
on 31 diverse downstream tasks. We show that a general expert trained on public
data can be effectively combined with independently trained experts from other
data owners, leading to an average 41% relative improvement while allowing
users to opt out of certain data based on data licensing or permission
requirements. Our approach also outperforms prior model merging methods by
10.1% on average and surpasses the standard MoE trained without data
restrictions using the same training FLOPs. Altogether, this research presents
a solution for both data owners and researchers in regulated industries with
sensitive or protected data. FlexOlmo enables benefiting from closed data while
respecting data owners' preferences by keeping their data local and supporting
fine-grained control of data access during inference.