FlexOLMo: Open Language Models for Flexible Data Use
Authors: Weijia Shi
Akshita Bhagia
Kevin Farhat
Niklas Muennighoff
Jacob Morrison
Evan Pete Walsh
Dustin Schwenk
Shayne Longpre
Jake Poznanski
Allyson Ettinger
Daogao Liu
Margaret Li
Mike Lewis
Wen-tau Yih
Dirk Groeneveld
Luca Soldaini
Kyle Lo
Noah A. Smith
Luke Zettlemoyer
Pang Wei Koh
Hannaneh Hajishirzi
Ali Farhadi
Sewon Min
Venue: NeurIPS 2025 spotlight ()
Keywords: Language Model
Primary area: deep_learning

Abstract: We introduce FlexOLMo, a new class of language models (LMs) that supports (1) distributed training without data sharing, where different model parameters are independently trained on private datasets, and (2) data-flexible inference, where these parameters along with their associated data can be easily included or excluded from model inferences with no further training. FlexOLMo employs a mixture-of-experts (MoE) architecture where each expert is trained independently on private datasets and later integrated through a new nonparametric routing without any joint training across datasets. FlexOLMo is trained on FLEXMIX, a corpus we curate comprising seven restricted sets, either real or realistic approximations, alongside publicly available datasets. We evaluate models with up to 37 billion parameters (20 billion active) on 31 diverse downstream tasks. We show that a general expert trained on public data can be effectively combined with independently trained experts from other data owners significantly benefiting from these restricted sets (an average 41% relative improvement) while allowing flexible opt-out at inference time (e.g., for users without appropriate licenses or permissions). Our approach also outperforms prior model merging methods by 10.1% on average and surpasses the standard MoE trained without data restrictions using the same training FLOPs. Altogether, FlexOLMo enables training on restricted data while keeping data local and supports fine-grained control of data access at inference.

FLEXOLMO: Open Language Models for Flexible Data Use
Weijia Shi now
Akshita Bhagia
Kevin Farhat *0
Niklas Muennighoff Jacob Morrison 6:0
Pete "
Dustin Schwenk
Shayne Longpre" "n
Jake Poznanski
Allyson Ettinger
Daogao Liu "
Margaret Li
Dirk Groeneveld "
Mike Lewis "
Wen-tau Yih
Luca Soldaini
Kyle Lo Noah A. Smith
Luke Zettlemoyer
Pang Wei Koh awe
Hannaneh Hajishirzi
Ali Farhadi am Sewon Min +ab
Allenn Institute for AI
"Uiierriity of Washington University of California, Berkeley
Staanford University "MIT
sw104190uw edu
akshitabeallenai.com sewonm@berkeley
Model
hf ht.co//allccaa///cxoooooo-2x
Code
github.ccmm/aleenni//eeeeee0
Blog
al aeeeaai.rr//bbog/ffeee
Abstract
We introduce FLEXOLMO, new class of language models (LMs) that supports (1) distributed training without data sharing, where different model parameters are independently trained on closed datasets, and (2) data-flexible inference, where these parameters along with their associated data can flexibly included or excluded from model inferences with further training. FLEXOLMO employs mixture-of-experts (MoE) architecture where each expert is trained independently closed datasets and later integrated through new domain-informed routing without any joint training. FLEXOLMO is trained on FLEXMIX, a corpus we curate comprising publicly available datasets alongside seven domain-speciff sets, representing realistic approximations closed sets. We evaluate models with up 37 billion parameters (20 billion active) 31 diverse downstream tasks. We show that general expert trained on public data can be effectively combined with independently trained experts from other data owners, leading relative improvement while allowing users to opt out of certain data based on data licensing or permission requirements. Our approach also outperforms prior model merging methods by 10.1 % on average and surpasses the standard MoE trained without data restrictions using the presents solution for both data owners and researchers in regulated industries with sensitive or protected data. FLEXOLMO enables benefiting from closed data while respecting data owners' preferences by keeping their data local and supporting fine-graine -raingr control of data access during inference.
average 41%
training FLOPs. Altogether, this research
1 Introduction
Pretraining language models (LMs) typically requires centralized access all data during training and does not have any mechanism to track control the influence of specific data points on model parameters. Model developers must therefore make a one-time decision on which data sources to
*Core contributors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

+
FFN
FFN
Output
FlexOlmo
Altention
Publio Model
Model
BIAD
FFN
Router
Router
Attention
Attention
in
Attention
++++
@ FFN
FFN
Rouber
Input
Attention
An overview of FLEXOLMO. Data owners can contribute without sharing the data by training their own expert modules (FFNs and router embeddings) with a shared public model as an anchor point. At inference, these modules are integrated into a MoE model via a novel router embedding concatenation. This design enables flexible inclusion or exclusion of experts and striet opt-out guarantees, e.g., Github data can be excluded at no cost (blurred) during inference.
Figure I:
Moreover, this centralized approach precludes the of closed data that data owners cannot share with model developers for confidentiality, regulatory, or other reasons. Although solutions have been proposed allow training without sharing the data, such federated learning 3,, their practical adoption remains limited due to performance degradation and the high cost of synchronized training 16. 7
include, with limited ability to remove the effect of certain data after training 2 3
We introduce FLEXOLMO, a new class of LMs that enables distributed training locally maintained datasets while enabling flexible opt-in and opt-out during inference. FLEXOLMO (Figure I) employs mixture-of cactuur-off-xpeeee of-exeerss (MoE) architecture [8, 91. where each expert is trained independently on closed datasets and later integrated into an MoE. This design allows data owners to contribute asynchronously without sharing their data, while also enabling continual updates with new data and providing strong guarantees for data opt-out during inference. Our approach can be seen as an instance of model merging [IO], which merges different models into unified one [1] 12]. However, our model is designed to address the unique challenges in our problem setup combining models pre-trained on completely disjoint datasets with different which makes prior model merging techniques like ensembling output probabilities [II or merging model weights 12 suboptimal.
A key challenge in training FLEXOLMO is ensuring the merging of independently trained experts without joint training. We introduce training algorithm where each data owner independently trains an expert module using the frozen public model as shared anchor (Figure T). This approach teaches independently trained experts to coordinate with the same public model and, by extension, with each other. Additionally, the router, module that determines which experts process each token, typically requires joint training. We address this by assigning each expert a router embedding, initialized from its domain embedding using an off- she-lf embedder [13] and further finetuned its corresponding data during individual expert training. These embeddings are then concatenated to form the router during merging, removing the need for joint training.
To validate FLEXOLMO, we curate a data mixture called FLEXMIX, which includes a public training set along with seven domain-specific sets (c.g., news, educational text, and Reddit). These domains are chosen to simulate scenarios where high-quality data that can benefit LM training is not publicly available.
We train FLEXOLMO first on public data, then extend by merging expert modules trained inde- pendently on our simulated closed sets. While continued pretraining on these sets improves some downstream tasks, it suffers from catastrophic forgetting and inconsistent performance. FLEXOL improves upon the public model by 41% % and also outperforms prior merging techniques such as model soup and ensembling by 10.1 % across 31 downstream tasks. We observe the largest improvements tasks related to closed sets. Notably, even benchmarks where individual
contrast,

---

closed set improved performance over the public model, combining multiple experts yielded signifi- cant gains, demonstrating synergies among independently trained modules. Our qualitative analysis demonstrates that sparse expert activation across layers through the MoF. architecture is key to these gains. Our qualitative analysis shows that the MoE architecture's ability to selectively activate different experts per layer per token is crucial to these gains by combining the strengths of each specialized expert. We hope our work enables research with a broader range of closed datasets for LM training, particularly for organizations interested in collaborating on scientific research through the new features that FlexOlmo provides.
2 Background & Related Work
2.1 Background: Data Restrictions
The standard LM training practice requires model developers aggregate all data centrally and make a one-time decision on which data source to include and exclude. But many real-world data come with sharing and usage restrictions and necessitates (1) model training without data pooling and (2) model inference that can flexibly select different data sources based on use case and access privileges.
Data Sharing Constraints Organizations in regulated industries require LMs that can leverage their closed datasets while maintaining strict data privacy and access controls. Healthcare institutions, financial firms, and other entities possess valuable domain-specific data but cannot share it externally due to HIPAA, GDPR [4 [5], data sovereignty laws [16], and intellectual property (IP) protections. These organizations need training paradigms that enable Al improvement on their sensitive data while ensuring such sensitive data never leaves certain environments and can be removed from the model after training, e.g., when data usage rights expire. In such settings, modular training approaches, where individual experts are trained independently and asynchronously on locally maintained data, essential.
Data Use Constraints The inclusion of certain data depends specific use cases and end users. Privileged access: User-facing applications often involve closed data restricted to specific, authorized users 17.. For example, GitHub Copilot must tailor code suggestions to reflect internal repositories based on an engineer's role and access rights [18]. Copyright and data consent: Legal and ethical considerations on training data for AI are evolving and uncertain [19 20 21 22, 23
24, and often
depend on the data's s intended use, e.g., licenses may prohibit commercial use or limit certain query
types (25 26]. Model control: Training data often include sensitive content (27 28 29 which may be beneficial in certain contexts but harmful in others. For instance, one may want to activate the use of toxic content for toxicity classification in research setting, but deactivate it in applications presented to general audience.
2.2 Related Work
Federated Learning Federated I Learning (FL.) trains single model over distributed datasets by
synchronously aggregating client updates [5] 4 30 Fl. methods range from classical approaches that
iteratively aggregate parameter updates from local clients [1] to parameter-effieient techniques
which have been adapted for LMs [32 33 34], FL can guarantec data privacy using techniques such
as homomorphic encryption 355 and differential privacy (DP) [36. However, FL has seen limited adoption in LM training due to the high cost of synchronization and performance degradation [6 7 and remains susceptible to privacy attacks due to inter-client communication 38).
Similar to 39, our approach avoids data sharing but differs fundamentally by supporting independent, asynchronous training without costly inter-client communication, and allowing real-time opt-in and opt-out. Like FL., our model allows data owners to optionally apply DP training locally for privacy guarantees. Because DP is orthogonal to our architecture, each contributor can independently choose whether to apply it, providing flexibility without compromising the overall design.
Model Merging Our work builds on recent efforts 40 10 that advocate for developing machine learning models like open-source softwares, where sub-parts of the model can be trained independently and subsequently merged into unified systems. This can be achieved through various methods, including weight merging, output ensembling, and expert routing. Model soup-merging model weights trained on different datascts from the same initialization can boost performance [4] 12
3

---

42 43 44
especially with weighted combinations 45
46
47
50 5T] Weighted output 48
49
ensembling (e.g., BTM [52 (31) is also effective when models are trained on distinct datasets initialized from the same seed model. These approaches can be applied to our setting, where each expert independently trained starting from the same public model then merged into a unified one. Our experiments ($5) show that these methods are less effective, primarily because they lack learned connections between different modules, which constrains the expressivity of the resulting models.
An alternative line of work focuses on expert routing methods, such as DEMix S4 BTX SS and its extensions [6 (7.58))
[58] which merge dense, independently trained experts into a mixture-of-expens (MoE) framework. We draw inspiration from this work, as we also integrate independently trained models into MoE. However, these methods require joint training on union of all datasets used in expert training after merging. By contrast, FLEXOLMO removes the need for joint data access to enable training on locally maintained datasets. Our work is also related to ModuleFormer (59), which induces sparse modularity from uncurated data using novel load balancing and concentration losses.
Related efforts in parameter-efficiee training have explored merging LoRA adapter weights trained on separate datasets [60] 61 62 633, particularly to reduce communication overhead in collaborative
settings and support fine-graing ane-grained data access control and opt-out use cases 64 63 Unlike these
methods, which focus on merging lightweight adapters, our approach merges full expert models into a standard MoE architecture.
Mixture-of-Experre (MoE) MoE models 8 9 66]. consisting of many small feedforward networks called experts, have gained popularity for their training and inference efficiency. Our work leverages the MoE architecture; however, our motivation and training method are fundamentally different as our primarily goal is to support modularity rather than efficiency.
3 FLEXOLMO: LMs with Flexible Data Use
3.1 Problem Setup
Let Mpub be a model trained on a publicly available dataset Dpob, and D = D2.... Dn3 represent collection of locally maintained datasets with separate owners. Our objective single model M which is constructed via composing Mpub and set of modules (M1, My...., .bb...,,M,, Mn).
where each M; is independently trained by the owner of D,, who also has access Mpub.
This model satisfies two requirements: (1) training Minal does not require anyone to have joint access to the full dataset collection D, each M, is trained independently by the owner of dataset D;; (2) removing any module M; from Minal guarantees complete removal of its associated data D,
The key modeling challenges are: (1) to develop algorithm that creates M; using D, and M Mpub- and (2) to design the merging algorithm that combines Mpab, M1, M2...., Mn into Minal-
3.2 Model Architecture
FLEXOLMO follows the standard MoE architecture: it replaces the feedforward network (FFN) in each transformer block with router and n small FFNs called expert modules [Mpub. MI, M,, Note that we omit the layer index for each expert in our notation for simplicity. Given a processed input token embedding x E IRA, the MoE module computes output representation y:
Σ (r()))
softmax(r(x)))
where the router function computes the expert probabilities from Unlike standard MoEs where experts are trained jointly, our experts are trained asynchronously on distinct datasets (D1,... D,)-
3.3 Training Algorithm
Standard Moïs train all experts and the router jointly on all data. In contrast, FLEXOLMO trains experts independently by teaching them coordinate ($3.3.T) and merges them inference using a domain- informed router ($3.3.2.. Optional router tuning can further improve performance ( 3.3.3

---

3.3.1 Training Experts to Coordinate
A straightforward way to train each expert would be to directly continue to train each expert M; on its own data D; [52]. We found that this method causes the experts diverge too much from one another and from the original seed model, which makes merging after isolated training difficult.
To prevent such divergence, Figure T). We use Mpub as an anchor that teaches experts to coordinate with Mpu and, by extension, with each other. Specifically, during training, for dataset D, we construct a MoE model with two expert modules-both initialized from the same FFNs from Mpub. During training. we freeze Mpub expert and the shared attention layer, while the other expert (A4) is trained D.. As each data owner updates only their own FFNs while keeping all other parameters (those inherited from Mpub such as attention layer) frozen, the learned FFNs are designed to naturally coordinate with each other later during merging at inference time. Importantly, with this approach, a router is learned so that each expert can be integrated into MoE architecture without additional training (details $3..22
train experts independently while teaching them to coordinate
3.3.2
Router
The router plays a critical role in MoE: the router function r maps an input vector x to a distribution over expert modules, including the public model as of the experts:
rxx) W,x W, ((++1)xh
In typical MoEs, W, is trained end-to-end alongside all expert modules, using access to the full training dataset. Instead, we decompose W, into individual expert-specific router embeddings, where each r; represents the router embedding for expert M,, learned only from D,
r pub
W,
where r;
Σ d.ES.
D..
JS
In
These router embeddings can be initialized by averaging domain speccifc embeddings of samples from each D, obtained by encoding subsets of data using an off-the-shelf embedder E [67 that maps a document into an h-dimensional vector. This method is motivated by prior model merging
work that leverages domain embeddings for routing 68 69 70
72 73
During coordinated training of experts ($3.3.1, we learn the router embeddings in pairs: [Fpub. r:] The public embedding Tpub remains frozen across all experts, while r; is finetuned separately alongside the parameters of M4. At inference time, merging the expert embeddings into the complete router matrix W, , directly integrates all expert modules into one unified MoE. Furthermore, experts can be flexibly added or removed by simply adding or removing their corresponding router embedding.
Adding a Bias Term Unlike standard router learning that is learned among all experts jointly. coordinated training of experts only learns pairwise routing decisions between one expert and the public model. This means the model never directly compares experts Mj and M, during training, potentially limiting generalization during inference. To alleviate this issue, we add a negative bias term by for each independent trained expert (M1 M,,..., .,,.. We select expert M; when:
r; + b: Fpub x Vi (1.2....,
Otherwise default to Mpab- This helps the later merging process, where each expert competes not just with the public model but with all other experts. Further details and justifications are provided in $
3.3.3 Optional Router Training on Proxy Data
With our proposed model design, expert modules can be merged without any additional training. However, if data owners are willing to identify proxy samples within the public dataset Mpab that resemble their closed data, we can optionally perform lightweight router tuning step after merging, using only public data from Ppun- Specifically, we assume each data owner selects a small proxy set
D, C D Dpab. where D < 0.01 x Di), chosen approximate the distribution of their closed dataset D.. While D, is too small train expert modules, it still provides useful signals for improving router quality. To construct D, we train a binary classifier to distinguish D; from Dpub and select public samples with the highest predicted likelihood of belonging to D.. After merging, we tune the router embeddings r., Fn, Tuu the combined set D DD,, and Dpab sampled uniformly.
5

---

Experimental Setup
4.1 Training Data: FLEXMIX
Our corpus comprises single Public Mix and closed sets, either real simulated, which designed to be disjoint from each other. Figure i $B provides the statistics.
Public Mix represents general web text based on Common Crawl (CC) Specifically, we took the Baseline version of DCLM (74), excluding news and creative writing content (described below).
This represents a public dataset that can be used without restrictions.
News includes news content from DCLM-Baseline, obtained by applying the classifier from [75 and selecting documents classified as News Articles. While included in CC when downloaded, many of the original sources are subject to closed access (20
Creative Writing includes creative content from DCLM-Baseline, obtained by applying the classifier from 751 and selecting documents classified as Creative Writing.
Code includes code repositories from Starcoder 76 77 with additional quality filtering as in 78
Academic includes open-access academic papers obtained from [79]; these are papers from 80
81] but re- processed using olmOCR (79 for cleaner plain text.
Educational Text includes educational text from digitized PDFs, converted to plain text using olmOCR [79].
Math includes math-relevant content, including web pages about or using math and math problem sets, obtained by combining Dolmino Math Mix [78] and FineMath4+ [82].
Reddit contains posts and comments originally sourced and released by Dolma (83) further filtered and processed to improve quality (details in Appendix [B). As of this writing, this Reddit data is no longer unrestrictedly downloadable due to Reddit's 2023 policy change
These seven sets are designed to represent datasets with at least one the following characteristics: (I) historically closed and not publicly available; (2) previously publicly available but now closed; or (3) domains with scarce high-quality public data.
4.2 Evaluation
We evaluate our models and baselines on large and diverse collection of well-established benchmarks, consisting of 31 tasks across 10 categories, broadly grouped into (1) general-purpos LM benchmarks and (2) domain-specific evaluations. More details are provided in SC
General-purpose Evaluation
We report results on (1) MC9, nine multiple-choice datasets in-
cluding ARC-Easy [84), ARC-Challenge (84, BoolQ 85, CSQA (86). HellaSwag [7]. Open-
BookQA 88 PIQA 89], SocialIQa [0], and WinoGrande 91], (2) GEN5, five generative tasks
including CoQA 1921, SQuAD 193, Natural Questions 194, TriviaQA 195) and DROP 196). as well
as (3) MMLU 197.. (4) MMLU-Pro (88, (5) AGIEval (99 consisting of tasks from college
admission tasks, and (6) BBH (TO) consisting of 23 challenging BIG-Bench tasks.
Domain-specific Evaluation While peneral-purpose evaluation benchmarks already include some math assessment, we further evaluate math ability using (6) Math2, which encompasses two special- ized math benchmarks: GSM8K [0I and MATH 102] To evaluate coding capabilities, we use (7) Code4, coding benchmarks including MBPP 103
MBPPPLUS 104, HUMANEVAL [105], and HUMANEVALPLLUS [T04]. To measure scientific literature understanding. we report on (8) SciRIFF5: comprising 5 subtasks from SciRIFF [166]. Finally, we include (9) NewsG: news generation and (10) PoemG: poem generation tasks, both evaluated using LM judge.
4.3 Baselines
We compare our method against several baselines, either taken directly from prior work minimally adapted our problem setting. All baselines, except for "Unrestricted training,' train set of dense
https: http:///comenncraaaaoo org
Reddiit's s 2023 policy change restricts third-party access and use of its data, including for language model development; see Sytinessoom/0033//////ccnnnllylo oog.gg//eeeee..ii.oooooooooooooe google..mm

---

models independently by continuing pretraining from the public model Mpub on each simulated closed set, without architectural changes, and merge them using model merging techniques.
Prompt-based Routing We use LM-based domain classifier via prompting to route each query to the most suitable model, which then used exclusively. We use Llama-3. 1aama--..-8B---- 1-8B-Instruct [107] and OLMo-2-1124-7B--mm Instruct 78] classifiers. More details are in $A.1
Model Soup We apply both average and weighted parameter averaging across all models, following [1]. The weights are derived by applying a softmax over the log-likelihodds cach model on the test example input.
(BTM) We follow BTM [I. which ensembles models by computing weighted average of their output probabilities. Weights are obtained via softmax over the log- likelihoods of the test example input of each model. As in the original BTM, ensembling can be restricted to the top-k models by zeroing the weights of all other models and renormalizing. See $ for full details.
Tranch-Train-Vee
BTX We follow BTX [55] which upcycles an MoE from independently trained dense models. copies the dense model parameters to the corresponding experts in MoE while averaging non-expert parameters such as attention layers for merging. The original BTX requires training all model parameters combined datasets after merging. To approximate it closely possible while adhering to our setting, we perform this post-merge training on the public set only.
Unrestricted MoE assess how closely our method approaches the benefits full data access while preserving data separation, we construct upper-bound reference model: sparse MoE initialized from the public-only dense model and trained on the combined dataset, including all closed sets and Public Mix. As MoE training incurs roughly 2x the FLOPs of our approach for the same data size, we report both compute-controlles (1x FLOPs, 0.5x data) and data-controlled (2x FLOPs, 1 x data) comparisons.
4.4 Training Setup
For the public model Mpub. we use a dense model with 7 billion parameters following the OLMo 2 architecture (78). This model contains 32 layers with hidden dimension 4,096 and is trained on our public mix for 1 trillion tokens. Following [78], we use learning rate of 0.0009 and the AdamW optimizer with parameters 31 0.9 and fo 0.95 and a cosine learning rate scheduler. The public model is pretrained using 512 H100 GPUs with global batch size of 4 million tokens for three days.
Each data owner then takes this checkpoint and performs on their own data (totaling 400B tokens across all experts). For the optional router training, we use billion tokens in total. The final FLEXOLMO, trained sets, has 37 billion total parameters with 20 billion active (4 active experts out of 8). More details can be found in $A.2
for 50 billion tokens
5 Results and Analysis
We conduct ablation studies and compare against a comprehensive set of baselines at small scale with four experts-Public mix, math, educational text, and code (Table Ip. We then evaluate our final model on the full setup including the Public mix and all seven simulated closed sets TTalle 2}. Finally, we present an in-depth analysis illustrate the behavior and effectiveness of FLEXOLMO.
5.1 Main Results
Individual experts excel at their specialized tasks As shown in Table experts trained on each domain-specific set demonstrate strong performance in their specialized domains: the Math expert achieves the highest scores on math tasks, while the Code expert performs the best coding benchmarks. However, these experts exhibit considerable performance degradation when evaluated on tasks outside their domains. Notably, the Code expert performs poorly on general benchmarks.
FLEXOLMO outperforms individual experts FLEXOLMO outperforms individual experts in most cases. It improves upon the model trained solely on public data, achieving an average 41 % relative gain. Largest improvements appear on benchmarks where closed data significantly boosts
7

---

Table Evaluation of FLEXOLMO trained on four sets (public mix, math, educational text and code).
tested on 24 tasks with 100 samples subtask.
MC9
GENS
MMLU MMLU Pro
AGI Eval
BBH
Math2
Code4
Avg.
Prev. Public model
68.4
58.8
57.0
27.1
39.0
35.6
8.1
1.0
36.9
Individual experts
Math
63.8
46.3
51.1
24.0
40.7
45.4
50.4
42.5
Code
38.7
41.4
30.0
14.6
29.0
38.2
6.0
22.4
27.5
Educational Text
63.0
52.8
57.7
26.8
39.6
40.0
13.1
4.3
37.2
Prior model merging work
Model soup (average)
53.8
54.7
42.4
17.5
8.2
39.6
Model soup (weighted)
69.6
56.3
58.6
30.5
45.4
43.5
18.5
14.8
42.2
BTM
69.0
58.5
59.6
29.0
43.6
43.6
21.2
22.3
43.4
Prompt-hase hased routing (router: OL.Mo)
59.9
48.8
50.0
25.4
38.7
41.3
41.5
20.7
40.8
Prompt-boe based routing (router: Llama)
64.2
53,4
57.7
26.4
39.9
39.8
21.5
17.3
40.0
BTX
69.6
57.9
56.2
28.5
43.1
41.3
16.8
6.4
40.0
Ours
FLEXOLMO optional training)
71.1
58.6
58.1
28.4
44.8
43.4
51.5
18.2
46.7
bias
67.9
55.6
57.5
28.6
43.9
45.5
50.0
17.6
45.8
no domain embedding init, bias
70,0
55.4
56.1
25.9
41.1
44.9
44.9
16.6
44.4
training to coordinate
64.4
51.5
55.5
24.7
43.1
41.2
19.3
10.3
38.8
FLEXOL.MO
71.0
59.8
59.9
30.8
45.8
47.1
50.7
17.3
47.8
Reference model (upperbound)
Unrestricted MoE. FLOPs,
68.0
53.8
57.8
28.9
41.5
48.6
49.4
46.3
Unrestricted MoE FLOPs, Data)
73.3
60.2
63.1
32.5
48.1
54.4
53.4
27.0
51.5
Table 2: Evaluation of FLEXOLMO trained on eight sets (public mix and seven simulated closed sets) on 31 tasks across 10 categories, tested with 1,000 samples per subtask. "no RT" indicates no optional router training on proxy data ($3.3.3).
MC9 GENS MMLU MMLU Pro AGlEval BBH Math2
NewsG PoemG SciRIFF'S Code4 Avg.
Prev. Public model
68.7
58.8
55.9
26.2
39,9
35.7
76.0
47.8
48.1
1.1
42.4
Individual experts
Math
62.5
44.3
50.6
24.1
42.0
45.6
53.1
42.6
28.0
50.7
41.8
Code
40.5
39.4
29.5
14.5
27.4
38.1
45.1
28.2
Educational Text
64.3
52.1
56.5
27.0
39.7
40.3
13.6
57.6
51.8
51.7
3.0
41.6
News
46.5
48.6
36.4
15.2
25.7
30.9
2.5
77.7
26.9
47.0
0.0
32.5
Creative Writing
42.7
43.9
31.5
11.6
23.3
27.6
56.9
67.5
42.4
0.0
31.7
Academic
41.0
45.2
33.8
14.8
24.1
32.4
$1.8
23.0
0.0
29.5
Reddit
64.7
36.5
56.1
25.5
35.5
19.7
2.5
54.1
8.6
32.7
1.7
30.7
Combined model
BTM (top-2)
68.7
$7.7
59.4
28.3
43.2
44.3
23.1
73.6
54.4
46.3
24.0
47.6
FLEXOLMO (no RT)
69.2
53.2
58.8
34,0
43.4
42.1
52.1
78.2
60.1
54.4
18.6
$1.3
FLEXOLMO
70.8
59.8
60.4
30.9
45.1
46.4
48.5
80.7
62.2
54.3
52.4
individual expert performance, e.g., 35.6 47.1 BBH, 8.1 -> 50.7 math, and 1.0 17.3
coding. Notably, FLEXOLMO even matches or exceeds the performance of specialized experts on their respective tasks (c.g., on BBH and Math2).
FLEXOLMO achieves more effective merging than baselines We also compare FLEXOLMO with baseline merging methods ($4.3). All baselines outperform the model trained on Public Mix only. However, their performance is inconsistent: model soup and BTX are generally weak F while prompt-based routing is highly unstable: it performs well when the classifier selects the correct expert, but degrades sharply when it does not. Among the baselines, BTM yields the best performance. Nonetheless, FLEXOLMO outperforms all prior model merging methods, beating the best baseline BTM by 10.1 1% relative on average. We attribute this to the MoE-based design of our model, which selectively activates different experts per layer, effectively combining the complementary strengths of each specialized model (see further analysis in (5.2).
"This is likely because training disjoint datasets causes experts to diverge from each other and from the seed model, making model soup limited, and training BTX the public data only not optimal.

---

Table 3: Impact embedding initialization methods on model performance. The GRIT embedder 67 consistently outperforms public model embeddings across most benchmarks.
Embed Init.
MC9
GENS
MMLU
MMLU Pro
AGI Eval
BBH
Math2
Code4
Avg.
FLEXOLMO
Public
70.5
55.5
58.3
26.5
40.2
40.1
48.4
8.7
43.5
FLEXOLMO
GRIT
71.1
58.6
58.1
28.4
44.8
43.4
51.5
18.2
46.7
Textbook
Writing
Layer
Layer
Layer 32
120
Routtng probability
0
120
120
code
Mustie
tarebook
Creation
Expert
Figure 2: Routing pattern analysis. We visualize how text from different domains activate experts (four experts activated). The horizontal gray lines indicate uniform routing.
Comparison to the unrestricted MoE Compared to the unrestricted MoE trained without consider- ing data restrictions ($4.3). FLEXOLMO outperforms the FL.OP-contoolled setting (1 x FL.OPs, 0.5x Data). It slightly underperforms the data-controlled model (2x FLOPs, x Data). This indicates that FLEXOLMO enables training without direct access to the data (requiring only model sharing) and flexible opt-in and opt-out functions while retaining strong performance.
Ablations FLEXOLMO We further evaluate FLEXXOLOO by removing different components introduced in $3.3 learning to coordinate, router initialization, and the bias term. Our results show that each component plays important role, with the removal of any one leading performance drop. In particular, we observe that randomly initializing router embeddings leads to the final learned router embeddings being very similar to each other, making the later merging of multiple experts harder. Furthermore, we confirm that FLEXOLMO benefits from additional router training ($3.3.3) and using external embedders for router initialization, compared using the public model's 's hidden states as the initialization (Table 3).
Final FLEXOLMO in the full setup Finally, we evaluate FLEXOLMO in the full eight-expert setup and compare it against the public-only model, individual experts trained on closed datasets, and BTM (top-2), the strongest baseline from Table 1] This was done by simply adding four additional experts, benefiting from FLEXOLMO's flexibility in easily adding new datasets. Consistent with earlier findings, FLEXOLMO outperforms all individual models (the public baseline and individual experts), demonstrating the synergistic effect of combining independently trained modules. Compared to the strongest bascline BTM, it achieves 10% relative improvement average (ablle 2). FLEXOLMO excels on benchmarks where specialized experts perform well (BBH, Math2, NewsG, PoemG, SciRIFF5, Code4), matching surpassing the experts, and also shows strong results on tasks where no single dataset suffices (e.g., MC9, Gen5, MMLU, MMLU Pro, AGI Eval).
5.2 Model Behavior Analysis
Routing patterns Figure 2 visualizes the router's token distribution across experts for various domain imputs. The router tends to activate the corresponding domain expert (e.g., math inputs activate the math expert), demonstrating its ability to identify the most relevant module. We also observe frequent activation of the public expert, likely due to our coordinated training strategy, where each expert is designed to complement the public expert. Also, different combinations of experts are activated at different layers. This highlights the model's layer-specific specialization and its greater expressivity than approaches that route inputs to single expert (c.g., prompt-based routing).

---

60.0
Performance (%)
80
80.7
Full model
71.5
70.8
70.2
Opt out News expert
MMLU (
57.5
60
50.2
55.0
4.
52.5
20
17.2
NewsG
Code Evaluation metrics
Math2
Number of active experts
Figure 3: Effect of active expert count
Figure 4: Opting out of news data.
Number of active experts We further analyze how the number of active experts affects downstream task performance. As shown in [igure 3) performance consistently improves as more experts are activated, up to four experts, after which plateaus. This suggests that the final model can operate efficiently as sparse model by actívating only four experts per input during inference.
Data opt-out FLEXOLMO offers straightforward mechanism for opting out of specific datasets by removing the corresponding expert module at inference time. In Figure 4 we evaluate the model's performance after excluding the news expert. As expected, performance drops on in-domain tasks such as news generation. However, on unrelated tasks, performance remains largely unaffected.
5.3 Data Extraction Analysis
FLEXOLMO enables data owners contribute to the model without sharing their data by instead sharing model weights trained locally on their data. A natural and important concern is whether their data can be extracted from these shared weights [T08] 109 [1O]. This risk is particularly relevant when training data includes private or confidential information.
We empirically assess this risk by implementing training data extraction following prior work [108] Specifically, we sample 10,000 documents from the math data x 32-token prefix and use it generate 256-token continuation using top-k sampling (k 50), top-p sampling (p = 0.95), and a temperature 1.0. We sample times per prefix, and if any of the generated outputs achieves normalized Levenshtein similarity of 0.9 or higher with the original document, we consider that document to be extracted. To validate our implementation. we apply it to model overfitted on the dataset (trained for 100 epochs), and observe 60% extraction rate.
From each document, we extract
Our results are as follows: (1) public model that has not seen any math data yields an extraction rate of 0.1 %. (2) A dense model trained on the math dataset (i.e., math expert) yields 1.6%. (3)
FLEXOLMO with the math expert included yields 0.7%.
These results lead to the following conclusions. First, in practice it is difficult to extract substantial portion of the training data, which line with previous findings (108). However, if model includes any weights trained on the data, nonzero (though small) fraction of the data may be extractable. If data owners are comfortable with this minimal leakage, as long as a meaningful fraction of the data remains not extractable, we believe that FLEXOLMO, in its current form, is a viable solution. If the owners' data includes any private or sensitive information, we recommend training experts using differentially private (DP) learning methods before contributing them to the model, which provides formal privacy guarantees. Applying DP is orthogonal to our architecture, and different data owners can make independent decisions, providing flexibility without compromising the overall design.
6 Conclusion
We introduce FLEXOLMO, a new class of LMs that solves real- world data constraint challenges with (1) modular, distributed training, where different model parameters are independently trained on disjoint and locally maintained datasets, and (2) dlta-flexible inference, where data can be selectively included at inference-time, with guarantees. We show that FLEXOLMO significantly outperforms competitive baselines, while providing the benefits of distributed training and flexible inference.
*We chose the math data because it is the smallest among our simulated closed sets and the math expert trained for three epochs (instead of one), making it more susceptible extraction. Therefore, the extraction rates with the math data likely represent upper bound.
10

---

Acknowledgements
We thank Preston Jiang, Colin Raffel, Percy Liang, Matei Zaharia, Peter Henderson, David Q. Sun, Kevin Kuo, Virginia Smith, and Ai2 members for valuable discussion and feedback.
SM was supported in part by grant from DARPA the Simons Institute for the Theory of Computing. PWK was supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Information under the Al Visiting Professorship Programme (award number AIVP AIVP-2024-001), and by the AI2050 program at Schmidt Sciences.
References
[1] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling. 2024.
[2] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six- way evaluation for language models. arXiv preprint arXiv:2407.06-400 2024.
[3] A Feder Cooper, Christopher A Choquette- Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, et al. Machine unlearning doesn't do what you think: Lessons for generative ai policy, research, and practice. arXiv preprint 2rXiv:2412.06966, 2024.
[4] Jakub Konecný, Brendan McMahan, X Yu Felix, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. CoRR, 2016.
[5] Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera y Arcas. Communiiattoon-Fifiiieee Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273-1282. PMLR, 20-22 Apr 2017.
[6] Kang Wei. Jun Li, Ming Ding, Chuan Ma, Howard Hua Yang. Farokhi Farhad, Shi Jin, Tony Q. S. Quek, and Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions Information Forensics and Security, 15:3454- 3469, 2019.
[7] Xiaojin Zhang, Yan Kang, Kai Chen, Lixin Fan, and Qiang Yang. Trading off privacy, utility, and efficiency federated learning. ACM Transactions on Intelligent Systems and Technology, 14:1 - 32, 2022.
[8] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of mixture-of-expprts layer. arXiv preprint arXiv: 066383888..., 2017.
[9] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of- experts language models. arXiv preprint arXiv:2409. 2024.
[10] Prateek Yadav, Colin Raffel, Mohammed Mugeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. CoRR, 2024.
[11) Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models, 2022.
[12] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith.

---

et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International conference on machine learning, pages 23965-23999 PMLR, 2022.
[13] Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning, 2024.
[14] BIS Innovation Hub Nordic Centre. Project aurora: the power of data, technology and collaboration to combat money laundering across institutions and borders. Technical report. Occasional publication No. 66.
[15] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge- Maaeledg--Bssedd Systems, 216:106775,, 2021.
[16] Patrik Hummel, Matthias Braun, Max Tretter, and Peter Dabrock. Data sovereignty: A review. Big Data & Society, 8(1 6(1:20539517229982022 2021.
[17] Martin Grund, Stefania Leone, Herman van Hövell, Sven Wagner-Boysen Sebastian Hillig, Hyukjin Kwon, David Lewis, Jakob Mund, Polo-Francois Poli, Lionel Montrieux, et al. Databricks lakeguard: Supporting fine-grained access control and multi-user capabilities for apache spark workloads. In Companion of the 2025 International Conference Management of Data, pages 418-430, 2025.
[18) Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simis- ter, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium Machine Programming, pages 21 21-29, 2022.
[19] Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24(400):1-79, 2023.
[20] Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. Consent in crisis:
The rapid decline of the ai data commons. Advances in Neural Information Processing Systems, 10000002-180088 2024.
[21] Ziv Epstein, Aaron Hertzmann, Investigators of Human Creativity, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R Frank, Matthew Groh, Laura Herman, Neil Leach, et al. Art and the science of generative ai. Science, 380(6650): 1111-111111111111111111111111111 2023.
[22] Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej. Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index. arXiv preprint arXiv:2310. 2023.
[23] Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and copyright questions, 2023.
[24] Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marmu Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt D. Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, and Sara Hooker. A large-scale audit of dataset licensing and attribution in ai. Nat. Mac. Intell., 6(8):975-987) 987, 2024.
[25] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprini 100.0220887772 2021.
[26] Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing attribution in ai. arXiv preprint arXiv:2310.16787 2023.
12

---

[27] Abeba Birhane, Sanghyun Han, Vishnu Boddeti, Sasha Luccioni, et al. Into the laion's den: Investigating hate multimodal datasets. Advances in neural information processing systems, 36:21 20221268-210 268-21284, 2023.
[28] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, 2021.
[29] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations.
[30] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, 14(1-2):1 1-:1-110111111-11111111001011111 2021.
[31] Tian Li. Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-4555 2020.
[32] Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang, and Guogang Zhu. Fedlora: When personalized federated learning meets low-rank adaptation. 2024.
[33] Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-effiieenn and privacy- privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5. IEEE, 2023.
[34] Kevin Kuo, Arian Raje, Kousik Rajesh, and Virginia Smith. Federated lora with sparse communication. arXiv preprint arXiv:2406.05233, 2024.
[35] Pascal Paillier. Public-key cryptosystems based composite degree residuosity classes. In Proceedings of the 17th International Conference on Theory and Application of Cryptographic Techniques, EUROCRYPT 99, page 223-238, Berlin, Heidelberg, 1999. Springer-Verlag
[36] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 933-)):11-4077 2014.
[37] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16070-16084. Curran Associates, Inc., 2020.
[38] Erfan Darzi, Florian Dubost, Nanna M Sijtsema, and Peter MA van Ooijen. Exploring adversarial attacks in federated learning for medical imaging. IEEE Transactions on Industrial Informatics, 2024.
[39] John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International conference on artificial intelligence and statistics, pages 3581-3607. PMLR, 2022.
[40] Colin Raffel. Building machine learning models like open source software. Communications of the ACM, 66(2):38 400888400000000 2023.
[41] Shangbin Feng. Zifeng Wang, Palash Goyal, Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia Tsvetkov, Chen-Y Yu Lee, and Tomas Pfister. Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems, 2025.
13

---

[42] Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027, 2023.
[43] Jyothish Pari, Samy Jelassi, and Pulkit Agrawal. Collective model intelligence requires compatible specialization. ArXiv, abs/2411.02207, 2024.
[44] Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, al. Model swarms: Collaborative search to adapt Ilm experts via swarm intelligence. arXiv e-prints, pages arXiv 2410, 2024.
[45] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties- merging: Resolving interference when merging models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 7093-7115. Curran Associates, Inc., 2023.
[46] Michael Matena and Colin A Raffel. Merging models with fisher-weighted averaging. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 177061 Curran Associates, Inc., 2022.
[47] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
[48] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403. 13188, 2024.
[49] Enneng Yang, Zhenyi Wang. Shen, Shiwei Liu, Guibing Guo, Xingwei Wang. and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
[50] Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Pang Wei Koh, Jesse Dodge, and Pradeep Dasigi. Merge to learn: Efficiently adding skills to language models with model merging, 2024.
[51] Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, Tomas Pfister, Yejin Choi, and Yulia Tsvetkov. When one Ilm drools, multi-llm collaboration rules, 2025.
[52] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03:: 2022.
[53] Suchin Gururangan, Margaret Li, Mike Lewis. Weijia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery. arXiv preprint 1xXix23033.1477 2023.
[54] Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5557-5576, 2022.
[55] Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-train-mix: Mixing expert Ilms into a mfxture--o-eexperts 1lm. arXiv preprint arXiv:2403. 0771077866. 2024.
[56] Peter Schafhalter, Shun Liao, Yanqi Zhou, Chih-Kuan Yeh, Arun Kandoor, and James Laudon. Scalable multi-domain adaptation of language models using modular experts. arXiv preprint arXiv:2410.1073 2024.
14

---

[57] Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, et al. Bts: Harmonizing spe- cialized experts into generalist llm. arXiv preprint arXiv:2502.00005. 2025.
[58] Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sebastian Ruder, A. Ustun, and Acyr F. Locatelli. Bam! just like that: Simple and efficient parameter upeycling for mixture of experts. ArXiv, 08827 2024.
[59] Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Mod- uleformer: Modularity emerges from mixture-of-experts arXiv preprint arXiv:2306.0466 2023.
[60] Ted Zadouri, Ahmet Üstön, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint wwXiv:2309005444 2023.
[61] Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo, Jindong Chen, and Lei Meng. Sira: Sparse mixture of low rank adaptation, 2023.
[62]
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang. Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1932-1945, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
[63] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückléé Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: arXiv:2005.0022 2020.
task composition for transfer learning. arXiv preprint
[64] William Fleshman, Aleem Khan, Marc Marone, and Benjamin Van Durme. Adapterswap: Continuous training of Ilms with data removal and access-control guarantees. CoRR, 2024.
[65] Kevin Kuo, Amrith Setlur, Kartik Srinivas, Aditi Raghunathan, and Virginia Smith. Exact unlearning of finetuning data via model merging scale. arXiv preprint arXiv:2504004626, 2025.
[66] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models. arXiv preprint arXiv:240J.0660 2024.
[67] Niklas Muennighoff, Hongjin SU, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Conference on Learning Representations, 2025.
[68] Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker, and Ahmet Üstün. Nexus: Specialization meets adaptability for efficiently training mixture of experts. arXiv preprint MXXiv:2408159001 2024.
[69] Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang. Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. arXiv preprint arXiv:2402.09999, 2024.
[70] Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In International Conference on Machine Learning, pages 14702-14729, PMLR, 2023.
[71] Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Bertasius. Dam: Dynamic adapter merging for continual video qa learning. In 2025 IFEF/CVE Winter Conference on Applications of Computer Vision (WACV), pages 6805-68 17. IEEE, 2025.
15

---

[72] Hyunji Lee, Luca Soldaini, Arman Cohan, Minjoon Seo, and Kyle Lo. Routerretriever: Routing over a mixture of expert embedding models. arXiv preprint arXiv:2409.02653 02685, 2024.
[73] Joshua Belofsky. Token-level adaptation lora adapters for downstream task generalization. In Proceedings of the 2023 6th Artificial Intelligence and Cloud Compuling Conference, pages 168-172,77 2023.
[74] Jeffrey Li, Alex Fang. Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng Yu Hsieh, Dhruba Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2025.
Ghosh,
[75] Alexander Wettig. Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation, 2025.
[76] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier. João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Naday Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao. Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf. Arjun Guba, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023.
[77] Denis Kocetkov, Raymond Li. Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack; tb of permissively licensed source code, 2022.
[78] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda. Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025.
[79] Jake Poznanski, Jon Borchardt, Jason Dunkelherger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. olmocr: Unlocking trillions of tokens in pdfs with vision language models, 2025.
[80] Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical re- port, Allen Institute for AI, 2023. ODC-By.httts:..... http://///ggtttu.b.coaaaaeee.aa
[81] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
Online, July 2020. Association for Computational Linguistics.
16

---

[82] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez. Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydliček, Agustin Piqueres Lajarin, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model, 2025.
[83]
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha. Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafford, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024.
[84] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. 1vxiv:180300545700 2018.
[85] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
[86] Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph, 2018.
[87] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019.
[88] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. EMNLP, 2018
[89] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
[90] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019.
[91] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at scale. arXiv preprint arXiv: 1064/, 2019.
[92] Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge, 2019.
[93] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. questions for machine comprehension of text, 2016.
Squad: 100,000+
[94] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019.
[95] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension, 2017. 2017.
[96] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019.
17

---

[97] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, Proceedings of the International Conference on Learning Representations (ICIR), 2021.
[98] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024.
[99] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
[100] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Chal- lenging big- big-bench tasks and whether
can solve them, 2022.
[101] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[102] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[103] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
[104) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
[105] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
[106] David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzi- lay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi. and Arman Cohan. Sciriff: A resource to enhance language model instruction- following over scientific literature, 2024.
[107) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshom, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cris tian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovie, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan
18

---

Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee. Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin R. Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla. Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, (...), Yu Wang. Yuchen Hao, Yundi Qian, Yuzi He. Zach Rait, Zachary De Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models. ArXiv, abs/2407.2 21 21733, 2024.
[108] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633-2650, 2021.
[109] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Erie Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium ( (USENIX Security 23), pages
2023.
[110] A Feder Cooper, Aaron Gokaslan, Amy B Cyphert, Christopher De Sa, Mark A Lemley, Daniel E Ho, and Percy Liang. Extracting memorized pieces of (copyrighted) books from open-weight language models. arXiv preprint arXiv:2505. 1o.000...2544 /2546, 2025.
[111] Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. Aretic-embed: Scalable, efficient, and accurate text embedding models. arXiv preprint arXiv:2405. 05374, 2024.
[112] Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830-839, 2020.
[113] Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkin- son, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yul- ing Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam. Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo., Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[114] Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training, 2024.
[115] Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Mclis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017.
[116] Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannanch Ha- jishirzi. Olmes: A standard for language model evaluations, 2025.
[117] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022.
[118] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. Muse: Machine unlearning six six-way evaluation for language models, 2024.
19

---

A Model Details
A.1 Baseline Details
Prompt-based routing We implemented domain-specific classifier where each input query is categorized into exactly of n domains using the prompts shown below. Once classified, the query routed exclusively to the corresponding expert model for processing. In particular, the following prompt was used for our primary experimental setup:
"""Classify the given text into one of the following domains:
0: Math
1: Code
2: General science and educational innal content
3: News
4: Creative writing and literature
5: Reddit
6: Research and academic content
7: Other
Provide the number between and """ For ablations with four datasets instead of eight, we used the following prompting.
"""classifi fy the viv text into one of the following domai ns:
0: Math
1: Code
2: General science and academic content
3: Other
Provide the number between and """
Branch-Train-Merge (BTM) BTM first assigns a weight to each expert and then combines their predictions through ensembling.
Formally, for test instance æ and the expert we compute the and convert it to a weight with a temperature-controllee softmax:
loss (x)
exp( (xx//T)
wy (x)
Σ exp( (x))/))
where т >> 0 sharpens (r < or flattens (T> 1) the distribution over the M experts. We optionally retain only the & largest-weight experts S5(2) (A: < M) and renormalize:
T (x(x) (x)x) 1[ Sx(z)]
Ejes,()) (z)
At inference time, let z,, R VI be the logit vector produced by the expert i at generation step t. We combine logits boken-by-token:
(x(v)= = Σ
UE V.
(E)
A.2 Training Details
In FLEXOLMO, to initialize router embeddings with domain embeddings, we sample 1,000 documents from each data source, process them through GritIM/GritMMe [13 to obtain document embeddings, and then average these embeddings.
To obtain proxy data D, we train a binary classifier to distinguish D; from Dpub and select pub- lic samples with the highest predicted likelihood of belonging to D,. Specifically, we finetune Snowflake www.wwwweeeeeeeeeeeeee laake-akkcct www.aarrctteeeedee -ewiicceeebbxxeee [TI], which contains 22M parameters, using a learning rate of 3 x 10-6, The classifier is trained on balanced dataset of 500,000 samples (250,000 documents from each source public and private). The classifier quickly achieved accuracy above 95% across all datascts considered.
20

---

B
Data Details
Table 5 presents the statistics of our training data described in Section
Figure 5: Statistics of data
mix (descriptions in $4.1).
Name
# Tokens (B)
B.1 Reddit Data Processing
Public Mix
2.37 x 10*
The construction of this dataset involved three major phases.
News Creative Writing
158.0
201.9
1. Reddit data filtering A dataset of submission/comment pairs was derived from the PushShift Reddit dataset [12] (bulk dump as of March 2023) the same dump used for Dolma Red- dit /etttps:/huugiian do dolma).
Math
20.3
StarCoder
83.0
Academic
58.6
Educational text Reddit
102.2
co wttp:/cttttcccceee/eeaoaaoo /alenaii/
9.9
To derive our initial dataset, we extracted each submission and concatenated it with its top-scoring, top-level comment. We then performed further rule-based filtering with the following constraints:
Filter out deleted/removed content.
Filter out content marked as over_ _8___1
Filter out all posts from a list of 26,123 banned or NSFW subreddits.
Filter posts from likely bot authors (drawn from https://botrank.pasttmm eut of Sept 2024).
Filter out posts containing non-text media.
Perform document-level text deduplication via Bloom filter.
Retrieval-based subreddit selection Dense retrieval was then used identify academically- relevant subreddits for further filtering. We adapted search queries from MMLU test questions, and performed dense retrieval with these queries the filtered Reddit data from Step #2, retaining the top 5 hits for each query. Based on these retrieved outputs, we selected 151 subreddits meeting the following criteria:
Subreddit has 20 *unique* retrieved items for queries within given MMLU category; OR
Subreddit has >=100 retrieved items for queries across all MMLU categories.
We then filtered the dataset from Step #1 to retain only documents from subreddits on this list of 151 subreddits.
Format rewriting Finally, the data from Step #2 was input to a synthetic rewriting pipeline to generate academic QA items with coverage of diverse question formats. We defined 7 categories of question format inspired by variation observed in MMLU, and used these construct prompts for QA text generation. The format categories are as follows:
1. open-ended
2. statement completion
3. fill-in-thebllank
4. statement truth verification
5. which-of-ffllowingg-- har-thhaett-p--------
6. which-of-followingg-
7. in-question options
For each format category we constructed a prompt for generating questions that category given an input text. Below is example prompt, for the "in-question-optionss" category. Prompts for other categories differ in 1) the content of the "For format paragraph and 2) the in-context examples (1-3 examples per prompt).
21

---

will ask to convert text into multiple-choice questions. Here is the text
"(text))
Instructions:
Corvert the information in the text into academic multiple choice questions. ONLY include questions that are academic. DONOT reference the text in the question.
For format, questions that provide options within the question and give choices for which options are true. Examplex:
Dogs have which of the following properties?
1. They are mammals
II. They have five legs.
III. They have tail.
A. I only
B. Il only
C. III only
D. I and III
Answer:
% %%%%
Which of the following are cities in the US?
1. Paris
II. Athens
III. Chicago
A. I only
B. II only
C. III only
D. Il and III
Answer:
Separate ALL questions with "v"""
For generating rewritten QA data, we prompted GPT-4o mini (Jan 2025 version). We iterated over the submission/comment pairs in the data from Step #2, and for each of these texts we sampled format category and prompted the GPT-4o mini generate QA pairs for that text and format category. For longer input texts, format categories were resampled and prompted for again, a number of times proportional the length the text.
Finally, GPT-4o mini outputs were parsed into separate QA items based on the "%% separator, and 50% of items were prepended with the prefix "Question:" ".
We validated these rewritten data in experiments with OLMo [113] models trained to 2T tokens, carrying out continued pretraining on a 50-50 mix of DCLM and Reddit data while annealing the learning rate to zero, a strategy used in [14 78 107 We run this continued pretraining with two versions of Reddit data: the filtered data from Step #2, and the rewritten data from Step #3. We find that the rewriting improves over the non-rewritten data in both MC9 and MMLU: MC9 improves from 0.74 to 0.76 and MMLU improves from 0.62 to 0.66.
22

---

Table 4: Evaluation benchmarks. Benchmarks are divided into general- purepose and domain- specific categories. Original dataset citations are listed at right. "SCIRIFF spans several subtasks (BioASQ factoid, general, and yes/no questions, COVID DeepSet QA, and PubMedQA) evaluated with different metrics. MATH includes seven subsets: algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus.
General Benchmarks
Domain-specifie Benchmarks
Benchmark
Metric
Citation
Benchmark
Metric
Citation
ARC-EASY
AcCrow
84
HUMANEVAL
Pass I
103
ARC-CHALLENGE
Acc
84
HUMANEVALPLUS
Pass 1
104
BOOlQ
AcCyow
85
MBPP
Pass I
103
CSQA
AcCeww
86
MBPPPLUS
Pass 1
104
HELLASWAG
A0C
87
SCIRIFFI
100
OPENBOOKQA
Acdeww
KN
MATH
Exact Match
102
PIQA
Acc
89
NEwS GENERATION
LM LMMas-judge
SOCIAL IQA
ACCraw
90
POEM GENERATION
LM-as-judge
WINOGRANDE
ACC
GSM8K
Exact Match
101
MMLU
Accove
MMLU-PRU
AcC
AGIEVAL (English)
ACCeww
BIG BRNCH HARD (BBH)
Exact Match
100
COQA
F1
92
DROP
FI
96
NATURAL QUESTIONS (NQ)
Fl
94
SQUAD
F1
93
TRIVIAQA
Fl
95
NARRATIVEQA
Fl
115
Aggregate scores (averages individual benchmarks)
GENS: COQA. SQUAD, NATURAL QUESTIONS, TRIVIAQA, DROP
MC9: ARC-EASY, ARC-CHALLENOE, BoOlQ, CSQA, HELLASWAG, OPENBOOKQA, PTQA, SOCIAL IQA, WINOGRANDE CODE4: MBPP, MBPPPLUS, HUMANEVAL. HUMANEYALPLUS
C Evaluation Details
All evaluations are done using the OLMES evaluation standard introduced by [116). following key metrics from the OLMO 2 framework (78). Table presents detailed breakdown of our evaluation datasets and metrics. General description provided in $4.2 here, we provide more details.
SciRiFF We select five subtasks that do not require structured prediction: PubMedOA, and COVID-QA., with tables reporting
BioASQ-Yes / /No, the average performance across all five tasks.
MATH We assess solution correctness through exact matching with ground truth answers, following the methodology established in [17].
News Generation We use the muse- mruse-bench/Meee MMNech/MUSE-Nww dataset from 118 selecting articles containing between 64 and 128 tokens. Models are prompted to continue an article given a prefix of the first 32 tokens, using a sampling temperature of 0.8. A Llama-3 3. 3- 3- IBBIInttuct model serves as the judge, evaluating each completion based on journalistic quality (2 points), topical coherence (2 points), and clarity/fluency (1 point), for total score between 0 and 5, which is then normalized 0-100 scale. To reduce variance, we generate five completions per prompt and report the average score.
Poem Generation We employ the merve /twt/ppptrttteooo ry datase filtering poems those containing 64-128 tokens. From a total of 176 poems (147 Renaissance, 29 Modern), we reserve five poems (three Renaissance, two Modern) for few-shot examples and 100 for testing. Models continue each poem from its first four lines, following the same prompting and evaluation settings as in NEWSGEN. A genre-aware aware L ama-3. 10-70-IInsttu judge evaluates each completion based on poetic craftsmanship (2 points), thematic coherence (2 points), and clarity/fluency (I point), with
ht httttps:/www.ket
data
23

---

scores normalized to 0-100 scale. As with news generation, five completions are generated per prompt, and we report the average score across all instances.
D
Methodology Intuition
D.1 Problem Setup
Given input data E RR*, the router learning can be viewed multi-class classification problem with n++1 classes: a public class Gpub and n closed classes (C; [= The scoring function 8; IR² - IR such that (x) r; represents the for class C., where r; is the router embedding and the higher scores indicate higher class membership likelihood.
Training:
Pairwise Binary Classification In training experts to coordinate 13.31,, we learn n binary classifiers (fr ) where each R + {Cpub, C discriminates between Gpbb and C,:
SC;
if s;) > Spab (x)
f((x)
Cpub otherwise.
The decision boundary h4 between Cpub and is defined as:
h4 = {x E IR* Spub (x) = s;(x)}
Inference: Multiclass Classification At inference time ($ 3.3.2 these binary classifiers combined into a unified multi-class classifier that maps from R Cpab, CI,...,C The final classification decision is determined by:
to the complete class space
F F(x)
arg
max (pub, m.
(x()..
The key question is: how could we make the ensemble binary classifiers ) route inputs as close to if we had trained one unified multiclass classifier F end-to-end on all data?
Our intuition just "C, vs. Cpuh", this could make decision boundary that tightly encircles C, which leads to better multi-class classification.
each binary classifier fa learns the decision boundary "C, vs. not C," rather than
D.2 Anchor Point and Negative Bias
Classification Boundary
To ensure each classifier fi learns better decision boundary h4s we propose two key techniques:
Freezing Tpub and Mpub as anchor points During training of each binary classifier fú. we fix the public expert Mpub and router embedding Fpub ensure all classifiers share common coordinate system by maintaining:
Feature
0m
(pub(x) = Puub X, Vi {1,... .n..
Feature
Figure 6: During training, the negative bias shifts the decision boundary, So that a more selective subset of data will be used to train the expert corresponding to Ct.
Without this constraint, each binary classifier would optimize both Fpah and r; independently, resulting in inconsistent coordinate systems where pub # rpub for i A during the training of binary classifiers fi and fy. Such inconsistency would invalidate the multi-class classifier F(x), would attempt compare scores computed in incompatible embed- ding spaces. By maintaining a fixed reference point, we ensure all decision boundaries h; are defined relative the same coordinate system, enabling their meaningful composition during inference without additional training.
(G)

---

Adding a negative bias During training of fú. each binary classifier only needs to satisfy r; X > Tpub x for x E C. When Cub and C; is separable, many possible r; vectors (no bias boundary in Figure 6) can satisfy this constraint without precisely characterizing the specialized region of Cj. We refine this by adding negative bias term to each expert:
x by rpub x Vi (1,2,....1)
As Figure 6 fillustrates, the negative bias term could help move the decision boundary hy closer to C₁ data points. This means, during training, a more selective subset of data will be used to train the local expert. This facilitates merging, where experts compete not only with Cpub but with each other.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe our core contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the result analysis $
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
No formal results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The code will be open-sourced and we describe all model experiments in detail.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer:
[Yes]
Justification:
Data and code will be open sper-sourced with instructions
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttps://nips.co
pub Sublic/ae /ttp////ddes/eeeeeeeeeeeee
ssionE oomdooliyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https) httpcc///ppbiicgiiieeeeoceeeeeee c/w.pp/cc/ppuliicc///////c/ccc//
//nips cc/
CooeeeooeeSSae ubmi iss aessonPoolise
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See hyperparameter in V
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Repeated experimental runs to produce error bars would be prohibitively ex- pensive; we instead capture evaluation noise through inclusion of a large suite of downstream benchmark scores, see $4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we report computer resources in detail in $4.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:pppww...eeee......./////aaa
Answer: [No] con
Justification: The research uses restricted datasets specifically to demonstrate the data flexibility and privacy compliance of the proposed model architecture FLEXOLMO.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss these in 2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, the paper describes safeguards that allow data opt-out at inference time.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, describe licenses in 4..1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, documentation is provided throughout the text.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There are no crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There are no crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
31

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs in non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http://eeiip..p..ee.. wttp://eeee../oeoeeeeeeeeeeeeeee / 2025/ LLM) for what should should not described.
32


Reviews:
[{"review_id": "ib5iUY8PIm", "reviewer": "Reviewer_JVHX", "summary": "This paper introduces FlexOLMo, a novel class of open language models that support both distributed training without data sharing and data-flexible inference. At the core of the approach is a Mixture-of-Experts (MoE) architecture where each expert is trained independently on a private dataset using a shared public model as the anchor. Experts are merged post-training via a nonparametric router constructed from domain-informed embeddings, avoiding any joint training. The system allows opt-in/opt-out control at inference time, addressing data sovereignty, licensing, and privacy concerns. The authors demonstrate FlexOLMo utility using a curated dataset (FLEXMIX) composed of a public mix and seven restricted domains. They show that FLEXOLMO consistently outperforms strong baselines such as model soups, BTX, BTM, and prompt-based, while also supporting modular extensibility and strict data-use guarantees.", "questions": "1. Could the authors provide more empirical evidence or analysis on how well the router generalizes? \n\n2. What is the hypothesized reason for FlexOlmo underperforming on Code4?\n\n3. Can the proposed framework accommodate settings where different data providers use different model architectures for their local experts? If not, what modifications would be needed to enable this?\n\n4. Why was federated learning not included as a baseline, given that it is a direct solution to the same data-sharing constraint? Additionally, have the authors considered comparisons to model routing methods such as:\n[1] Muqeeth, Mohammed, et al. \"Learning to route among specialized experts for zero-shot generalization.\" ICML 2024.", "limitations": "The authors have not included a dedicated section or explicit discussion of the limitations of their work. While the paper mentions challenges like underperformance in coding tasks (Tables 1 and 2), it does not systematically address limitations such as potential scalability issues and computational costs. Additionally, there is no discussion of potential negative societal impacts, such as biases in the restricted datasets or ethical concerns related to data opt-out guarantees.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper tackles two highly important and challenging problems in current language model development: data sharing constraints and data use constraints. These issues are increasingly central due to legal, privacy, and IP considerations, and FlexOLMo offers a meaningful solution. \n\n2. The proposed combination of modular, distributed training and data-flexible inference via a nonparametric MoE framework is both practical and elegant. The authors demonstrate solid performance improvements across a wide range of tasks, outperforming competitive baselines.\n\n**Weaknesses:**\n\n1. While the paper evaluates against several baselines, it omits comparison to relevant approaches based on federated learning or other model merging methods, which also address the same data restrictions problem. For example, works like Muqeeth et al. (ICML 2024) propose routing among specialized experts in a related context and could serve as valuable baselines. \n\n2. While the architecture is well-motivated and practically useful, the technical contributions over prior MoE-based approaches appear incremental rather than fundamentally novel.", "was_revised": true, "final_justification": "I support the paper’s acceptance, as it presents a promising direction that could benefit the community.", "rebuttal": "Thank you for the thoughtful review and for recognizing our work as a meaningful approach to the critical challenge of data constraints in language model development. We address each of your comments below and have incorporated new experiments and analyses: \n\n> Why was federated learning not included as a baseline, given that it is a direct solution to the same data-sharing constraint? Additionally, have the authors considered comparisons to model routing methods such as: [1] Muqeeth, Mohammed, et al. \"Learning to route among specialized experts for zero-shot generalization.\" ICML 2024.\n\n**Comparison to Muqeeth et al. (ICML 2024):** Thanks for the suggestion. We have adapted the SMEAR method from Muqeeth et al. to our experimental setup. As shown in the table below, our proposed method outperforms SMEAR across all evaluation categories. We will include this result in the final version of the paper: \n\n| Model | MC9 | Gen5 | MMLU | MMLU Pro | AGI Eval | BBH | Math2 | Code4 | Avg. |\n|-------|-----|------|------|----------|----------|-----|-------|-------|------|\n| SMEAR (Muqeeth et al.) | 69.6 | 56.3 | 58.6 | 30.5 | 45.4 | 43.5 | 18.5 | 14.8 | 42.2 |\n| FlexOlmo | 71.0 | 59.8 | 59.9 | 30.8 | 45.8 | 47.1 | 50.7 | 17.3 | 47.8 |\n\n**On Federated Learning (FL):** We experimented extensively with existing FL algorithms (e.g., FedAvg, FedProx) using the NVFlare framework (https://github.com/NVIDIA/NVFlare). However, we encountered substantial engineering challenges in scaling these methods to our setting (training models with up to 37 billion parameters), primarily due to the synchronization required in FL. Existing FL methods are not yet supported for training at this scale. Additionally, FL methods do not support data opt-out at inference time, as they are designed to produce a single, monolithic model.  We will add discussion of these challenges to the paper.\n\nOur paper includes comparisons to four strong model merging baselines, including BTM and BTX. We believe these, along with the new SMEAR results above, provide a comprehensive comparison. We are happy to include additional baselines in the final version if the reviewer has additional specific suggestions.\n\n\n\n> The technical contributions over prior MoE-based approaches appear incremental rather than fundamentally novel.\n\nWe respectively disagree with this assessment. While Flexolmo borrows the MoE architecture, its core contribution is a fundamentally new training method that enables capabilities not supported by standard MoEs. Standard MoE models are trained end-to-end with joint access to all data. In contrast, our method introduces a new, coordination-based training algorithm that enables two key features: (1) distributed training without data sharing and (2) opting in and out of subsets of data during inference. These features are important for leveraging restricted datasets where centralized data pooling is not feasible. \n\nAdditionally, we chose to keep the standard MoE architecture in Flexolmo to ensure our method can be easily integrated into existing large-scale pretraining LM pipelines.\n\n\n>  What is the hypothesized reason for FlexOlmo underperforming on Code4?\n\nWe attribute this to suboptimal routing. Ideally, coding tasks should be mostly handled by the code expert, but current routing sometimes directs them to other experts. To support this, we analyzed the router’s behavior on coding benchmarks and found that ~23% of tokens are routed to the Textbook and Math experts. We will include this analysis in the final version of the paper to accelerate future improvements.\n\n\n> Can the proposed framework accommodate settings where different data providers use different model architectures for their local experts? \n\nThe overall architecture of the local experts needs to be the same as the anchor model (e.g., same positional encoding). Varying the FFN’s hidden dimension is feasible for different local experts as long as the output vectors of the experts have the same dimension. The standard FFN expert has two linear layers: an up-projection to an intermediate dimension and a down-projection back to the output dimension. This intermediate dimension, and thus the FFN expert size, can vary based on a provider's local data size and compute constraints. \n\n> Limitation sections\n\nWe thank the reviewer for this suggestion. Below is a draft of a limitations section that we will include in the final version of the paper: \n\n**Data privacy and extraction risks.** A primary concern with sharing model weights is the potential for data leakage. As we quantify in our data extraction analysis (Section 5.3), while the rate of extractable training data from an expert's weights is low, it may still be too high for high-stakes domains like healthcare or finance. A direction for future work is to enable data owners to apply differential privacy (DP) training during their local expert training. Since DP is orthogonal to the FlexOlmo architecture, each contributor could independently decide whether to adopt it, providing formal privacy guarantees.\n\n**Scaling and model size.** The FlexOlmo architecture scales by adding new experts, which linearly increases the total number of model parameters. While the sparse activation could keep the number of active parameters constant during inference, the total model size can become substantial as more experts are integrated. Future work could explore techniques like expert distillation or pruning to manage model size. \n\n**Bias and ethics concerns of FlexOlmo.** While the modularity of FlexOlmo provides powerful data control, it also introduces risks that require careful consideration. One concern is the potential for malicious use, where an adversary could intentionally contribute a low-quality expert trained on misinformation. Integrating this low-quality expert could compromise the integrity and safety of the entire model.  Similarly, low-quality experts could compromise the model even without any ill intent. Addressing such scenarios would be valuable future work. Furthermore, the framework's guarantee of data owner control raises governance and ethical questions. For instance, an unethical data owner could engage in discriminatory gatekeeping: deciding to withhold a valuable expert (e.g., in medicine or finance) from certain users.\n\nWe hope our response and the additional results have addressed the reviewer's concerns. Please let us know if you may have any questions."}, {"review_id": "sdIjWdOAMV", "reviewer": "Reviewer_7oic", "summary": "This paper presents FlexOLMo, a novel LLM specifically designed to address two key real-world constraints: distributed training without data sharing, and data-flexible inference. The model employs a MoE design, where each expert is trained independently and asynchronously on private data using FlexMix where each experts are later integrated via a nonparametric router, leveraging  as an open-source anchor model, without requiring any joint training. FlexOLMO outperforms both prior model merging baselines and individual experts across a diverse set of benchmarks.", "questions": "1. Could the authors elaborate on what they mean by \"fine-grained control of data access during inference\"? How exactly does this mechanism work, and what are the specific benefits? Is there any quantitative analysis as support during evaluations that demonstrates the advantages of this approach?\n2. Why does training on math-mix experts improve code performance? What is the underlying connection between mathematical reasoning and coding capabilities?\n3. What is the specific accuracy of the router? For example, when the router classifier is replaced and ground truth labels are directly provided for each routing operation, how is FlexOlmo's accuracy affected?\n4. How can the combined model outperform individual expert X that was specifically trained on domain X (Table 1, Math-mix increase Code4 accuracy by 14.7%)? What is the intuition behind this?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n - Authors propose a novel MoE training framework where each expert trains independently before being merged and enabled to communicate with one another, which shows promising future research due to its adaptability.\n- They conduct comprehensive evaluations across 31 different taks, along with detailed ablation studies examining factors such as expert count effects and routing pattern analysis.\n- The proposed approach is both practical and efficient for real-world applications, outperforms some prior merging methods.\n\n### Weaknesses\n- The main limitation of the proposed work is its exclusive focus on large-scale datasets and continued pre-training. The scope of FlexOlmo would be more comprehensive if it could demonstrate SFT and RL based training for individual experts across different domains. The motivation for relying solely on large-scale datasets, specifically creating FlexMix and continued pre-training remains unclear.\n\n- As far as I understand, experiments are conducted using only one public model in 7B scale. To validate the scalability of the proposed method, it would be more reliable to include two other open-source models such as Llama-based or Qwen-based architectures. Moreover, it would be interesting to examine the parameter scalability of FlexOlmo by testing whether the observations hold across varying model sizes (e.g., Qwen-3B, Qwen-7B, Qwen-32B).\n\n- Authors mention that FlexOlmo trains each expert independently compared to joint training, which is promising for scalability and adaptability to new tasks and domains. However, it remains unclear how this coordination is achieved. If experts are trained in complete independence, how do they learn to coordinate effectively? Is there a specific optimization mechanism implemented during independent training to facilitate this coordination?", "was_revised": true, "final_justification": "I believe this work presents a novel and promising direction that should be accepted. My previous concerns, including those regarding the choice of CPT, have been resolved. Assuming the revised manuscript will address the motivation for CPT, challenges with SFT and RL, and discuss cross-domain generalization, I am increasing my scores for the significance, my confidence in this work, and the overall score.", "rebuttal": "We thank the reviewer for their time and constructive feedback. We appreciate the acknowledgment that our method is both practical and efficient for real-world applications. We address each of your comments below.\n \n> The main limitation of the proposed work is its exclusive focus on large-scale datasets and continued pre-training. The scope of FlexOlmo would be more comprehensive if it could demonstrate SFT and RL based training for individual experts across different domains. The motivation for relying solely on large-scale datasets, specifically creating FlexMix and continued pre-training remains unclear.\n\nWe agree that extending the FlexOlmo framework to SFT and RL is a valuable direction for future research. Our choice to focus on contined pre-training is motivated by pressing, real-world cases where data owners possess large amounts of restricted, unstructured data rather than instruction-tuning or preference datasets. For example, the Reddit data [1] in FlexMix represents real-world unstructured data that now comes with access restrictions. Similarly, major publishers like The New York Times [2] have recently restricted the use of their raw news corpus for pretraining LMs. Anthropic [3] faces legal charges for using pirated books like libgen for pretraining. These access restrictions create a critical bottleneck: valuable data for improving LM is locked away. \nFlexOlmo addresses this problem by giving data owners more control over their data in pretraining. While we believe FlexOlmo's modular training approach would generalize effectively to SFT and RL, exploring that is a natural next step that remains beyond the scope of our current paper and opens up technical considerations that would require extensive experimentation. We will include a discussion on extending our work to SFT and RL in the future work section. Thanks for the suggestions. \n\n> Extending to Other Open-Source Architectures\n\nWe note that we cannot use Llama or Qwen models directly, as our research requires pre-training a model from scratch to ensure no overlap between the pre-training data and high-risk data added later. While it would be possible to pre-train models based on Llama- or Qwen-based architectures, it is computationally costly, requiring 512 H100 GPUs for three days, which exceeded our budget. We leave the exploration of FlexOlmo under alternative architectures for future work.\n\n\n> If experts are trained in complete independence, how do they learn to coordinate effectively\n\nThis is related to the core technical innovation of FlexOlmo. Our training method is explicitly designed to enable coordination of experts without joint training. We achieve this through two key mechanisms:\n\n1. Coordinated Training with an Anchor Model: Instead of joint training, each data owner trains their expert alongside a frozen copy of the public model. **This public model acts as a shared \"anchor,\"** ensuring that all independently trained experts learn to operate within a common functional space and can coordinate effectively when merged.\n\n2. Domain-Informed Router Initialization: The router, which directs tokens to the appropriate experts, is initialized using domain-specific embeddings derived from each expert's training data. These embeddings encode useful information on how to differentiate between the experts, providing a strong starting point for the router. \n\n> Could the authors elaborate on what they mean by \"fine-grained control of data access during inference\"? How exactly does this mechanism work, and what are the specific benefits? Is there any quantitative analysis as support during evaluations that demonstrates the advantages of this approach?\n\n“Fine-grained control” means that a single FlexOlmo model can deploy different combinations of experts (and their associated data) to different users based on specific access criteria such as permissions, access policies, or the user’s own preferences. Here is a concrete example – suppose a major newspaper contributed an expert. They could restrict access so that only users with an active subscription to that paper have the option to activate that expert, while others cannot. We quantitatively demonstrate the effect of data opt-out in section 5, showing that deactivating the News expert successfully removes its influence on in-domain tasks while having minimal impact on unrelated tasks.\n\n\n> Why the math expert improves coding performance\n\nThe improvement in coding performance from the Math expert is likely due to cross-domain generalization of reasoning abilities. The structured  step-by-step problem-solving inherent in math text appears to enhance the model's underlying reasoning capabilities. This phenomenon has been observed in prior work; for example, [4] showed that training on mathematical data can significantly boost performance on coding benchmarks and vice versa. \n\n\n> What is the specific accuracy of the router? How can the combined model outperform individual expert X that was specifically trained on domain X\n\nA traditional accuracy metric is not applicable because our router operates at the **token level, not the sequence level**. There is no single ground-truth expert label for a token. This token-level routing can dynamically select the most appropriate expert for each token in the input sequence. For example, consider an example that contains both math concepts and coding elements. With token-level routing, our model can route the math tokens to the math expert while directing the coding-related tokens to the coding expert. This token-level routing allows the model to leverage each expert’s expertise precisely where it's needed within a single example, rather than being constrained to use one expert for the entire example. \n\nWe hope our responses have addressed the reviewer's concerns. Please let us know if you may have any questions. \n\n[1] https://the-decoder.com/reddit-ends-its-role-as-a-free-ai-training-data-goldmine/\n\n[2] https://www.popsci.com/technology/nyt-generative-ai/\n\n[3] https://fortune.com/2025/07/28/a-copyright-lawsuit-over-pirated-books-could-result-in-business-ending-damages-for-anthropic/\n\n[4] Magistral"}, {"review_id": "YvyHcF3aRE", "reviewer": "Reviewer_HG3R", "summary": "FLEXOLMO is a novel class of Language Models (LMs) designed to overcome the significant challenges associated with centralized data access and control in traditional LM training. Unlike conventional methods that require aggregating all data centrally and making a one-time decision about data inclusion, FLEXOLMO enables distributed training without data sharing and provides flexible, fine-grained control over data access during inference. FLEXOLMO employs a mixture-of-experts (MoE) architecture. In this design, individual \"experts\" (small feedforward networks or FFNs within each transformer block) are trained independently on private, locally maintained datasets. A shared \"public model\" (Mpub) trained on publicly available data serves as an anchor point. FLEXOLMO supports flexible inclusion or exclusion of experts and their associated data during inference with no further training. This provides strong guarantees for data opt-out. For example, the paper demonstrates that removing a \"news expert\" significantly reduces performance on news-related tasks but has minimal impact on unrelated tasks. This directly addresses societal impacts related to data access, intellectual property, and privacy regulations (like GDPR and HIPAA).", "questions": "1. While the breadth of evaluation is a strength, the absence of statistical significance measures (like error bars or confidence intervals) for the reported performance metrics (e.g., in Table 1 and Table 2) is a notable limitation. \n2. authors should clarify: How inference-time FLOPs/latency scale as the total number of experts (n) increases and as the number of active experts (k) varies (e.g., does activating only 4 out of 8 experts significantly reduce compute compared to a dense model of equivalent capacity?).\nClarify, how the overall training cost scales when adding new experts? While experts are trained independently, there might be implications for the shared anchor or router components.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The paper is well-organized, with clear sections detailing the problem, proposed architecture, training algorithm, experimental setup, and results.\nFLEXOLMO employs a mixture-of-experts (MoE) architecture where individual experts are trained independently on private datasets, a stark contrast to traditional centralized training. A key innovation is the nonparametric routing mechanism, which allows experts to be integrated without any joint training, a major challenge in decentralized settings. This is further refined by a learnable negative bias term for each independently trained expert, which helps the router narrow decision boundaries and improves generalization. The model was extensively evaluated on 31 diverse downstream tasks across general-purpose and domain-specific categories. \n\n\nweakness -\n The paper does not report error bars for experimental results due to \"prohibitive expense\". While a large suite of benchmarks is used to mitigate this, the absence of statistical significance measures is a noted limitation. While compute resources are reported, a formal analysis of how FLEXOLMO's computational complexity scales with the number of models (N) and objectives (K) is not provided, which could be useful for practitioners.\n\n\n◦", "was_revised": false, "final_justification": "", "rebuttal": "> The paper does not report error bars for experimental results due to \"prohibitive expense\". While a large suite of benchmarks is used to mitigate this, the absence of statistical significance measures is a noted limitation. \n\nWe thank the reviewer for noting this. The costs associated with training the models unfortunately make it impossible to perform multiple runs for each experiment, which is required for reporting error bars. We acknowledge this is a limitation and will include this discussion in the paper.\n\n> While compute resources are reported, a formal analysis of how FLEXOLMO's computational complexity scales with the number of models (N) and objectives (K) is not provided, which could be useful for practitioners. How inference-time FLOPs/latency scale as the total number of experts (n) increases and as the number of active experts (k) varies (e.g., does activating only 4 out of 8 experts significantly reduce compute compared to a dense model of equivalent capacity?). Clarify how the overall training cost scales when adding new experts? While experts are trained independently, there might be implications for the shared anchor or router components.\n\n- Training Cost: The cost of adding a new expert is independent of the total number of existing experts and only depends on the local expert size. Our model can scale efficiently in training as there is no joint training of all experts. \n\n- Inference Cost: Similar to standard MoE, Inference costs are dependent on the number of active experts (k), not the total number of experts in the model. This means that activating only 4 out of 8 experts significantly reduces compute compared to a dense model of equivalent capacity. \n\nWe believe the FlexOLMo approach is very appealing in this regard, and we will include a detailed discussion regarding computational complexity scaling in the final version of the paper. \n\nWe hope our responses have addressed the reviewer's concerns. Please let us know if you may have any questions."}, {"review_id": "CXn0V981fj", "reviewer": "Reviewer_F5CK", "summary": "This paper introduces **FlexOlmo**, a new class of language models that supports distributed training without data sharing and data-flexible inference, where parameters along with their associated data can be flexibly included or excluded with no further training.  \nThis paper also introduces **FlexMix**, a corpus comprising seven restricted sets, either real or realistic approximations, alongside 10 publicly available datasets.", "questions": "**Questions**:\n1. What is the motivation behind processing and filtering the original Reddit dataset? Why not directly use the same version adopted in previous studies?\n2. What is the rationale for selecting these eight domains? Are there any references or prior work supporting the inclusion of domains such as \"Creative Writing\" or \"Reddit\"? Furthermore, is there any semantic relationship or overlap among these domains?\n***\n**Suggestions**:\n1. Some sentences in the paper could be improved for clarity. For example:\n> \"A corpus we curate comprising seven restricted sets, either real or realistic approximations, alongside 10 publicly available datasets.\"\n\nThis sentence is somewhat overloaded in its current form. The phrase \"we curate\" may be unnecessary, as it is already implied by the context. \n2. For Figure 4, I believe results on more benchmarks (not just MMLU) will be more informative.\n***", "limitations": "I think the paper needs to open up a separate section to discuss its limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:  \n1. I like the core idea of FlexOlmo, which leverages a public dataset as a medium for coordination across domains. While conceptually straightforward, it yields meaningful performance improvements as demonstrated in the experiments.\n2. The paper includes a comprehensive set of evaluation benchmarks and conducts necesary ablation study within the page limit. \n3. Generally, this paper is clearly structured and easy to follow.\n***\n**Weaknesses**:  \n1. The training procedure of the expert model $M_i$ and route embedding $r_i$ are not clearly explained, particularly regarding how their parameters are jointly updated across the dataset $D_i$, in Section 3.3.1 and 3.3.2.\n2. For Section 3.3.3, the distribution of private data may also be privacy-sensitive. \n3. In section 2.1, the authors discuss two types of data constraints. For the first constraint on data sharing, I think decentralized training can adequatly address. However, for the second constraint on data usage, I'm not fully convinced how FlexOlmo can handle restrictions including copyright restrictions. In certain scenarios, take toxicity classification as an example, it would be helpful if the authors could provide a more detailed explanation of how FlexOlmo addresses this challenge and why it is preferable to conventional methods, such as directly fine-tuning on datasets that contain toxic content. \n4. The core contribution and motivation of the constructed dataset FlexMix is unclear.  And I suggest the authors explicitly list their main contributions in the Introduction section.\n5. Lack of domain-specific evaluation. Specifically, there are no corresponding evaluations for *Reddit* and *Textbook* domains.\n6. From experiments, the majority of FlexOlmo's performance gains appear to come from \"Math2\". If the math domain is excluded, does FlexOlmo still outperform BTM and BTX?\n7. I notice that the training of FlexOlmo requires substantial computational resources in Section 4.4. In practical decentralized training scenarios, however, data owners may not have access to such extensive resources for continual training. This limitation should be discussed.\n***", "was_revised": true, "final_justification": "Most of my concerns are addressed, particularly weakness 1,3,4,5. \nFor weakness 2, I understand there is always a risk and the proposed method draw an acceptable trade-off.", "rebuttal": "We thank the reviewer for their time and constructive feedback. We appreciate that you like the core idea of FlexOlmo. We address each of your specific comments below.\n\n> The training procedure of the expert model and route embedding are not clearly explained, particularly regarding how their parameters are jointly updated across the dataset, in Section 3.3.1 and 3.3.2.\n\nThe training process of Flexolmo is as follows (as shown in Figure 1):\n1. First, a public model is trained on data without any restrictions.\n2. Then, each data owner trains their local expert alongside a frozen copy of the public model. This public model acts as a shared \"anchor,\" ensuring that all independently trained experts learn to operate within a common functional space. During this local training stage, the public model and its public router embedding remain frozen. Only the local expert's parameters (FFN layers) and its corresponding router embedding are updated.\n3. Finally, in the merging step, we combine the frozen public expert and all the independently trained local experts by simply concatenating their learned router embeddings into a joint table. \n\nWe thank the reviewer for this feedback and will make the explanation more clear in the final version of the paper.\n\n\n> For Section 3.3.3, the distribution of private data may also be privacy-sensitive.\n\nWe agree with the reviewer that the selection of public data based on the private distribution may carry the risk of indirectly revealing information about it. We clarify in the paper that this is an optional step a data owner would only undertake if they are comfortable with this risk.\n\n>  I'm not fully convinced how FlexOlmo can handle restrictions including copyright restrictions. In certain scenarios, take toxicity classification as an example, it would be helpful if the authors could provide a more detailed explanation of how FlexOlmo addresses this challenge and why it is preferable to conventional methods, such as directly fine-tuning on datasets that contain toxic content.\n\nTraining a single FlexOlmo model allows for deploying different combinations of experts (and their associated data) to different users based on specific access criteria such as permissions, policies, or preferences. Here are two concrete examples:\n\n- Copyrighted Data: Suppose two newspapers contribute experts trained on their proprietary data. They could restrict access so that only users with an active subscription to their respective papers have the option to activate the associated expert, or activate both experts if the user subscribes to both, while others cannot.\n- Content Moderation: The FlexOlmo architecture allows for the on-demand deployment of specialized experts, such as one for toxicity detection. This expert can be activated specifically for sensitive applications like toxicity detection, while remaining inactive in other use cases where child-safety is important\nIn summary, we train a single FlexOlmo model, but it can function as if it were trained on many different data combinations, customizing its usage to match specific deployment requirements and constraints.\n\n> Lack of domain-specific evaluation. Specifically, there are no corresponding evaluations for Reddit and Textbook domains.\n\nThank you for this suggestion. We include many domain-specific evaluations as detailed in Section 4.2. For datasets like Textbook, which consists of broad texts from STEM domains, its benefits are reflected in improved performance on STEM-related benchmarks like BBH, Math, and Code. Similarly, some datasets like Reddit are deemed to be generally helpful for improving the model's world knowledge and conversational abilities and are not limited to specific domain evaluations.\n\n> I notice that the training of FlexOlmo requires substantial computational resources in Section 4.4. In practical decentralized training scenarios, however, data owners may not have access to such extensive resources for continual training. This limitation should be discussed.\n\nThe substantial computational resources mentioned in Section 4.4 (512 H100s for 3 days) are for training the base public model on 1T tokens. Training the individual local experts requires significantly less compute (e.g., 50B tokens for one expert). \nOur framework is applicable to cross-silo federated learning, where clients are typically organizations with access to reasonable compute resources. Furthermore, as we will discuss in the limitations section, future work can explore extending our model to accommodate clients with varying compute budgets by allowing them to train different-sized local experts.\n\n> From experiments, the majority of FlexOlmo's performance gains appear to come from \"Math2\". If the math domain is excluded, does FlexOlmo still outperform BTM and BTX?\n\nYes, it does. After excluding the Math2 results, FlexOlmo's average score is still better than the baselines. The average scores are: FlexOlmo (47.4), BTM (46.5), and BTX (43.3).\n\n> What is the motivation behind processing and filtering the original Reddit dataset? Why not directly use the same version adopted in previous studies?\n\nIn the case of the Reddit data, we drew on data from a parallel line of work, which was motivated by the observation that prior versions of Reddit data showed limited performance gains on downstream tasks, particularly on knowledge-based QA tasks. The hypothesis of that work was that Reddit's valuable, specialized knowledge could be better leveraged by more targeted filtering and data refinement. The process to arrive at the final data involved (1) constructing thread contexts inspired by QA structure, (2) filtering for high-quality, academically relevant subreddits, and (3) rewriting the content to reduce noise and match standard QA formats. The effectiveness of this approach was validated experimentally: when continuing pretraining on OLMo 7B, the rewritten Reddit data yielded substantial improvements over simply using the filtered data – **MMLU performance improved from 0.62 to 0.66, and MC9 improved from 0.74 to 0.76.** This Reddit data is released to facilitate future research.\n\n> What is the rationale for selecting these eight domains? Are there any references or prior work supporting the inclusion of domains such as \"Creative Writing\" or \"Reddit\"? Furthermore, is there any semantic relationship or overlap among these domains? The core contribution and motivation of the constructed dataset FlexMix is unclear. And I suggest the authors explicitly list their main contributions in the Introduction section.\n\nThe seven domains in FlexMix were chosen to represent common, realistic scenarios where data access is restricted. For example, \"Creative Writing\" serves as a proxy for copyrighted materials like fiction and books, and \"News\" data represents copyrighted news data, which correspond to real-world lawsuits where major publishers like The New York Times [2] have recently restricted the use of their raw news corpus for pretraining LMs and Anthropic [3] face legal charges for using pirated books for pretraining. \"Reddit\" in FlexMix represents online discussion forums, which currently face tangible access restrictions; notably, this data is not downloadable following Reddit's 2023 access policy change [1]. These domains exemplify cases where data is not openly available due to its proprietary nature or user privacy concerns. While semantic overlap can exist between domains (e.g., textbooks containing math text), this complexity reflects real-world data.\n\nFlexMix provides a valuable testbed for advancing research in privacy-conscious and data-flexible model pretraining. On this benchmark, we show that FlexOlmo can effectively leverage these restricted sets to produce a better LM.\nFinally, thank you for the suggestion regarding the contribution list; we will explicitly list our main contributions in the Introduction section of the final paper.\n\nWe hope our responses have addressed the reviewer's concerns. Please let us know if you may have any questions. \n\n[1] https://the-decoder.com/reddit-ends-its-role-as-a-free-ai-training-data-goldmine/\n\n[2] https://www.popsci.com/technology/nyt-generative-ai/\n\n[3] https://fortune.com/2025/07/28/a-copyright-lawsuit-over-pirated-books-could-result-in-business-ending-damages-for-anthropic/"}]

Author rebuttal:

Meta-review: The submission presents FlexOLMo, a modular mixture-of-experts (MoE) framework designed to enable distributed training without data sharing and flexible inference with expert opt-in/opt-out. The key contributions include a new training recipe where local experts are trained alongside a frozen public model anchor, and a nonparametric router that integrates experts without joint training. Evaluations on the curated FlexMix dataset and 31 downstream benchmarks show consistent gains over strong baselines, with notable improvements in restricted-data domains. Strengths highlighted by reviewers include the strong practical relevance of addressing data sovereignty and licensing constraints, thorough empirical evaluation, and modular extensibility. Reviewers also noted limitations (e.g., lack of federated learning comparisons, limited discussion of computational costs), but the rebuttal provided new comparisons, new clarification, and detailed explanations of technical designs. Overall, this is a strong technical paper and potentially has a broad impact in the community, so I recommend to accept this paper with spotlight.
Decision: Accept (spotlight)