OLMoE: Open Mixture-of-Experts Language Models
Authors: Niklas Muennighoff
Luca Soldaini
Dirk Groeneveld
Kyle Lo
Jacob Morrison
Sewon Min
Weijia Shi
Evan Pete Walsh
Oyvind Tafjord
Nathan Lambert
Yuling Gu
Shane Arora
Akshita Bhagia
Dustin Schwenk
David Wadden
Alexander Wettig
Binyuan Hui
Tim Dettmers
Douwe Kiela
Ali Farhadi
Noah A. Smith
Pang Wei Koh
Amanpreet Singh
Hannaneh Hajishirzi
Venue: ICLR 2025 Oral (2025)
Keywords: large language models
mixture-of-experts
open-source
Primary area: foundation or frontier models, including LLMs
TL;DR: A state-of-the-art Mixture-of-Experts LLM with 1B active and 7B total parameters trained for 5T tokens that is 100% open-source

Abstract: We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present novel findings on MoE training, define and analyze new routing properties showing high specialization in our model, and open-source all our work: model weights, training data, code, and logs.

Published as a conference paper at ICLR 2025
OLMOE: OPEN MIXTURE-OF-EXPERT LANGUAGE MODELS
Niklas Muennighoff" 3
Luca Soldaini
Dirk Groeneveld "
Kyle Lo
Jacob Morrison "
Sewon Min
Weijia Shi
Pete Walsh
Oyvind Tafjord
Nathan Lambert' David Wadden
Yuling Gu
Shane Arora
Akshita Bhagia" Dustin Schwenk
Alexander Wettig
Binyuan Hui
Tim Dettmers" Douwe Kiela Ali Farhadi aw
Noah A. Smith aw
Pang Wei Koh aw
Amanpreet Singh Hannaneh Hajishirzi aw
Allen Institute for AI "Contextual AI Uniiersityy of Washington P Prinnetton University
D. www.eiggnooiooooo.ooo f www.aall.........ommmmmmmmmaalle
gmail. com
hannah$allema org
ABSTRACT
We introduce OLMOE, fully open, state-of-the-art language model leveraging sparse Mixture-of- Experts (MoE). OLMoE-1B-7B has billion (B) parameters but uses only 1B per imput token. We pretrain it on 5 trillion tokens and further adapt Our models outperform all available models with similar active parameters, surpassing larger like Llama2- 13B-Chat and DeepSeekMoo--l
it to create OLMOF-1BB-B--NE
We present novel findings on MoE training, define and analyze new routing properties showing high specialization in our model, and open-source all our work: model weights, training data, code, and logs.
Weights
Data
Code
httpss//ggttuub..co/allleaaa///p
Logs
-ele-mmVllllo2o000ooiii OLMoE -
Mixture-af Ex-pprss
Densc LMs
open open MoEs?
65
ratlo
Best performance/cost
Uoma3.1
Mistral-7B
Name
Model
Duta Godo
Owen1.5 14B DCLM-78
Perfomnance (MLL)
60
Brck-86B-114B
X
X
X
55
OLMME-18-78
OLMo 7B (0724)
Uama2-133
DBRX-360-1329
Moxtral-398-1415
11
X
X
Skywork-222 246-1446B
X
X
X
.DCLM-1B
JetMoe-22--B
DeepSeekv2-216
X
X
45
DeepSeekWoE-3B-B2
"Jama2-78
Aretic-178-488
V
II
Falcon-7B
Owen2-148-5m 8-8
/
X
X
StableLM2 2B
MP1-76
Mixrra-139-479
X
X
25
78
Jamba- 125-52-
/
LLMMc118 (0724)
DeepSeekdvo 38 168
!!
X
X
Ulma-78
5-39-149
x
X
JinyLlama-12 18
CpenMoE 28 9B
BLOOM-78
OpenMoE-33--2 -3-590
V
6
25
8
JetMoE 28
V
!!
X
Cost (Billion active parameters)
OLM0E-18-78
Figure 1: Performance, cost, and degree of openness of open MoE and dense LMs. Model names contain rounded parameter counts: model-act ive-total for MoEs and model-tota. for dense LMs. 4 ckkpts is the number of intermediate checkpoints available. We highlight MMLU as summary of overall performance; see $3 for more results. OLMoEE-B--7 performs best among models with similar active parameter counts and is the most open MoE.
INTRODUCTION
Despite significant advances in Large Language Models (LMs) on various tasks, there remains a clear trade-off between performance and cost in both training and inference. High-performing LMs inaccessible for many academics and open-source developers as they are prohibitively expensive to
This paper describes the first OLMoE from 09/2024. See Appendix I. for overview of newer version.

---

Published as a conference paper at ICLR 2025
build and deploy. One approach to improve the cost-performance trade-off lies in using sparsely- activated Mixture-of- -f-preetssss (MoEs) (Shazeer et al., 2017). MoEs have several experts in each layer, only subset of which is activated at a time (see Figure C1). This makes MoEs significantly more efficient than dense models with a similar number of total parameters, which activate all parameters for every input (Yun et al., 2024). For this reason, industry frontier models use MoEs including Gemini-1.5 (Team et al., 2024a) and reportedly GPT-4 (Chintala, 2024).
Most MoE models, however, are closed-source: While some have publicly released model weights (DeepSeek-Al et al., 2024b; Jiang et al., 2024; Shen et al., 2024; Team et al., 2024d; Team, 2024b), they offer limited no information about their training data, code, or recipes (see Figure 1). While there have been prior efforts to make language modeling research fully accessi- ble (Biderman et al., 2023; Groeneveld et al., 2024; Li al., 2024a; Liu et al., 2023; Workshop et al., 2023; Zhang et al., 2024a) discussed in detail in Appendix A, they have been largely limited to dense LMs. This comes despite MoEs requiring more openness as they add complex new design questions to LMs, such as how many total versus active parameters to use, whether to use many small or few large experts, if experts should be shared, and what routing algorithm to use. The lack of open resources and findings about these details prevents the field from building cost-efficient open MoEs that approach the capabilities of closed-soure source frontier models.
To address these issues, introduce OLMOE, fully open ooxture-of-Expetrr language model with haae-of-the-an performance among similarly-sized models. In particular, we pretrain OLMoF- 1B-7B for 5.1 trillion tokens with 6.9B total parameters, of which only 1.3B are activated for each input token. This leads to similar inference cost as using dense models with around IB parameters, such as OLMo IB (Groeneveld al., 2024) TinyLlama IB (Zhang al., 2024b), but requires more GPU memory to store its 7B total parameters. Our experiments show that MoEs train ~2x faster than dense LMs with equivalent active parameters (Figure 2). In Figure 1, we show that OLMoE- 1B-7BB significantly outperforms all open IB models and displays competitive performance to dense models with significantly higher inference costs and memory storage (e.g.. similar MMI.U scores to Llama2-133 13B, which is ~ 10x more costly). Via instruction- and preference tuning, we create including Llama2 13B-Chat (Touvron et al., 2023b), OLMo-7B-Instrr (0724), and DeepSeekMoE- 16B (DeepSeek-AL et al., 2024a) on common benchmarks (MMLU, GSM8K, HumanEval, etc.).
-B---IINTRUCC which we find exceeds various larger instruct models
Our comprehensive set of controlled experiments highlights key design choices for MoEs (see Table 1) and LMs in general. One critical design decision for making MoEs performant is using fine-grained routing with granular experts SeekpSeek-AT et al., 2024a):
employ 64 small experts in each layer with 8 being activated. The choice of routing algorithm is also important: we find dropless (Gale et al., 2022) token-based routing (Shazeer et al., 2017) outperforms expert-based routing (Zhou et al., 2022). Our findings also include those that challenge prior work, such as the ineffectiveness of shared experts (DeepSeek-AI et al., 2024a) and the limited benefits of sparsely upcycling pretrained dense LM into an MoE (Komatsuzaki et al., 2023) unless under small compute budgets. Finally, we present novel ways analyze routing behavior in Mixture-of offture-ff-Experts finding that for OLMOE- ---ME-BB--BB routing saturates early in pretraining, experts are rarely co-activated, and experts exhibit domain and vocabulary specialization. We intend our fully open MoE to facilitate more research and analysis to improve our understanding of these models. We release training code, intermediate checkpoints (every 5000 steps), training logs, and training data under open- source licenses (Apache 2.0 http: //ww: apa oro/ http//icennsssIIIee /LICENSE--EEEEEEE 2. 0 or ODC- By 1.0 https: http:s////ppenaaaaccoooooooo httprr///weeeeeeeeeeeeeeeeeeeeee
PRETRAINING AND ADAPTATION
Pretraining OLMoE is decoder-only LM consisting of Nr, transformer (Vaswani et al., 2023)
layers. The feedforward network (FFN) in dense models like OLMo (Groeneveld et al., 2024) is replaced with an MoE module consisting of Ng smaller FFN modules called experts, of which a subset of k experts is activated for each processed input token (also see Figure CI):
MoE module(x) = Σ softmax (r(x)), Ex(x)
(1)
xETop=A(r(=)
FFor example, even with 16 H100 GPUs and several optimizations, Ilama 3 405B only achieves decoding throughput of around 100 tokens per second (Dubey et al., 2024).
2

---

Published as a conference paper at ICLR 2025
Table Key MoE design choices and our setup for OLMoE-1B-7B based on our experiments. Full configuration for OLMoE-IB-7B is in Appendix C.
Design choice
Description
Exper- iment
OLMoE- 1--0--B--BB
Active params
# active parameters per input token
$4.1
1.3B active
Total params
Total of parameters the model
$4.1
6.9B total
Expert granularity
Using fine-grained small experts vs. few large $4.2 experts (Dai et al., 2024)
64 small experts with 8 activated
Routing algorithm
How inputs assigned experts, e.g.. a per token basis (e-g., 2 experts per token) per expert basis (e.g., 2 tokens per expert), and if tokens get assigned or some get dropped
$4.3
Dropless (Gale et al., 2022) MoE with to- ken choice
Expert sharing
Whether to share experts (Dai et al., 2024)
$B.1.1
No shared expert
Sparse upcycling
Whether to start from dense model (Komatsuzaki ct al., 2023; Zhang et al., 2024c)
$B.1.2
Not used
Load balancing loss
Auxiliary loss to penalize unequal assignment to experts harming performance (Shazeer et al., 2017)
$..1.3
Used with weight 0.01
Router z-loss
Auxiliary loss to penalize large router logits that $B.1.4 may cause instabilities (Zoph et al., 2022)
Used with weight 0.001
where called the router, is learned linear layer mapping from the input logits to the chosen experts. A softmax applied the router outputs compute routing probabilities for all Ng experts. Each selected expert F, processes the input the output of which is then multiplied with its respective routing probability. The results are then summed across all chosen Top-k experts to constitute the output of the MoE module for single layer of the model out of its N2 total layers. Key decisions in designing an MoE model include determining the number of activated and total parameters, the design of the experts (e.g., granularity, whether not to include shared experts), and the choice of the routing algorithm. Moreover, training an MoE model can involve initializing from dense model (sparse upcycling) and changing the training objective, such including auxiliary load balancing and router z-losses. We run experiments to investigate each of these design choices in isolation in $4 and B..1. We summarize our final decisions in Table 1: We use 1.3B active parameters out of a total of 6.9B, with 8 activated experts out of 64 per layer. We use dropless token choice routing (Gale et al., 2022): For each input token, the learned router network determines 8 experts to process it. We train OLMoF-1B. 1B-7BB from scratch with two auxiliary losses: load balancing loss (Cr.n) (Shazeer et al., 2017) and router z-loss (Zzz) (Zoph al., 2022), which we define and experiment with in $..1.3 and $B. 1.4, respectively. We multiply them with respective loss weights, and 3, and sum them linearly with the cross entropy loss (Lcg) to arrive at our final training loss:
== LCF + aL + 3L
(2)
For our pretraining data, we mix data from DCLM (Li et al., 2024a) and Dolma 1.7 (Soldaini et al., 2024), which includes: (1) quality-filtered subset of Common Crawl, referred to DCIMM-asellngg (2) StarCoder, Algebraic Stack and arxiv, used in both DCLM and Dolma 1.7, and (3) peS2o and Wikipedia from Dolma 1.7. We refer to our pretraining dataset as OLMOE-MIX. We train for total of 5.13 tokens (1.3 epochs following Muennighoff et al. (2023b)) and provide data statistics in Table C1. Our full pretraining configuration for OLMoEE 1--o-----7- 1B-7B in Appendix C.
Adaptation We create OLMoE- SENNOE-BB-775 IB-7B-INSTRUCT by following standard adaptation recipe split into instruction tuning (Mishra et al., 2022: Wei al., 2022; Sanh al., 2022; Shen al., 2023a; Zadouri et al., 2023) followed by preference tuning (Christiano et al., 2023; Bai et al., 2022; Rafailov et al., 2023) building on prior open models (Tunstall et al., 2023; Ivison et al., 2023; Wang al., 2023). In our instruction tuning dataset, we add more code and math data to boost performance on downstream coding and math applications. Other models, such as GPT-4 (OpenAl et al., 2023) and Llama 3 (Dubey et al., 2024) similarly include samples from math datasets like GSM8k (Cobbe et al., 2021) or MATH (Hendrycks et al., 2021b) during pretraining. We also include No Robots and a subset of Daring Anteater they of high quality and add diversity, two key factors for successful
3

---

Published as a conference paper at ICLR 2025
adaptation (Wang et al., 2023; Zhou al., 2023a; Longpre al., 2023a; Muennighoft al., 2023a). We describe our adaptation datasets in Table C2 and hyperparameters in Appendix C.
RESULTS
Our evaluation procedure consists of three parts: During pretraining (Appendix F), After pretrain- ing, and After adaptation. We detail the setup for each in Appendix
Table OLMOE-IB-7 after pretraining. We compare with LMs of similar active parameters (IB, approximating speed and cost) or total parameters (7B, approximating memory). Model names include rounded parameter counts: model -act ve-total for MoEs and model-tote for dense LMS (leading to differences from official names, e.g., "Gemma2-28" has 2.6B active and total parameters (Team al., 2024c)). We rum all evaluations ourselves with few-shots (Appendix D).
Active
Open Data
Hella-
ARC-
ARC-
Wino- Grande
MMLU
PIQA
params
Swag
Chall.
Easy
LMs with -7-9B active parameters
Llama2-7B
6.7B
X
46.2
78.9
54.2
84.0
71.7
OLMo-7B (0724)
6.9B
V
54.9
80.5
68.0
85.7
79.3
73.2
Mistral-7B
7.3B
X
64.0
83.0
78.6
90.8
82.8
77.9
DCLM-7B
6.9B
64.4
82.3
79.8
92.3
80.1
77.3
Llama3. 1.....1-8B 1-8B
8.0B
X
66.9
81.6
79.5
91.7
81.1
76.6
Gemma2-9B
9.2B
70.6
87.3
89.5
95.5
86.1
78.8
LMs with -23B active parameters
OpenMoE-3B-99
2.6B
27.4
44.4
29.3
50.6
63.3
$1.9
Stablel.MM2B
1.6B
X
40.4
70.3
50.6
75.3
75.6
65.8
DeepSeek-3BB -B-B16 16B
2.9B
45.5
80.4
53.4
82.7
80.1
73.2
JetMoEE2B-99
2.2B
49.1
81.7
61.4
81.9
80.3
70.7
Gemma2-3B
2.6B
X
53.3
74.6
67.5
84.3
71.8
Qwen -5-3B-14B
2.7B
X
80.0
77.4
72.3
LMs with
1B active parameters
Pythia-IB
1.1B
31.1
48.0
31.4
63.4
68.9
52.7
OLMo-IB IB (0724)
1.3B
32.1
67.5
36.4
53.5
74.0
62.9
TinyI.lama-18
1.1B
33.6
60.8
38.1
69.5
71.7
60.1
Llama3.2-1B
1.2B
X
38.2
67.3
43.5
71.6
73.7
62.5
DCLM-
1.4B
48.5
75.1
57.6
79.5
76.6
68.1
OLMoF-1B--BB
1.3B
54.1
80.0
62.1
84.2
79.8
70.2
After pretraining Table 2 we benchmark L.MME--1B-75 on common downstream tasks. We find that OLMOE-1B-7B performs best among models that less than 2B active parameters, making it the most economical option for many use cases of LMs. For larger budgets, Qwen 1.53B- 14B has stronger performance but has more than double the active and total parameters than OLMOE- 1B-7B. We find that despite requiring -6-7x less compute per forward pass, LLMoE-1BB-7. outperforms some dense LMs with 7B parameters such Llama2-7B (Touvron al., 2023b), but falls short of others like Llama3. 1-8B (Dubey et al., 2024). Figure 1 compares MMLU performance with active parameters, a proxy the value model given cost, of OLMOE- 1--BB and other LMs. OLMOE- B-7B is the state of the art in its cost regime.
After adaptation In Table 3, we benchmark our instruction (SFT) and preference (DPO) tuning of OLMOE-1B-77 SFT improves our model all tasks measured. We observe >10x gain on GSM8k, likely due to our inclusion of additional math data to account for the relatively small amounts of math data during pretraining ($2). DPO helps most tasks, especially AlpacaEval. Our DPO model, which we refer to as OLMoE-1B- --B--1-----NSTRCCT has the highest average among all models benchmarked. We find it outperform the chat version of Qwen1 1--3--31444 14B despite Qwen
4

---

Published as a conference paper at ICLR 2025
Table 3: OLMOE-IB-7B after adaptation. Model names contain rounded parameter counts: for MoEs and model-total for dense LMs. We run all evaluations ourselves (Appendix D). Models use different data mixes and setups for adaptation.
Human-
Alpaca- Eval 1.0
Task (-)
MMLU O-shor EM
GSM8k
BBH
Eval
XSTest
IFEal
Avg
Setup (-)
CaT
3-chot
O-shor
O-shot
0-shot
Metric (-)
EM
EM
Pass@10 10
%wiin
F1
Loose Acc
OLMo-IB IB (0724)
25.0
7.0
22.5
16.0
67.6
20.5
+SFT
36.0
12.5
27.2
21.2
41.5
81.9
26.1
35.9
+DPO
36.7
12.5
30.6
22.0
50.9
79.8
24.2
37.4
OLMo-7B (0724)
50.8
32.5
36.9
32.3
80.8
19.6
+SFT
54.2
25,0
35.7
38.5
70.9
86.1
39.7
49.3
+DPO
52.8
9.0
16.6
35.0
83.5
87.5
37.9
49.1
JetMoE-2B-9E
45.6
43.0
37.2
54.6
68.2
20.0
+SFT
46.1
53.5
35.6
64.8
69.3
55.6
30.5
50.4
DeB
37.7
18.5
39.4
48.3
65.9
13.5
+Chat
48.5
46.5
40.8
70.1
74.8
85.6
32.3
$7.0
Owenl.5-33-11
60.4
13.5
27.2
60.2
73.4
20.9
+Chat
58.9
55.5
21.3
59.7
83.9
85.6
36.2
$7.3
OLMoE-IB-7B
49.8
3.0
33.6
22.4
59.7
16.6
+SFT
51.4
40.5
38.0
51.6
69.2
84.1
43.3
54.0
+DPO
51.9
45.5
37.0
54.8
84.0
82.6
48.1
57.7
having >2x more parameters and its pretrained model outperforming OLMOE-IB-7BB in Table 2. The 84% score on AlpacaEval also outperforms much larger dense models on the leaderboard (Li et al., 2023b), such as Llama2- 13B-Chat (Touvron et al., 2023b).
Training loss
Validation loss (C4)
HellaSwag
3.2
3.0
60
3.5
2.8
50
3x less FLOPs
or tokens
40
2.6
30
2.4
10
40
70
100
130
10
40
70
100
130
10
40
70
100
130
3.2
Tokens
Tokens (B)
Tokens (B)
3.0
60
3.5
50
2x faster
2.8
40
2.6
3.0
MoE
30
Dense
2.4
1
6 Training time (h)
2
4 5
7
1
2
3
4
6
7
1
3 4 6
Training time (h)
Training time (h)
Figure 2: MoE Dense.
We train 1.3B parameter dense model and 1.3B active, 6.9B total parameter MoE model, each on 128 H100 GPUs. Apart from MoE-related changes, we train both with the same configuration for 130B tokens. The MoE has 64 experts per layer, 8 of which activated with an FFN dimension of 1,024. The dense model has an FFN dimension of 8,,99. Thus both have the with ~3x fewer tokens (or FLOPs, as both have the same active parameters ignoring the trivial router parameters). Bottom: Due to some memory overhead, this equates ~2x faster training. More results, logs, and configurations: https: : wandb. a /all//i2.2 11m ol olmoe///////oooooooooooooooo htto://m//o/o//////rtess./ reports/ Plot-MoE- M www-enge-----ee Vml Vm1122040MM00
number of active parameters. Top: The MoE reaches the final dense performance
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTING WITH ALTERNATIVE DESIGN CHOICES
This section contains some experiments that led OLMoE- ULMMO--IB-BB with many more in Appendix
4.1 MIXTURE-OF-EXPERTY VS. DENSE
Prior work reports various speed-ups of MoEs over dense models: Artetxe al. (2022) state MoEs require 2-4X less compute to match dense models, MoMa (Lin et al., 2024b) exhibits 2.6x FLOP savings for language tasks, Arctic (Snowflake, 2024b) yields 4x FLOP savings but for very different dense and MoE configurations, and Switch Transformer MoEs (Fedus et al., 2022) train 2-7x faster but for encoder-decoder models while the other works study decoder-only LMs (Radford et al., 2019).
In Figure 2, compare MoEs and dense models in a controlled setup. We find that our MoE reaches the performance of the dense model with -3x fewer tokens equivalent to -3x less compute measured in FLOPs. However, due to the additional memory overhead of training the MoE with its 7B total parameters, processes fewer tokens per second than the dense model (23,600 tokens per second per GPU for the MoE 37,500 for dense). Thus, in terms of training time, it reaches the performance of the dense model only -2x faster. There are likely optimizations possible that would bring the speed-up closer the 3x token speed-up, which leave future work. Based on these results, we select an MoE configuration with 6.9B total and 1.3B active parameters matching OLMo-7B in total and OLMo- IB in active parameter count, respectively (Groeneveld et al., 2024). We provide more reasons for this configuration in Appendix
4.2 EXPERT GRANULARITY
Dai et al. (2024) propose and thus make the model more flexible. For example, the Mixtral model (Jiang et al., 2024) uses the common configuration of 8 experts per layer, 2 of which are activated. This allows for ( 28 com- binations per layer. By halving the size of each expert and therefore doubling the number of experts maintain the same compute and parameter budget, we can increase the possible combinations to
small fine- fine-grained experts to allow more combinations of experts
(16) = 1, 820. Krajewski et al. (2024) investigate compute-optimal granularity configurations finding that higher compute budgets warrant more granular experts.
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
3.0
3.5
Performance
2.8
60
35
3.2
# experts
2.6
3.0
30:
32
2.4
2.8
8
10 40 70
100
130
10
40
70
100 130
10 40 70
100
130
10 40 70
100 130
Tokens (B)
Figure 3: Expert granularity. We vary the number experts in tandem with the FFN dimension to ensure that active and total parameters and thus compute cost remain the same. For example, for 64 experts, the FFN dimension is 1,024 and experts are active, while for 32 experts is 2,048 with active experts. More results, logs, and configurations: https: //wadbd.... a / 11mi1mmmmmmmmmm httmmmmmomoooooo mooe/ reports/Plot Granularity - Wm 1mm1120440TIX0TE.
In Figure we observe that more granular experts improve training loss, validation loss, and downstream performance. The 8-expert configuration uses 1 active expert, which yields (9) combinations. By quartering the size of each expert but increasing the number to 32 with 4 active ones (32) 35, 960 combinations), we observe an improvement of around 10% on HellaSwag and MMLU at around 130 billion tokens. However, we find that there are diminishing returns to granularity. The additional increase to 64 experts with 8 active ones ((41 4, 426, 165, 368 combinations) improves downstream metrics by smaller amount of 1-2%. For OLMoF-11 1B-
)

---

Published as a conference paper at ICLR 2025
7B compute budget' of 3 10 Krajewski al. (2024) predict optimal number of experts of 256 (G 32 in their paper). However, their predictions are for compute-optimat models (Hoffimann et al., 2022: Clark et al., 2022), while train for 5T tokens, which is orders of magnitude beyond what would be conventionally considered optimal for our model size. Thus, their predictions may not extend to our setup, and we stick with 64 experts for OLMoE- 1B-BB also due to the diminishing returns in Figure 3.
4.3
EXPERT CHOICE VS. TOKEN CHOICE
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
3.5
Performance
30
3.5
50
3.0
28
3.0
TC
2.5
30
EC
26
10 50 100 150
200
10 50 160 150 200
10 50 100
150
200
10 50 160 150 200
Tokens (B)
Figure 4: Expert choice (EC) vs. token choice (TC). Both models have an 8-expert MoE in every 2nd layer. For TC, experts are activated per token, while for EC the capacity factor is 2. Thus, both models use the same number of active parameters. More results, logs, and configurations: https: / //waadb... ai a aii22221 1112mmmmmmmm moe reports /oltt EC-
The MoE router determines which experts process each input token ($2). There are two common types (Liu et al., 2024b): expert choice (EC) (Zhou et al., 2022) and token choice (TC) (Shazeer et al., 2017). For EC, each expert selects fixed number of tokens per sequence. By design, this leads to each expert processing the same number of tokens. This is the main benefit of EC as ensures perfect load balance, which improves training throughput and removes the need for load balancing loss. The main issue of EC is that is impractical for autoregressive generation where a single token is processed each step rather than the entire sequence one (Raposo al., 2024). Another potential downside is EC can lead to token dropping, where some tokens are not selected by any expert, which can hurt performance (Gale et al., 2022). At the same time, can lead some tokens being processed by multiple experts, which could also be beneficial as it allows the model to allocate more compute to some tokens (Zhou et al., 2022). For TC, each token selects a fixed number of experts. This can lead to many tokens choosing the same expert, hurting training efficiency. Thus, TC is often used with a load balancing loss (Shazeer et al., 2017) to encourage equal distribution.
In Figure 4, we benchmark EC and TC. We find that TC outperforms EC for the same token budget for tasks depicted. While Zhou et al. (2022) find EC to be better, our configuration slightly differs in that use dropless MoEs (Gale et al., 2022) with load balancing loss. Thus, our TC variant is expected to perform better than the TC variant in Zhou et al. (2022). We confirm findings that EC runs around 20% faster 29,400 tokens per second per device versus 24,400 for TC (Zhou et al., 2022). EC may be more beneficial in a multimodal setup (Lin et al., 2024b) as dropping noisy image tokens is likely less harmful than text tokens. Thus, while we stick with TC for this release of OLMoE we may revisit EC for future multimodal models.
5 MoE ANALYSIS
By advancing open and cost-eflicient models ($1), OLMoE-1B-77 enables new research into LMs and MoEs. Making use of our released intermediate checkpoints, data, and code, we define and analyze four properties specific to MoEs: Router saturation ($5.1), Expert co-activation ($5.2), Domain specialization ($5.3), and Vocabulary specialization ($5.4).
5.1 ROUTER SATURATION
Router saturation, as function of time f, represents the proportion of overlapping activated experts between the final checkpoint and an intermediary checkpoint at time Router saturation thus
Approximated via 6 N D (Kaplan al., 2020) with active parameters N (IB) and tokens (5T).
7

---

Published as a conference paper at ICLR 2025
Top-k=1
Top-k=8
Router saturation (%)
80
60
Layer ID
40
0
4
8
12
- 5 _
9
13
20
6
10
14
3
7
11
15
10
20
40
10
20
40
Pretraining stage (%)
Figure 5: Router saturation during pretraining measured on random 0.5% of the C4 validation data. We compute saturation by comparing the routing to the top-k experts at four intermediate checkpoints (1, 10, 20, and 40% of pretraining) the final pretraining checkpoint (Equation 5).
corresponds to whether the router weights are still learning which expert will process certain data. value of 100% indicates that the router at the intermediate checkpoint will route to the same experts as the final checkpoint router. See $H.1 for the detailed formula used to calculate the value.
Figure 5 shows that, after 1 of pretraining, up to -60% of routing to the top-8 activated experts has already saturated (right). Thus the model already uses the same experts for given input data as it will at the end of pretraining. This early saturation aligns with prior work (Xue al., 2024). At 40% of pretraining, saturation reaches up to -80%. However, which top-1 expert has the highest routing probability saturates slower (left). We find that routing in later layers saturates earlier during pretraining. Layer 0 is an outlier saturating significantly more slowly than other layers. Dai et al. (2024) do not the first layer. This is likely linked to our findings on saturation. Because routing in the first layer saturates slower, the experts that certain input data get routed to frequently change. These changes may lead to one expert suddenly getting significantly more data than others thereby impairing load balancing. We are excited about future work further investigating what happens in the first layer by building our open release.
MoE in the first layer as they find that load balaneing converges more slowly for
Layer
60
5.2 EXPERT CO-ACTIVATION
56
45
We define expert co-activation as the proportion of times two specific experts, E, and E,. are simultaneously activated out of the total number of activations of one of them. A co- co-activation of 100% indicates that if E, is activated, E, also always activated. A value of 0% indicates that the experts never co- occur. See 1 for the formula used to calculate the value.
19.
49.
30
15
59
No inin Aoinnsnii eewen benr
Figure 6 shows there is no strong co-activation among experts in layer 7, with only few exceptions. This may indicate that there is little redundancy across different experts. Layers 7 and 15 (Figure H1) show similar co-activation patterns with several groups of 3 or 2 experts that tend to get activated together. We investigate tokens that activate these experts in $5.4. Further, in $H.2 (Figure H8), we investigate whether experts across layers, rather than within one layer, tend to process tokens together.
Co-activation of OLMoE-1BB7B experts random 0.5% of the C4 validation data. We display the 32 experts with the highest maximum co-activation score via their expert IDs on the and y-axis. See Figure HI for Layer 0 and 15.
Figure 6:
5.3 DOMAIN SPECIALIZATION
We define domain specialization the specialization of expert E, to domain D, specifically the proportion of tokens from a particular domain that get routed to a particular expert E, (see $ for the formula). A value of 100% indicates that all data from that domain is routed to F., whereas
8

---

Published as a conference paper at ICLR 2025
GitHub
arXiv
Books
C4
OLMoE Layer 0
OLMeE Layer? 7
Mixtral Layer 0
Mixtral Layer?
100
100
Domain specialization (%)
vak=
Domain specializenn specialization (%)
100
100
100
100
0
100
100
16
32
Expert ID
Expert
Figure 7: Domain specialization of OLMO (left) vs. Mixtral-8x7B (right). We visualize how often tokens from different domains get routed to the 64 (OLMoE) or (Mixtral) experts after pretraining. We consider tokens routed any of the k 8 (OLMoE) experts (Equation 7), Horizontal gray lines are random chance or uniform routing (8/64=12.5%% per expert for OLMOE-1B-77 with 8 active out of 64 total experts per layer and 2/8-25% for Mixtral with 2 active out of 8 total experts per layer). See Figure H7 for k = results.
k 2 (Mixtral) active
0% indicates the expert is never used for that domain and can be removed from the model without affecting performance in that domain.
Figure 7 (left) shows many examples of experts that are activated significantly above or below random chance for specific domains. E.g., for arXiv, which has a very specific distribution with lots of scientific text, the first expert in layer O is nearly 100% specialized. This suggests that there is little redundancy in the knowledge of the experts in OLMOE-IB-7B, kinds of data. GitHub and Xiv are often activated together in layer 7, which we explore further in $5.4. For generic domains, such as C4 (Raffel et al., 2023), which kinds of data, expert activations in OLMOE- 1 B-7B are much more balanced. This highlights that the load balancing ($3.1.3) works as intended and the model makes proper use of all experts for generic data. Mixtral-8x777 (Jiang et al., 2024) in Figure 7 (right), however, exhibits little domain specialization across both unique and generic domains. Experts are activated close to the uniform routing baseline for all layers and domains. Thus, there may be more redundancy across experts in Mixtral, as they likely contain similar knowledge. We hypothesize that this is due to Mixtral being upcycled from Mistral (Cai, 2023). The initialization from a dense model may limit the amount of possible specialization in the experts as they all start from the same local optimum. This is likely why training from scratch eventually outperforms upcycling our pretraining experiments (.B.1.2).
they specialize in different
crawl containing various
5.4
VOCABULARY SPECIALIZATION
Vocabulary specialization refers to how specialized a particular expert is on a token ID x (also called vocabulary element), defined as the proportion of tokens with token ID that are routed to one particular expert E; out of all experts in that layer. We distinguish input and output variants of this specialization, where is either the input token ID or the next output token ID (either the ground-truth next token ID or the token ID predicted by the model). A value of 100% indicates that all occurrences of that vocabulary element, input data is routed to E,, whereas 0% indicates an expert that is fully irrelevant for that vocabulary element and can be effectively removed from the model without affecting performance whenever the token ID appears.
In Figure H2 we find that vocabulary specialization is higher in later layers, similar to how later layers saturate earlier (55.1). Later layers also specialize more on predicted output token IDs rather than input token IDs, i.e., the routing is decided more by the token the model is about to predict rather than the original input token. This intuitive as in earlier layers there is more uncertainty about which token the model will predict. At ~90% expert 27 specializes the most, which find in Table 4 to activate for many non aa--haehhettcc tokens, such as Cyrillic and Devanagari letters. Expert 43 shows specialization geographic terms in both input and output tokens. Experts 48 and 23 both focus

---

Published as a conference paper at ICLR 2025
Table 4: Vocabulary specialization in the 7th layer of OLMOE-IB-7B. We use k (Equation 8) and random 0.5% of the C4 validation data excluding token IDs with <10 appearances.
Expert ID Input token IDs
Predicted output token IDs
27
C (100%)
(100%)
(100%)
(100%)
(100%) $ (100%)
(100%) J (100%)
(100%)
(100%)
(100%)
(100%)
(100%)
(100%)
S
(100%)
R
(100%)
LU (100%) NH (100%)
a
(100%)
c
(100%) (100%)
(100%)
43
Armenian (100%) ijan (100%) enia (96%) enia (90%) invasion (80%) Arabia (76%)
Iraq (95%)
Iranian (92%)
Iran (92%) inregular (66%) regions (64%)
border
Saudi (90%) northern (90%) Lebanon (63%) Kong (61%) 1) ians (61 ())
hases
(90%)
Singapore (88%) Turkey (88%)
(60%)
Republic (59%) Ireland (58%)
4
sq (89%) Main (70%) reversal (69%) YR (90%) Character (88%)
(77%) Os
YR (63%) GC (56%) Overall (50%) 79
(76%) GHz (71%) fluence (60%) amycin
(50%)
main (50%) RE (46%) PCR (46%)
(60%) pixels (56%)
(53%)
48
compared (42%)
(41%) Then (41%)
except (60%) tennis (41%) Marks (40%)
(40%) ). (35%) (35%) instead (33%)
Dunn (33%)
tears (30%) Arizona
(30%)
23
(58%)
Therefore (55%) So (46%)
(53%)
Republican
(50%)
Jack
(46%)
And (44%) According (41%)
(47%) THIS (40%) Democratic (40%)
(41% % (40%) ?" (38%) But (38%)
according (39%) So (38%) Step (33%)
3
grandmother (92%) brother (91%) Daisy
hood (36%) mother (35%) inde (31%)
(83%) daughter (78%) mum (75%)
boy
(29%) girl (28%)
married (27%)
connector words, such as Then and Therefore This is likely because they commonly process tokens together with high acteivavtiion of 60% in Figure 6. Based on our findings in $5.3 that for GitHub and arXiv often the same experts in layer 7 activate, we display one such expert (expert ID 4) in Table 4. It seems specialize in measurements, such as sq YR (year), and GHz These common terms in scientific papers corresponding the arXiv domain and likely also in GitHub code for computations related to measurements. They are less likely to appear in books, which explains the low activation of expert ID 4 in layer 7 for book data in Figure 7. Expert 3 among the three most active experts of layer 7 for book data in Figure 7 (fourth yellow bar for layer 7). This resonates when looking its specialization on family terms in Table 4 (final row), which are far more common in books than scientific papers code. Overall, domain specialization and vocabulary specialization closely linked to one another, as domains are usually characterized by their distinct word distribution. In Figure H5, we link them more closely by comparing the extent of vocabulary specialization across domains and expert IDs. In Figure H3 and Figure H4 we also find that OLMoE-1B-77 exhibits stronger vocabulary specialization than Mixtral-8x7B.
6 CONCLUSION
We open-source OLMOE-1B--RR and OLMOE--BB-7--NSTTRIIE including model, data, code, and logs. IB active and 7B total parameters, our models yield state-of-the-art performance among models with similar amount of active parameters even outperforming larger models including DeepSeekMoE- 16B and Llama2-13B-Chaa We share ~20 training experiments yielding novel insights into Mixture-of--xperts
Further, define and analyze new properties of MoDs showing that OLMo E. 1--BBBBB exhibits early router saturation, weak expert co-activation, and some evidence of domain and vocabulary specialization. We intend our fully open release to serve basis for more research into MoEs given their critical importance ($1). We are excited about more iterations of OLMoE to close the gap between frontier models and fully open models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teadoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi-Ling Chen. Parul Chopra, Xiyang Dai, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Victor Fragoso, Dan Iter, Mei Gao, Min Gao, Jianfeng Gao, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi. Xin Jin, Piero Kauffmaan, Nikos Karampatziakis, Dongwoo Kim. Mahoud Khademi. Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars I Liden, Ce Liu, Mengchen Liu, Weishung Liu, Fric Lin, Zeqi Lin, Chong Luo, Piyush Madan, Matt Mazzola, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Recker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase. Olli Saarikivi, Amin Saied. Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Swadheen Shukla, Xia Song, Masahiro Tanaka. Andrea Tupini, Xin Wang, Lijuan Wang. Chunyu Wang. Yu Wang. Rachel Ward, Guanhua Wang. Philipp Witte, Haiping Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang. Jianwei Yang. Ziyi Yang. Yifan Yang. Donghan Yu, Lu Yuan, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang. Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: / /wwp/vaaavvvv///v/////////aaaaaa v. org/ / abs/ 24044 14219.
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Open foundation models by Ol.ii, 2024. URL ht https: http///axvvv/.////e///ee/ea.a org/ abs / 2200334..0 04 652.
Joshua Ainslie, James Thorpp Michiel Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URL
https: / //wpivaaavvvv/aa//////.a..aaaa wtt/abs/////e/.....//..g//..a /ttp//abb/200..22255333 13245.
Alon Albalak, Yanai Elazar, Sang Michael Xic, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff. Bairu Hou, Liangming Pan, Haewon Jeong. Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024. URL https://arxiv: org/abs abs 2202. 18277
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars!, 2023.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024. URL https: http::::///ggggggga/ceee co/blog/ smollm.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Danicle Mazzotta, Badreddine Nounc, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey. Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey. Yanping Huang. Kathy Meier-Hellstenn Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder. Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann. Lucas Gonzalez, Guy Gur-Ant Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li. Hyeontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Sacta, Rajkumar Samuel, Renee Shelhy, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang. Pidong
11

---

Published as a conference paper at ICLR 2025
Wang. Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023, URL https: /tt./aaa.v.o/o.////eaa wtwp/b///////eeeeeeeee/eeeeeee// 2305. 10403.
Quentin Anthony, Stella Biderman, and Hailey Schoelkopf. Transformer math 101. https://blo eleuther. wtw.aa/rraoooomr-mmaate 2023.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman. Xian Li. Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O' O'Horo, Jeff Wang. Luke Zettlemoyer. Mona Diah, Zomitsa Kozareva, and Ves Stoyanov. Efficient large scale language modeling with mixtures of experts, 2022. https:::: / /awxiv v. org/ abs 22122.112.. C684.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton, Layer normalization, 2016.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin. Runji Lin. Dayiheng Liu. Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. 2023a.
Jinze Bai Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023b. URL https:tt////t /tta::///////i wwwpaaawwwwi...aaaaaaaaeaaeaaeaa als 2308 12966.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-I Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, 7ac Hatfield-Dodle Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback, 2022. URL https: / /wwxvv.vvvvv. www.xiii..rr..oooo crg/ abs 2. 12. 08073.
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung. Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, Meng Lee, Emad Mostaque, Michael Pieler, Nikhil Pinnaparju, Paulo Rocha. Harry Saini, Hannah Teufel, Niccolo Zanichelli, and Carlos Riquelme. Stable Im 1.6b technical report, 2024. URL ht tps: /arrivvvvvv/////a//.aaa/aaaaavva www.v.owg........aaa.o.....oooot alss 2402 17834.
Emmanuel Bengio, Pierre- Luc Bacon, Joelle Pincau, and Doina Precup. Conditional computation in neural networks for faster models, 2016. URL httpt://// wwwpawwwwg/aa...aaat.......w.ort /b/////////// 1151..0697..
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herhie Bradley, Kyle O'Brien, Eric Hallahan, Moham- mad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https:////// arxiv. org/ ab///b////// www.////...... 01373.
Stella Biderman, Hailcy Schoelkopf, Lintang Sutawika, Lco Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Lee, Haonan Li, Charles Lovering. Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the trenches on reproducible evaluation of language models, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https: / /ttxxvaxivv///////////e///aaaaaa www.aiiww.g..............o /191//19111111 11641.
Sid Black, Ieo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive lan- guage modeling with mesh-tensorfoow, 2021. URL htt.hhttss////e //oii http:///oooooo////.////.. 1 D. 5281 1 /enodoo. 5297711
12

---

Published as a conference paper at ICLR 2025
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding. Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20bb An open-source autoregressive language model, 2022.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betly Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index, 2023. URL https://aaaxiii. abs/ 1910.12941.
Tom B. Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020.
Tianle Cai. 173418874911715322
Mixtral from mistral, 2023.
URL
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Gc, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang. Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong. Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lx, Haijun I.x, Kai Lv, Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song. Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang. Jiayu Wang. Rui Wang. Yudong Wang, Ziyi Wang. Xingjian Wei, Qizhen Weng. Fan Wu, Yingtong Xiong, Chao Xu. Ruiliang Xu, Hang Yan, Yirong Yan. Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu. Yuhang Zang, Chuyu Zhang. Li Zhang, Pan Zhang, Peng Zhang. Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang. Xingcheng Zhang, Xinyue Zhang, Hui Zhao. Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Gen- erative pretraining from pixels, 2020. URL httpps:/cctt openai aeteaai.coo//aaee cww/papers///////aaaoaooa ppppprraeeeeeaaiieeeee Generative prvetraining_ffee wwrnnn__roo____ee Pixels __...ddf........
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Paylov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Soumith Chintala. Gpt-4 moc, 2024. 1671267150101121099
URL http:://x..com//omitttttttt
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roherts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways, 2022.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep reinforcement learning from human preferences, 2023.
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoe, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford. Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre. Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models, 2022. URL https::///aaaa org/ 2.0/ 01169.
Christopher Clark, Kenton Lee, Ming-Wo Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL abs/1 1905. 10044.
org/
13

---

Published as a conference paper at ICLR 2025
Peter Clark. Isaac Cowbey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL
buttp::/arxive wwtp//iii..../aaaaaoo /ts///////aaaoooo///////aaa/eaea http//aass////../ 1801. 05457.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Together Computer. Redpajama: An open source recipe to reproduce llama training datasct, 2023, URL https::/git thw.b.ttttuu... hwt.ggtth......ttteeeeeeeee wwom//togeteercomttteee////// /RedPa jama jama-Data.
Róbert Csordás, Kazuki Iric, Jürgen Schmidhuber, Christopher Potts, and Christopher D. Manning. Moeut: Mixture-ofeexperts universal transformers, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Damai Dai, Chengqi Deng. Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024.
URL https:/ /tappii/////xii..///aap/ap orgg abs /ts.///11l.. 66066.
Datahricks.
Dbrx, 2024.
URL open- 11m.
https://ww.aattbriccces
introduiing- wwwprxxxnwwwttttt-eeeeeee
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, 2017. URL http:: ptttpps///rxivvv /wwpiv.vvvv..////a/////////aaa.a org/ wbsp//12../.eee..e.eeeeeeeeeeeee 08083.
DeepSeek-AL, :. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong. Qiushi Du, Zhe Fu, Huazuo Gao. Kaige Gao. Wenjun Gao, Ruiqi Ge. Kang Guan, Daya Guo, Jianzhong Guo. Guangbo Hao, Zhewen Hao, Ying He. Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang. Fangyun Lin, A. X. Liu, Bo Liu. Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nic, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang. Shiyu Wang. Yaohui Wang, Yongji Wang. Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseck I1m: Scaling open-source language models with longtermism, 2024a.
DeepSeek-AL. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang. Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Hanwei Xu, Hao Yang. Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao. Kang Guan, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang. Minghui Tang. Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang. Qihao Zhu. Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin. Ruiqi Ge. Ruizhe Pan. Runxin Xu, Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu. Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Liu, Xin Xie, Xingkai Ya, Xinnan Song. Xinyi Zhou, Xinyu Yang. Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang. Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang. Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma. Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren. Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zähui Gu, Zilin Li, and Ziwei Xie. Deepseck-v2: strong, economical, and efficient mixture-of-experr language model, 2024b.
Mostafa Dehghani, Stephan Gouws, Orial Vinyals, Jakob Uszkoreit, and Fukasz Kaiser. Universal transformers, 2019. URL https: / /awxi v. wwwggbb////////eooooo/ooosss/s/s http///ss/1/00// 101111187800000 9.
14

---

Published as a conference paper at ICLR 2025
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arna,, Xiao Wang. Carlos Riquelme, Matthias Minderer, Joan Puigoerver, Utku Evei, Manoj Kumar, Sjoerd van Steenkiste, Gamalcidin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay. Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučié, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters, 2023.
Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane. William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestmess, Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. URL https: arxiv. org/ abs /b34 03208.
Danny Driess, Fei Xia, Mehdi M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023. URL https: / aarii.iiiiiv///.. org/ abs /2003/333/. 2333.03378.
Nan Du Yanping Huang. Andrew M. Dai, Simon Tong, Dmitry Lepikhin. Yuanzhong Xu, Maxim Krikun, Yangi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang. Yu Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoe V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts 2022.
Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan. Tricks for training 08246.
sparse translation models, 2021. URL https: http///xxwvccrggabaa/a//e
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle. Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, et al. The llaaa 3 herd of models, 2024. URL https:://///.. http/aaxiivvoooo..//ooo
2407 21778.2188 3...
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators, 2024. URL https: http:///axvv.....e.. /204 04475.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations a deep mixture of experts, 2014. URL https: //atts:://axiv... .rgg abs/1112 4314.
Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, and Kristoffer Laigaard Nielbo. The scandinavian embedding benchmarks: Comprehensive assessment of multilingual and monolingual text embedding, 2024. URL https: /awpiviivii.////a//.aaa//aaaa..a org/ abtp///../222.../// 02396.
Kawin Ethayarajh. Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Croissantllm: A truly bilingual french-english language model, 2024. URL http:::///// /arxiv.ivvv.////ii www.xii.rr///..ee..o org/ alss 2402. 00786.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022.
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mereat, Alex Fang, Jeffrey I.,, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilaarco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave. Reinhard Heckel. Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and downstream tasks, 2024.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-off-expers 2022. URL ht tattps//aaat //ap:i////vvv..///// alss 2211
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Naheshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https: /ttp//aavvv////////v//..//...a wttp://////b/////g 00027.
15

---

Published as a conference paper at ICLR 2025
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, A framework for few-shot language model evaluation, September 2021. URL https: /cci//iii//// crg/ /0/.//1...0 5281/ zenodo. 5371628.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Yin. Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong. Mingdao Liu, Minlie Huang. Peng Zhang, Qinkai Zheng. Rui Lu, Shuaiqi Duan, Shudan Zhang. Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang. Xixuan Song. Xunkai Zhang. Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong. Zehan Qi, Zhaoyu Wang, Zhen Yang. Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b glm-4 all tools, 2024.
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model, 2024. URL https: /arxivv...// org/ /
2405.16712.
Andrew Gordon, Zornitsa Kozareva, and Mclissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning, 2012. URL http:s:/aacaaathoo.o org/s12-100
Dirk Groeneveld, Anas Awadalla, Iz Beltagy, Akshita Bhagia, Ian Magnusson, Hao Peng. Oyvind Tafjord, Pete Walsh, Kyle Richardson, and Jesse Dodge. Catwalk: A unified language model evaluation framework for many datasets, 2023. URL https: //tr///aaiiv wwtpsiiiiirrreeeeeeeeee org/ htt///bss2/2/// 10253.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khy- athi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nis- hant Subramani, Mitchell Wortsman, Pradeep Dasigi. Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo. Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating science of language models, 2024.
Sam Gross, Mare' Aurelio Ranzato, and Arthur Szlam. Hand mixtures of experts for large scale weakly supervised vision, 2017. URL https: //axp/i://////////////a//sss/sss www.oorgg///..//....o... abs///bs//11/ 7714/774444 704 6636..
Yuling Gu, Oyvind Tafjord, Bailey Kuchi, Dany Haddad, Jesse Dodge, and Hannanch Hajishirzi. Olmes: standard for language model evaluations. 2024. URL https: /aaaaaaxvvvvrr..aaaaaaa. aa///b///24/0 08446.
Suriya Gunasekar. Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi. Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lec, and Yuanzhi Li. Textbooks are all you need, 2023. URL httph:ttp///cc //axxiv. abs 11644.
Xu Owen Hc. Mixture of million experts, 2024, URL htt.tttts///xxv..coom
/207 04153.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL htt.t:/tps//ee :/attpip:///rxv...// org/ abs 2009.. 03300.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Mcasuring mathematical problem solving with the math dataset, 2021b. URL hpsps: // arxiv. wwwg.org/aaa.. abt///b///// / 2103.03874.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy. Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL httpt::tt//// /awwvv org/ abs///bs//22/// 155556.
Jiwoo Hong, Noah Lee, and James Thome Orpo: Monolithic preference optimization without reference model, 2024. URL http:: /aaxxi wwwp:wwwwww////////////// 07691.
16

---

Published as a conference paper at ICLR 2025
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang. Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jic Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Miniepm: Unveiling the potential of small language models with scalable training strategies, 2024.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, lan Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer, 2018. URL https:/ //tp:///aavvvv//a org/ abs / /1809,800000 04281.
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi. Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023.
Sebastian Jaszczur, Aakanksha Chowdhery. Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse enough scaling transformers, 2021. URL https://arxiv org/abs /20 2211111111111111111 12763.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford. Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antomiak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL
https: httppp://aaxvv.../ wwt./brg//////..e...oo..eoe. abs htsp//bs/20000 /2001. 08361.
Andrej Karpathy, Llm model size competition is intensifying... backwards!, 2024. URL httpp:://// com/ Saappathy/sttatuu/1144488833399
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Casey A Fitzpatrick, Peter Bull, Greg Lipstein, Tony Nelli, Ron Zhu, et al. The hateful memes challenge: Competition report, 2021. URL htttps://prcceedd ing. mml...........................mm mlr. press htt////33////eee //lllllllllllla htmi.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Werra, and Harm de Vries. The stack: tb of permissively licensed source code, 2022, URL https::/arxiv..ot / 2211. 1 15533.
Aran Komatsuzaki, Joan Puigcerver, James Lee- Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslic, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints, 2023.
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóčdz, Piotr Sankowski. Marek Cygan, and Sebastian Jaszezur. Scaling laws for fine-grained mixture of experts, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu. Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing Frontiers Open Language Model Post- Most-Trannin, 2025. URI. https: : //aer///reee aaaii.. www.wvvw..................oo.... oorg/////aaooo....o..... aag//b///// 2241 15124.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020. URL https: http///xxvv.rrrgee/e
/ 2000000000222220 6668.
Mike I Lewis, Shruti Bhosale, Tim Detmmes, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying 66766
training of large, sparse models, 2021. URL
17

---

Published as a conference paper at ICLR 2025
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsich, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang. Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zeutlemoyer Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang. Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp- In search of the next generation of training sets for language models, 2024a.
Margarct Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train--merg:: Embarrassingly parallel training of expert language models, 2022. URL https: // //aaxivv.eo .rrg abs/ 22082200222 03306.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!, 2023a.
Xuechen Li. Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Guirajani. Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instmutiion-oollowing models, 2023b. URL https: / ttt.thb...ug.gggg/g///ggg......t wtw/tatttt//// htmm//atssuu-tt lab/ a ppaca_eval.
Yuanzhi Li, Séhastien Bubeck, Ronen Eldan, Allie Del Giomo, Suriya Gunasekar, and Yin Tat Lee. Texthooks all you need i:: phi-1.5 technical report, 2023c. URL wwtpp::///aai.../eeeaeepp www.://ggg/aaaaa//oooo/o...ssssa /2009. 05463.
Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal Ilms with mixture of experts, 2024b. URL httpt://p//aeee //axxiv org/ a20/ 11273.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Cc Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong. Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji. Omar Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Korceda. Holistic evaluation of language models, 2023. URL https: / /wwxiv..vvvvvv////// org/ abs/ / 22 1. 1.11.19111 09110.
Opher Licber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamba language model, 2024. URL https: // /ww.ai.wggg/aaaaaa.......ao..aaa abs 2403 19887.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning, and Li Yuan. Moc-Ilava: Mixture of experts for large vision-language models, 2024a. URL https: htt///aaxvvvv/. http://aaxxvv...abbeeesoe/ wwt/abs.////oooooooooo..oo.....s 2401. 2401.15947.
Stephanic Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Gosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: Efficient furlon pre-training with mixture of modality aware experts, 2024b. URL https: / arxiv, ortp:////aabbbbbbb/bb/// 21770.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin, Regmix: Data mixture as regression for language model pre-training, 2024a. URL https:
// ///arxiv.orgg abs/ 2407. 0440.014922 01492.
Tianlin Liu, Mathieu Blondel, Carlos Riquelme, and Joan Puigcerver. Routers vision mixture of experts: An empirical study, 2024b. URL https: http///axvvvv////// org/ abs /
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger. Hongyi Wang. Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto. Liping Tang, Nikhil Ranjan, Zhiqiang Shen. Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. Ilm360: Towards fully transparent open-source lms, 2023.
18

---

Published as a conference paper at ICLR 2025
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023a. URL https: /ttp/arxiivvv//// org/ abs/ 2301 13688.
Shayne Longpre, Rohert Mahari, Anthony Chen, Naana Oheng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kahbara, Kartik Perisctla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, and Sara Hooker. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai, 2023b. URL ht // /aaxivivvvv.////ia org/ abs/ 2albs/230 2310. 2310116787777 16787.
Shayne Longpre, Robert Mahari, Ariel Lee, Camphell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu. Tobin South. Cole Hunter. Kevin Klyman, Christopher Klamm, Hailey Schoelkopf. Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Chitongo. Da Yin. Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole. Jianguo Zhang, Joanna Materzynska. Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Mucnnighoff, Seonghycon Yc, Scungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. Consent in crisis: The rapid decline of the ai data commons, 2024. URL https: //tpss///xvvvv.o/ org/ abs abaaabas2220 /2407 11777.149 14933.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2019.
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan. Ryan Ignatius, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitchouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu 7hang, Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn, Muhammad Farid Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov, Wei Qi Leong. Quyet V. Do. Niklas Muennighoff. Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Ngee Chia Tai, Ayu Purwarianti. Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji. Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng-X Xin Yong, and Samuel Cahyawijaya. Scacrowd: A multilingual multimodal data hub and benchmark suite for southeast asian languages, 2024. URL https: //arxiv. org/ abs/ / 2406 10118.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Tery Yue Zhuo, Evgeni Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas KrauB, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL https: : / /aaxxivv.... www.i..wgg/.....o.........ooo abs 22022 19173.
Risto Luukkonen, Ville Komulainen, Jouni Luoma. Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala. Niklas Muennighoff, Aleksandra Piktus, Thomas Wang. Nouamane Tazi, Teven Le Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. Fingpt: Large generative models for small language, 2023.
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannanch Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge Paloma: A benchmark for evaluating language model fit, 2023. URL https: / arxiiv .rrg/ abs//bss/3333 10523.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanject Singh, Doug Kang, Ankur Jain, Hongyu Hè. Max Schwarzer. Tom Gunter. Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang. Ruoming Pang. Peter Grasch, Alexander Toshev, and Yinfei Yang. Mml: Methods, analysis & insights from multimodal I1m pre-training. 2024. URL https: /atxivvvvvvvv////////.../....v.a www./os.//./..oooooooo........./ ab/////s//233... C9611.
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao. Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. Openelm: An efficient language model family with open training and inference framework, 2024.
19

---

Published as a conference paper at ICLR 2025
Yu Meng, Mengzhou Xia, and Dangi Chen. Simpo: Simple preference optimization with reference-free reward, wawpgg/////////e 240...2 .45444344
2024. URL https:/ http:////axxvvv.rreeoe
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
URL atttps://araie wtt.:/iw.vvrgaaaaboooo wttp://gg////////////////// 07843.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https:://aatt /ttp://.wwva.ooeee....../.oo abs /aba/11 1809. 02 78
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task generalization via natural language crowdsoureing instructions, 2022.
Niklas Muennighoff. Vilio: State-of of-te-tf visio- Iinuuiiigiiitic models applied to hateful memes, 2020.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng. Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang. Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models, 2023a.
Niklas Muennighoff Alexander M. Rush, Boaz Barak, Teven Le Scao, Alcksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023b.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xir Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanic, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2023c.
Niklas Mucnnighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanprect Singh, and Douwe Kiela. Generative representational instruction tuning, 2024. URL https: / a arxiv, org/ www./rrg//////o///e//ooo/eeeeer/ abs/2402. 09906. 6.
Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing, 2024. URL https: / /arxiv..vvvvv.////va//.aa//aawiv org/ abs/ 22066 10330603370 03745.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby, Multimodal contrastive learning with limoe: the language-image mixture of experts, 2022. URL https: ftttp:://arxivv http://awxvv.rr///// /bs/ 2206. 02770.
Nvidia, :, Bo Adler. Niket Agarwal, Ashwath Aithal, Dong H. Auh. Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong. Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg. Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu. Eileen Long. Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Norozzi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prahhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat. Sanjeev Satheesh, Jane Polak Scoweroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun. Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https: : arxiv, org/ abs/ /24/22466222 11704.
Team OLMo, Pete Walsh. Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert. Dustin Schwenk, Oyvind Tafjord, Taira Anderson. David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Validad Miranda, Jacob Daniel Morrison, Tyler C. Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Chris Wilhelm, Michael Wilson, Luke S. Zettlemoyer. Ali Farhadi, Noah A. Smith, and Hanna Hajishirzi. 2 OLMo 2 Furious. arXiv preprint, 2024. URL https:/
/ttps://ap.... www.nteetti..e cettccicshaarrr argg/CorpusID 2752::::2222222 752130993
OpenAT, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida. Janko Altenschmidt, Sam Altman, al. Gpt-4 technical report, 2023.
Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training, sparse inference: Rethinking training of mixture-of-experts language models, 2024. URL htl.https:/// https//////xxvvxaeaeee 2204 05567.
20

---

Published as a conference paper at ICLR 2025
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu, Patrick LeGresley, Denys Fridman, Jared Casper, Ashwath Aithal, Oleksii Kuchaiev, Mohammad Shoeybi, Jonathan Cohen, and Bryan Catanzaro. Nemotron-4 15b technical report, 2024. URL https: //arxiv. orgg abs/ 2402. 16819.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023. URL https: //arxiv org/ abs 220//3366.00 01116.
Bo Peng, Fric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung. Matten Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong. Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom. Atsushi Saito, Guangyu Song. Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao. Peng Zhou, Qinghua Zhou. Jian Zhu, and Rui-Jie Zhu. Rwkv: Reinventing mns for the transformer era, 2023.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr. au2. Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song. Haoqin Tu, Stanislaw Wožniak, Ruichong Zhang. Bingchen Zhao, Qihang Zhao. Peng Zhou, Jian Zhu, and Rui-Jie Zhu. Eagle and finch: Rwky with matrix-valued states and dynamic recurrence, 2024.
Ofrr Press and Lior Wolf. Using the output embedding to improve language models, 2017. URL https: // /laaaiivorgg///aee.e.........a.e /aarg/////iiii 608. 05859.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language wtt.:///wwcpkkww..llees...
models are unsupervised multitask learners, 2019. URL net peet/btttrr-aatee wawtttrr-nnngggeemmneeeee aaggg/aaaaaaadd///agaagngaaaaaaa htt.://ugg..cmoolll.asaauuuuueee learners. .df.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. 2022. URL https:ttt//eet.. http:/:://///v/aaaae/eeeee org/ abs/ /212. 04356.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 2023.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots, 2023. URL https: http:::tttgggpiiiieeeeee http///hggggggggeeeeee wttp://aaaaaeeeaeeeaaaaaeaeeaaae http//////aaaaaoeaaeaaa wtw.aaetttttggiiiiie ggggginggggggggggeggngnggggggggg a cel4 aaee///oooobottt._
Samyam Rajbhandari. Jeff Rasley. Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020.
Samyam Rajhhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan. Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of experrsss inference and training to power gexeratnen ai scale, 2022. URL https: /waxivivv.v..///////a///aaaaaeva www..iww./w...w...........o.ooor org/ abaaabbs//000 /2001. 05596.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixureeoof-eepts: Dynamically allocating compute in transformer-hased language models, 2024. URL 2404. 02258.
/etts::/arxiv..r
Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer. M2d2: A massively multi-
language modeling dataset, 2022. URL https: hatas://arxxvv org/
abs hagg//bbb
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov. Andrey Bout, Irina Piontkovskaya, Jiansheng Wei, Xin Jiang, Teng Su. Qun Liu, and Jun Yao. Pangu-sigma: Towards trillion parameter language model with sparse heterogeneous computing. 2023. URL battps://aaaree. org/ abs 22033 10845.
Stephen Roller, Sainhayar Sukhhaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021. URL https: /arxi wttp://wwog///////////////// 64426.
21

---

Published as a conference paper at ICLR 2025
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: test suite for identifying exaggerated safety behaviours large language models, 2024.
Keisuke Sakaguchi, Ronan Le Bras. Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial arg/abs/// 906.10
winograd schema challenge scale, 2019. URL https::/arrkat
Victor Sanh, Albert Wehson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnand Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization, 2022.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019. URL https::tt///t. http//rxvvv.gg//eee
19449 09728.
Teven Le Scao, Thomas Wang. Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang. Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong. Julien Launay, and Iz Beltagy. What language model to train if you have one million gpu hours?, 2022.
Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. URL https: //arxiv. org/abs / 1911. 02150.
Noam Shazeer. Glu variants improve transformer, 2020.
Noam Shazeer and Mitchell Stem. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL https: / http://axvvvvaaaa org/abs//////ooooooooo.oo./. 1 804 04235.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017.
Sheng Shen, Le Hou, Yangi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou, Mixture-of-experss meets instruction tuning:a winning combination for large language models, 2023a.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts, 2023b. URL https:://arxiis org/ abs /3003 07226.
Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching Ilama2 performance with 0. Im dollars. 2024.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley. Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion paramcter language models using model parallelism, 2020. URL https: /ata:///arxivii org/ abs 1909. 08053.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wil- son, Marina Machado, Luisa Souza Moura, Dominik Krzemiáski, Hakimeh Fadaci, Trem Frgün, Ifcoma Okoh, Aisha Alaagib. Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder. Surya Guthikonda, Emad A. Alghamdi,, Sebastian Gehrmann, Niklas Muennighoff. Max Bartolo, Julia Kreutzer. Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. 2024.
Snowflake. Snowflake arctic: The best I1m for enterprise
efficiently in- URL https://wwet //www snowflake mnhowlaaee.com/bllg... seriic--ppneeffcienn------ www.---oinnnnnnnangnaaaa moseeemodels--- soaddessssnwwllake..
telligent, truly open, 2024a.
Snowflake.
Snowflake arctic cookbook series: Exploring
mixture of ex-
perts
(mae), 2024b.
URL. https: /tmeai//mdiiii um www.//co//////ooeeee snowflake/ www.oriiiiiioo orinn-nnnnn
snowflake-and
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset, 2023. URL https:// rithub..cmm wal.ennipppp22
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani. Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer. Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: open corpus of three trillion tokens for language model pretraining research, 2024.
22

---

Published as a conference paper at ICLR 2025
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Mcasuring massive multitask language understanding korean, 2024, URL https: arxiv. org/ abs/ 2402 11548.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL hwwps:tpp//// wttpaaawww....gaaaaaaaaaa 220.0/000144444..4... 098644.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-hert: Pre-training of generic visual-linguistie representations. 2020. URL https:/ www.ivivvvv....oa.aaaaaaaaav..a www.iiiv.gg......o..ooo.... org/ abs 190111908..0 08530.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen tau Yih, Jason Weston, and Xian Li. Branch-traing train-mix: Mixing expert Ilms into mixture-of- experts Ilm, 2024, URL https: / arxiv. org/abs 020.//0000..... 07816.
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H. Chi, Denny Zhou, and Jason Wci. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL https: / htt///aaxxaa /arxiv..vvv.///////////////a wwwrziiv.orr///o abs/ aabs/188 ///11/81111// 00937.
Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan. Sparse universal transformer, 2023.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies, 2024. URL https: //arxiv.org /bs/ 13623.
Chameleon Team. Chameleon: Mixed-modee early-fusion foundation models, 2024a.
Gemini Team, Rohan Anil, Sebastian Borgcaud, Yonghui Wu, Jean- Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, al. Gemini: A family of highly capable multimodal models, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober. Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024a.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botey, Alex Castro-Ros, Ambrose Slone, Amélic Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy. Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mellroy, Ruiho Liu, Ryan Mullins, Samuel Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri. Soham De. Ted Klimenko. Tom Hennigan, Vlad Feinberg. Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals. Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Percira, Eli Collins, Armand Joulin, Noah Ficdel, Evan Senter, Alck Andreev, and Kathleen Kenealy, Gemma: Open models based on gemini research and technology, 2024b. URL https: wawpiv.ivvv..///..//...//...avv. www.or/gaaaooooooo abs 22t.//403..822
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, et al. Gemma 2: Improving open language models at a practical size, 2024c. URL https: /ttsss///aaivveeee www.orgg/bba...... abs/ 2408 00118.
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg. Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakowsky, Frez Safahi, Frez Sehwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan
23

---

Published as a conference paper at ICLR 2025
Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Mcirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Ychoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. 1.5 Hybrid transformer-mamba models at scale, 2024d. URL https: / /aa.aariiiiv///e/ http:/aawww...a.aaa//////aaaaa// /2t////ss////... 2408. 12570.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023. URL http::/ /tm///mssii....... com/ com/blog //mptm// //b.
Qwen Team. Livenl.5-moo Matching 7b model performance with 1/3 activated parameters", February 2024b.
URL https: //we/////w//mmmmmm github. itllll//olloo qwen-moe/.
Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong. Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yugi Wang, Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models, 2024e. URL wattts://aasee org./ aa///////////////// 2404/4444444 12387.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen. Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu. Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal. Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas. Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie- Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https:/ / aarivviiiivv//.ia/aaaaaaaaaavvvt org/ ht/////////3300 09288.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexan- der M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2023.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A billion parameter autoregressive language model, 2021. URL https: / /gtthbb.t..bggggggg com/ httm//iinoofollll 2/ /tw/////etteeeeee mresh-trand rans former- jax.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Opendevin: An open platform for ai software developers as generalist agents, 2024a. URL https: / /ttp/aaxvvvo//// org/ abs/ 1404.16741.
Yizhong Wang. Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannanch Hajishirzi. How can camels go? exploring the state of instruction tuning on open resources, 2023.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Nar- simhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training per-prrfornnng reward models, 2024b. URL https:/ arxii. .rgg abs 2406 08673.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot leamers, 2022.
Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng. Biye Li, Weiwei Lü, Peng Cheng. Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. Skywork-moe: A deep dive into training techniques for mixture-of-experts language models, 2024.
24

---

Published as a conference paper at ICLR 2025
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017. URL https://arxiv.org// /1707. 06209,
BigScience Workshop. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvnn, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang. Benoit Sagot, Niklas Muennighoff, et al. Bloom: A 176b-parameter open-access multilingual language model, 2023.
Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. Omni-smola: Boosting generalist multimodal models with soft mixture of low-rank experts, 2024a. URL https: /atxxvaiivvvvv///// org/ abs/ 2312.222222 00968.
Shaohua Wu, Jiangang Luo, Xi Chen, Lingjun Li, Xudong Zhao, Tong Yu, Chao Wang, Yue Wang, Fei Wang, Weixu Qian, Houbo He, 7eru Zhang, Zeyu Sun, Junxiong Mao, and Chong Shen, Yuan 2.0-m32: Mixture of experts with attention router, 2024b. URL httptpps//......g////oo wttpg//b///0..1/////00
XAI. Open release of grok-1, 2024. URL http:///xx..a/bb//////o/oee
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Cheng Xu, Shuhao Guan, Derek Greene, and M- Maharr Kechadi. Benchmark data contamination of large language models: A survey, 2024. URL https:/ / / www.iiv.orgioo /0obs/2406 042 44.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-expeeeee experts language models, 2024.
Aiyuan Yang, Bin Xiao, Bingning Wang. Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv. Da Pan, Dian Wang. Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong. Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie. JunTao Dai, Kun Fang, Lei Su, Liang Song. Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang. Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang. Xiaoxi Chen, Xin Men, Xin Yu, Xuchai Pan, Yanjun Shen, Yiding Wang. Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models, 2023. URL https:: http:s//aaxxvvooo org/ abs/ 2309. 0305.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jiamxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijic Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL https: / /arxivivvvvv www.aas.//....ooooo..o.........a htte//bs///00 2407.10671.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Licret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024b. URL https: / h/taxxiv.oooo howp/oorg//aaoee abs/ 2405. 15793.
Zheng-Xin Yong. Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji. David Ifcoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Indra Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. Bloom+1: Adding language support to bloom for shot prompting, 2023. URL https:/ // aaaiiiiiiiv///a///aaaaaaaaaal.aa erg/ abs/2212 09535.
Longhui Yu, Weisen Jiang, Han Shi. Jincheng Yu, Zhengying Liu, Yu Zhang. James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models, 2024. URL https: http////aaxva///aaaaaee abs 2aabbs/230 2200//39100 12264.
Longfei Yun. Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang. Toward inference-optimall mixture-of- expert large language models, 2024. URI. https: : ///////axi/// org/abs htgg/bbs/2444 02852.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning, 2023. URL https: htt//aaxiii wrgoeeraaaa.. aat.//3/99.. 05444.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi, Hellaswag: Can machine really finish your sentence?, 2019. URL https: www.riivv.o/.aaaooo 005.07830.
25

---

Published as a conference paper at ICLR 2025
Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https://arxiv, abs/19 1990/191.... 07467.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng. Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng. Wei Pang. Xinrun Du, Yiming Liang. Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quchry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang. Yuclin Bai, Yuhan Zhang, Yuxiang Zhang. Zenith Wang. Zhenzhu Yang. Zijian Zhao, Jiajun Zhang, Wanli Ouyang. Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series, 2024a. URL https: //t/aaxxvvv///////////a/////aav. org/ abs /2405. 19327.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024b. URL http:t:s:: / /tt////xivvv//// www.orrgaass..... abs/24011 02385.
Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sehastian Ruder, Ahmet Ustun, and Acyr Locatelli, Bam! just like that: Simple and efficient parameter upcycling for mixture of experts, 2024c. URL https::///// /tt////axvvvv// org/ abs/ 2ass/248 2408. 08274.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab. Xian Li. Xi Victoria Lin, Todor Mihaylov. Myle Oit, Sam Shleifer. Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zeulemoyer. Opt: Open pre-trained transformer language models, 2022. URL https: / www.wiiww./aaaaaaa.e.a... /20.s///225552 01068.
Yanli Zhao, Andrew Gu. Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott. Sam Shleifer, Alban Desmaison. Can Balioglu, Pritam Damania, Bernard Nguyen. Geeta Chauhan. Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreeer:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture-of-expene for autoregressive language model pre-training, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu. Srini Iyer. Jiao Sun. Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followee evaluation for large language models, 2023b. URL https:://///: 2311 0111009911 1.
abs/
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-exppe expexss with expert choice routing, 2022.
Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng. Chang Lan, Da Huang, Siamak Shakeri, David So, Andrew Dai, Yifeng Lu, Zhifeng Chen, Quoe Le, Claire Cui, James Laudon, and Jeff Dean. Brainformers: Trading simplicity for efficiency, 2024. URL htt.tttt/////tt /ttpwaaiivvvvv////a///aa or r abs/ abs/2306. 00008.
Terry Yue Zhuo, Armel Zebaze, Nitchakam Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameten-efficient instruction tuning code large language models, 2024.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moc: Designing stable and transferable sparse expert models, 2022.
Simiao Zuo, Xiandong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts, 2022. URI. https:t//tssti...t//e/ crg//bb/////// abs/ 2110. 04260.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei- Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzich Fadace, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model, 2024.
26

---

Published as a conference paper at ICLR 2025
Introduction
2
Pretraining and adaptation
3 Results
Experimenting with alternative design choices
4.1 Mixture-of-Expert vs. dense
4.2 Expert granularity
4.3
Expert choice vs. token choice
MoE analysis
5.1
Router saturation
5.2
Expert co-activation
5.3
Domain specialization
5.4
Vocabulary specialization
6
Conclusion
10
A Related work
28
B Additional experiments on alternative design choices
29
B.1
MoE-specific pretraining settings
29
B.1.1
Shared experts
29
B.1.2
Sparse upcycling
B.1.3
Load balancing loss
30
B.1.4
Router z-loss
31
B.2
General
pretraining settings
32
B.2.1
Dataset experiments
32
B.2.2
Initialization
33
B.2.3
RMSNorm
................................
33
B.2.4
Decaying embedding parameters
34
B.2.5
QK-Norm
34
B.2.6
AdamW epsilon
35
B.3
Adaptation settings
36
C Training configuration
37
Evaluation setup
E Openness of models
41
Additional evaluation
43
Other experiments
47
H Analysis
49
H.1 Details of analysis in $5
49
27

---

Published as a conference paper at ICLR 2025
H.2 Additional Analysis
51
I Artifacts
58
J Selecting the number of total and active parameters
58
K Limitations and future work
59
L OLMOE-1B-7B-011
60
A RELATED WORK
Advances in MoEs Current LMs still largely follow the transformer architecture (Vaswani et al., 2023) with only few architectural changes that have been widely adopted, such as decoder-only training (Radford et al., 2019), SwiGLU activations (Shazeer, 2020; Dauphin et al., 2017), RoPE (Su et al., 2023), MQA/GQA (Shazeer, 2019; Ainslie et al., 2023) and RMSNorm (Zhang & Sennrich, 2019). Model sparsity via ofxture-of-Experree is modification still under active exploration with some early adoption but most LMs, including Llama 3 (Dubey et al., 2024), still rely on a dense architecture. There has been lot of progress in improving the sparsely-gated MoE layer since its introduction (Shazeer et al., 2017): New routing techniques (Lewis et al., 2021; Roller et al., 2021; Zuo et al., 2022; Gross et al., 2017; Jaszezur al., 2021; Dua al., 2021: Zhong al., 2024; Wu et al., 2024b: Muqeeth et al., 2024), fine-grained expert segmentation (Dai et al., 2024; He, 2024), stability (Zoph et al., 2022) and efficiency (Lepikhin et al., 2020; Rajbhandari et al., 2022: Du et al., 2022; Zhou et al., 2024; Li et al., 2022; Sukhbaatar et al., 2024; Pan et al., 2024: Ren et al., 2023) improvements. In this work, we perform many experiments to provide insights into training Mixture-of- -fpperrsss LMs. Subsequently. we train OLMoE- B-7B for ST tokens. No prior MoE has been overtrained (Gadre et al., 2024) to this extent to our knowledge making OLMOE-1B, 1--7-BB-BB the best testbed to research performance saturation of MoEs dense models. With OLMoE we hope to facilitate such and other research to help the field uncover whether MoEs should make into all future LMs and with what precise configuration.
Open LMs A variety of model families have been proposed under varying degrees of openness commonly categorized based on whether model weights are available. Closed-weigh models include GPT (Brown al., 2020; OpenAl et al., 2023), Gemini (Team et al., 2023; 2024a), PaLM (Chowdhery et al., 2022; Anil et al., 2023). Reka (Team et al., 2024e), and open-weight ones include Llama (Tou- vron et al., 2023a;b; Dubey et al., 2024), Mistral (Jiang et al., 2023; 2024), Gemma (Team al., 2024b;c), Falcon (Almazrouei et al., 2023; Penedo et al., 2023), MPT (Team, 2023), Qwen (Bai et al., 2023a; Yang et al., 2024a), GLM (GLM et al., 2024), Yi (AI et al., 2024), DeepSeck (DeepSeek-A) et al., 2024a;b; Dai et al., 2024), Nemotron (Parmar et al., 2024; Nvidia et al., 2024; Wang et al., 2024b), InternLM (Cai et al., 2024), Baichuan (Yang et al., 2023), Phi (Gunasekar et al., 2023; Li et al., 2023c; Abdin et al., 2024), StableLM (Bellagente al., 2024), OPT (Zhang et al., 2022), Zamba (Glorioso et al., 2024). However, besides model weights, training data and code are key to enabling scientific research of these models (Longpre et al., 2023b; 2024) and distributing their benefits broadly (Bommasani et al., 2023). There have been few releases also including data and code in addition model weights which we refer to as "fully open-source". BLOOM (Workshop et al.. 2023; Scao et al., 2022; Muennighoff et al., 2023c; Yong et al., 2023). GPT-NeoX (Black et al., 2022; 2021: Wang Komatsuzaki, 2021), StarCoder (Li al., 2023a: Lozhkov al., 2024; Allal et al., 2023; Muennighoff et al., 2023a; Zhuo et al., 2024), Pythia (Biderman et al., 2023). OLMo (Groeneveld et al., 2024), LLM360 (Liu et al., 2023). Cerebras-GPT (Dey et al., 2023), DCLM (Li et al., 2024a), MAP-Neo (Zhang et al., 2024a). RWKV (Peng et al., 2023; 2024), and SmolLM (Allal et al., 2024). For Mfxture-of-Experre of-Experts only OpenMoE (Xue et al., 2024) aims to be fully open-source, however, its poor performance limits its usefulness. We release OLMoE-1B- as the first ofate-of-the-ar art Mixture-of. -ixture-of-Expertts LM that is fully open- source: model weights, data, code, and logs.
28

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS ON ALTERNATIVE DESIGN CHOICES
In this section, we present an extension of pretraining and adaptation experiments that have led to OLMoE- 1--0-----BB (also see $4). We group them into experiments settings specific to Mixture- of-Experts ($B.1), experiments on settings applicable to both dense LMs and MoEs ($B.2), and adaptation experiments ($B.3). In pretraining experiments, MMLU (Hendrycks et al., 2021a) with varying few-shots and a different format that provides signal earlier during training. We describe our full evaluation setup in Appendix D and provide additional experiments in Appendix G. Each experiment links to Weights & Biases report with more validation and downstream results, and the full configurations of the runs. To isolate the impact of changes and minimize confounders, we vary only one hyperparameter for each experiment. Nevertheless, due to the large number of hyperparameters. some results may change under different configurations and we cannot guarantee the correctness of each of our hyperparameter choices. Models are not comparable across different experiments, as we vary the base model to incorporate successful findings.
often use MMLU Var, version of
B.
MOE-SPECIFIC PRETRAINING SETTINGS
B.1.1
SHARED EXPERTS
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
3.0
3.50
60
35
Performance
2.8
3.25
experts
2.6
3.00
40
32 routed
30
31 routed, shared
2.4
2.75
10
40 70
100130
10
40 70
100 130 Tokens (B)
10 40 70 100 130
10 40 70 100 130
Figure B1: Shared experts. Both setups have the same number of active and total param- eters and use the same number of FLOPs. 4 of the 32 routed experts activated, while is 3 for the routed experts of the other model, as it has always-active shared expert. More results, logs, and configurations: https: awt.nn/..aaaiiimmmia.aa 1tte-11mm/mmmmmeepprreeeee Plot-Expert- wahaariiggmmmmlloo000111111
Dai et al. (2024) propose training with a shared/fixed expert that is always used in addition to the routed experts. The intuition to encourage the shared expert to learn common information and allow the other routed experts to learn more specialized knowledge. This should reduce redundancy among experts and thus lead to a better model as it can store more total information.
In Figure B1, we benchmark having a single shared and single routed expert versus two routed experts. While both settings lead to similar performance, sharing an expert performs slightly worse. Sharing expert removes flexibility from the model and thus goes against the findings in $4.2 suggesting that allowing for more expert combinations improves performance. Specifically, the two models in Figure B1 have 4 35, 960 and (31) 4, 495 possible combinations per layer. Thus, removing one of the routed experts and turning it into shared one climinates almost 90% of possible combinations. This likely acts as a counterforce the potential benefits of isolating common knowl- edge a shared expert. Based on these results, we do not use shared experts in OLMoE-1B-7B but we do think that there is merit to the idea of experts that are activated more often or even always. However, rather than enforcing this behavior via shared expert, we believe that it should be learned by the model. This is difficult with current setups due to the necessity of a load balancing loss ($B.1.3) penalizing the model tokens are not distributed equally among experts. Potential future work can explore removing the load balancing loss to allow for more flexible usage of experts.
3
B.1.2 SPARSE UPCYCLING
Komatsuzaki et al. (2023) propose turning a dense model into a Mixture-of of of-Exxerts model via sparse upeycling: (1) The dense MI.P is cloned for each desired expert constitute MoF layers. (2) A newly
29

---

Published as a conference paper at ICLR 2025
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
35
Performance
10.0
4
60
7.5
30
5.0
3
40
Scratch
Upcycle
2.5
25
250
450
650
50
250
450
650
50
250
450
650
50
250
450
650
Tokens (B)
Figure B2: Sparse upcycling. We upcycle OLMo-1B (0724) at 2T tokens into an MoE with total experts of which are activated and train it for an additional 610 billion tokens. We compare it to model trained from scratch for 610 billion tokens. Except for this differ- ence, both models use the same config, which includes some suboptimal settings that con- tribute the instability, such as no QK-Norm ($B.2.5) and truncated normal init ($B.2.2). More results, logs, and configurations: hotps: /ettps::/wwadee http///ad.../..c////./ aitaaaaaaaaaaa owwoooomoo//////oooooo reportssp Plot Soott-Srracc-aae Swwssssppple----- -ml 1dz 4ND yOTo
initialized router is added in front of each MoE layer. (3) Pretraining continues with the new model so that the cloned MLPs can gradually specialize in different things and the router can be learned. They find that the upcycling approach maintains a performance advantage over language model trained from scratch for up 120% of the compute budget of the original dense checkpoint that the sparse model was upcycled from. For example, if sparsely upcycling 1.3B parameter model at 2 trillion tokens then only at 2.4 trillion tokens should an MoË trained from scratch catch up with the upcycled model. That is, the sparsely upcycled model would have been trained for another 400 billion tokens, thereby saving the equivalent of up to 2T tokens of compute. Other works such as MiniCPM (Hu et al., 2024), Qwen2 (Yang et al., 2024a) and reportedly Mixtral (Cai. 2023; Jiang al., 2024) have adopted sparse upcycling but only share limited information about their configuration.
In Figure B2, compare sparse upcyeling OLMo-1B (0724) (Groeneveld et al., 2024) with training an MoE from scratch. We find that after 500B tokens, an otherwise equivalent MoE trained from scratch already catches up with the upcycled model both on the metrics in Fig- ure B2 and our additional metrics at https: /tttts::/wandaa /wnddw ai ai/ai2- 1mlm/o o olmoe/reports Soot-seratch--ssUpccyyeeo www.eleeeeellleooo01ooo 4 At around 600B tokens, the MoE from scratch starts outperforming the upcycled MoE. Thus. it only requires 25% of the compute budget of the original dense model to catch up as opposed to the 120% reported in Komatsuzaki et al. (2023). However, they use expert choice routing and study encoder-decoder models (Raffel et al., 2023). Meanwhile, we use token choice routing ($4.3) and decoder-only models ($2). Further, upcycle model that has already been significantly overtrained (Gadre et al., 2024), i.c., IB model trained for 2T tokens. Its parameters are likely already in very optimal range for dense model, which may limit the amount of additional exploration possible after upcyeling. This motivates us to experiment with adding noise to the upcycled weights outlined in Appendix G, but we do not find lead to better performance. large disadvantage of upcycling that the upcycled MoE is constrained by some hyperparameters of the dense model. Specifically, OLMo- IB (0724) was trained without QK-Norm and normal initialization, both of which hurt stability in our experiments ($B.2.5, $B.2.2). While it may be possible to simply add new QK-Norms and train them from scratch similar the new router layer trained from scratch, it is impossible to change the initialization of the original dense model when upcycling it. Thus, as we want to change these hyperparameters and also train OLMoE- 1B-BB for around 250% of the compute budget of the dense model (5T vs. 2T tokens), we do not use upcycling.
1.1. LOAD BALANCING LOSS
Shazeer al. (2017) propose the load balancing loss to penalize the model if it is unbalanced, i.e., if it routes all tokens to only few experts. This is based on the observation that without such penalty, models tend to update only select few experts in each layer (Eigen al., 2014; Bengio et al., 2016). To compute the load balancing loss (C,R) we multiply the fraction of tokens f. routed to one expert E, with the total routing probability P, allocated to F, for one batch and sum it across the number of
30

---

Published as a conference paper at ICLR 2025
Training loss
Load balancing loss
Validation loss (C4)
Validation loss (Pile)
0.4
LBL
Performance
4.5
4.5
4.5
No LBL
4.0
4.0
0.2
4.0
3.5
3.5
0.1
10
10
10
1
10
Tokens (B)
Figure B3: Impact of applying a load balancing loss (LBL). The training loss plot excludes the load balancing loss for both models. More results, logs, and configurations: https: /awdd..
ai aal/ai2-11 10a12211mmal htt////c///eee htt.//////////////eeeooo/o -BL-ott-BB----- - LBL- Vm 16z1100kk4k
% of tokens in batch
load balancing
Load balancing
assigned to expert
100
Expert 0
Expert
Expert
Expert
Expert Expert
50
Expert
Expert
1
5 Tokens (B)
10 i
5 Tokens (B)
10
Figure B4: Expert assignment during training when using or not using load balancing loss for the first MoE layer. More results, logs, and configurations: https: /wwddba............. ai/ aaii/ai2-D00 1m1mm olmoe wttp:teportssseee LBL- Vs- -o---------- No---oo-- Poo-ILL-- Vml Vml1dzo40TKyNDgg
experts Ng:
Ng LiB = N > f, P.
The loss is further scaled by NE and loss weight a (see Equation 2), which is an optional weight to determine the magnitude of the loss commonly set to 0.01 (Zoph et al., 2022; Xue et al., 2024). do not experiment with changing the weight of 0.01.
In Figure B3 we investigate the performance impact of using the auxiliary load balancing loss. We find that across training loss and validation losses, using the load balancing loss leads better performance even after only a few billion tokens. We still measure the load balancing loss even when it is not used ("No LBL") and find that while it spikes initially, it slowly decreases the next few billion tokens. This behavior is also visible in Figure B4 (left), where initially all tokens in the first layer are assigned the 6th expert (pink). Eventually, the model also starts assigning some tokens to the 1st expert (yellow). However, all other experts remain largely flat and are thus "dead weights" that take up GPU memory but are not used. Given these results, we use the auxiliary load balancing loss with a weight of 0.01 following prior work (Shazeer et al., 2017; Shen et al., 2024). However, getting rid of the load balancing loss is an important direction for future research as it constrains the flexibility of the model by forcing it to use all experts approximately equally. This could prevent the experts from specializing in certain data domains and may be reason prior work has failed find strong evidence of expert specialization (Jiang et al., 2024: Zoph et al., 2022).
B.1.4 ROUTER Z-LOSS
Zoph et al. (2022) propose the router z-loss to improve both the stability and quality of MoE models. This auxiliary loss penalizes large logits coming into the gating network. Such large logits can lead numeric overflows in the large matrix multiplications happening in the MoF layer. It is computed
31

---

Published as a conference paper at ICLR 2025
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
4.0
4.0
35
60
Performanc:
3.5
3.5
3.0
30
Z-loss
3.0
2.5
No
z-loss
25
2.5
250
500
750
10
250
500
750
10
250
500
750
10
250
500
750
Tokens (B)
Figure B5: Router z-loss. We compare adding router z-loss with a loss weight of 0.001 versus no additional z-loss. More results, logs, and configurations: htttp:://wand..... aiabbb-1aaaimmm wmmoe/eeports///eeee Zloss vs none- Vm Vml1dzo4NDMNNJU
by exponentiating the logits zj right before the router layer summed across the number of experts NE and averaged across the batch B, thereby making larger logits lead to larger loss:
N
Cxz(x)
log > expla
(1)
The loss is further multiplied with an optional loss weight, 3 (see Equation 2), to determine the magnitude of the loss commonly set 0.001 (Zoph al., 2022; Shen et al., 2024). We do experiment with changing the weight of 0.001.
In Figure B5, we confirm that formance adding the router z-loss improves stability (less spikes) and quality (lower loss and higher downstream performance). Thus, despite it reducing throughput by ~2% we use the router z-loss for OLMoE- -B-77 with weight of 0.001 as in Zoph et al. (2022).
training loss, validation loss, and downstream per-
B.2
GENERAL PRETRAINING SETTINGS
B.2.1
DATASET EXPERIMENTS
Training loss
HellaSwag
MMLU Var
ARC-Challenge
4.0
Performance
0.5
3.5
30
0.4
3.0
OLMoE-Mix
2.5
0.3
Dolma 1.7
20
10
40 70 100 130
10
40 70 100 130
10 40 70 100
130
10 40 70 100 130
Tokens (B)
Figure B6: OLMOE-MIX Dolma 1.7. We compare data mix described in $2 with Dolma 1.7 used to train prior OLMo models. Lower training loss does not mean that dataset is better, but rather suggests which dataset is easier for the model to leam. More results, logs, and configurations:
www.lm/mmooeeeeeeeeeeeeeeeeeeeee -lot-Dolma- 1 1-7 vs Dolma OLMoE -Wl1112440TIITTT
aolmeerreports.
Li et al. (2024a) release the els than Dolma 1.7 and other datasets as measured on common benchmarks like MMLU (Hendrycks al., 20) This motivates us mix their DCLM dataset with some components from Dolma 1.7 that we deem to be high-quality: see $2. In Figure B6, we compare our mix, OLMOE-MIX. with Dolma 1.7 in controlled setup. We find that OLMoE-MIx leads clear gains on all three down- stream metrics, especially MMLU, DCLM- M-Maseiline has been created through a series of dataset abla- tions targeting MMLU and other downstream metrics, which explains these results. We also compare
dataset and establish that it leads to better language mod-
32

---

Published as a conference paper at ICLR 2025
adding Reddit and FLAN to mix detailed in Appendix G, but do not find consistent performance gains. We do not have a strong intuition for why adding these datasets does not help and more auto- matic approach dataset mixing may be desirable for future iterations (Liu et al., 2024a; Albalak et al., 2024).
We pretrain using our mix of DCLM-Baseline and Dolma 1.7 dubbed OLMoE-MIX
B.2.2
INITIALIZATION
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
8.0
60
Performan Performance
6.0
3.5
30
4.0
3.0
40
Trunc normal
Normal
25
2.5
200
400
600
10
200
400
600
10
200
400
600
10
200
400
Tokens (B)
Figure B7: Initialization. We compare a normal initialization with a standard deviation (std) of 0.02 with a truncated normal initialization with a maximum (minimum) cut-off of 0.06 (-0.06) corresponding to three stds (3x0.02). More results, logs, and configurations: https:/ /wandb. ai/ai2- 11/2011mmooll wwt.l//mmmmmaaameeeemmceecceeeee httpm//eeeeee////eeeeeeeeese/e/e
Init--Wiiin 4d224NDIZM255
Few prior works Mixture-of-Experte share their initialization strategy. Even the most open MoEs prior to this work, JetMoE (Shen et al., 2024) and OpenMoE (Xue et al., 2024), do not mention their initialization scheme. For DeepScekMoE (Dai et al., 2024) and DeepSeekV2 (DeepSeek-A) et al., 2024b), the authors share that they use normal initialization with standard deviation (std) of 0.006. For dense language models, normal initialization with an std of 0.02 has been commonly used as popularized by Shoeybi et al. (2020).
In Figure B7, find a truncated normal initialization leads more stable training and bet- ter performance than regular normal initialization. The difference between the two initializa- tions only becomes clear at around 450 billion tokens, where the model with the normal initial- ization starts to diverge. This is despite both models using the same configuration except for the difference in weight initialization. Having to train for hundreds of billions of tokens un- til an experiment provides clear signal is one of the key challenges of pretraining ablations. We use the truncated normal initialization for OLMOE-1B-7B.
B.2.3 RMSNORM
Total gradient norm
3
OLMo (Groeneveld al., 2024) uses non- non-parametric layer nor- malization (Ba et al., 2016), mainly as it is significantly faster
RMS
2-
Non-
than the commonly used RMSNorm (Zhang Sennrich, 2019; Mehta al., 2024). This is an unusual choice most LMs use RMSNorm, such as the Llama (Touvron et al., 2023a;b; Dubey al., 2024), Gemma (Team et al., 2024b;c), and Qwen (Bai et al., 2023a; Yang et al., 2024a) model families.
para- metric
1-
20
60 Tokens (B)
100
In Figure B8, observe that replacing the non-parametric layer normalization in OLMo with a parametric RMSNorm leads better performance. This is likely because the non- parametric layer normalization leads to a large number of spikes in the gradients as seen in Figure B10. We clip gra- dients at 1.0, which prevents these spikes from leading to very large and potentially disruptive parameter updates. However, the clipped gradients may still harm the performance of the model as they are no longer the true gradients. Thus, de- spite RMSNorm lowering our training throughput by 15%,
Figure B10: Total norm of the gra- dients when training with RMS or non non-maraeetric normalization We increase the logging interval of the RMS run at 75B tokens, hence its change thickness.
train final model with RMSNorm. We include the RMSNorm parameters in weight decay
33

---

Published as a conference paper at ICLR 2025
Training loss
Validation loss (C4)
HellaSwag
MMLU
3.4
60
2.8
30
Performance
3.2
2.6
RMS
2.4
3.0
Non-
28
2.8
para- metric
2.2
26
40
60
80100
20
40 60
80
20 60
20 40 60 80100 80
Tokens (B)
Non-parametric layer normalization vs. configurations: - -wll1doo4NDDyMTA
Figure B8:
RMSNorm.
More results,
logs,
and
https://wandb. at a htt.://///mmmmm/errs//////
Training Validation loss (C4)
HellaSwag (Acc %)
MMLU Var (Acc %)
3.0
Decay
Performance
2.8
2.9
No decay
30
2.6
2.8
29
2.4
45
1
10
20
30
10
20
30
10
20
30
10
20
30
Tokens (B)
Figure B9: Decaying the RMSNorm parameters. More results, logs, and configurations: https: Maaeay-----mm11oooo111101
//wandb... ai httm///mmm//eeerrt.eeaa//
as we find that it performs slightly better (Figure B9) even though it is common practice to exclude them."
B.2.4 DECAYING EMBEDDING PARAMETERS
Similar the RMSNorm parameters (SB.2.3), embedding parameters are commonly excluded from weight decay.² In Figure B11 we find that whether or not they are decayed has only minor impact on performance, with decaying being slightly better. Thus for simplicity, weight decay all parameters in L.MMOE-1B-77 including embedding and R.MNNrm.
B.2.5 QK-NORM
Some works have reported stability improvements from adding layer normalization after the query and key projections ("QK-Norm") (Team, 2024a; Mehta et al., 2024; Dehghani et al., 2023). QK-Norm can prevent the subsequent attention operation from leading very large logits that may lead to numeric overflows and destabilize the network, especially when training in low precision. Like layer normalization at other places in the model, the QK-Norm could be parameeeiii or use the parametric RMSNorm ($B.2.3).
In Figure B12, we compare using QK-Norm with normalization after the query and key pro- jections. We find that QK-Norm leads to some stability and performance improvements. We perform this experiment with non-parametric layer normalization as used in OLMo (Groeneveld et al., 2024), while we used parametric RMS layer normalization (Zhang & Sennrich, 2019) for OLMOE-1B. ($B.2.3). To ensure the benefit QK-Norm is not an artifact of comparing with non-parametric layer normalization, we run another experiment with RMS layer normalization and
667311625
'https://e http:////...ttb....c//aaaattttt
2441 ssuecomment- 1attmmeeett--n111111 67931 6025
34

---

Published as a conference paper at ICLR 2025
Training loss
Validation loss (C4)
HellaSwag
MMLU
3.0
3,4
Performance
50
29
2.8
3.2
2.6
3.0
Decay
No
2.4
2.8
30
decay
20
40
60
20
40
60
20
Tokens (B)
Figure B11: Decaying the embedding parameters. More results, logs, and configurations: https:
/ appi wwndb. www.aaans/aalllllmmmm/aaa/a 21m22222p
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
60
32
Performance
3.5
2.8
30
3.0
40
28
2.5
2.7
QK-Narm
No QK-Norm
25
50
150
250
350
50
150
250
350
50
150
250
350
50
150 250 350
Tokens (B)
Figure B12: Query-Key layer normalization (QK-Norm). Both models use non-parametric layer normalization. QK-Norm corresponds to additional layer normalization of the query and key projections. More results, logs, and configurations: https: /wwddb/a.....///////////...... ai ai/aaiaaaaaaa 11mm lmoe/ reports/Plots QKNorm-ve vs none Vml Vml1dzo4NDIIM22
still find QK-Norm to lead to slightly better training loss and to prevent a large grad norm spike. Thus,
QK-Norm for OLMoE-IB-7B despite reducing throughput by almost 10%.
B.2.6 ADAMW EPSILON
Training
Validation loss (C4)
HellaSwag
MMLU Var
3.4
30
Performance
3.2
28
1E-08
1E-05
2.8
2.
30
26
10
20
30
10
20
30
10
20
30
10
20
30
Tokens (B)
Figure B13: AdamW epsilon. More results, logs, and configurations: https://wand:: ai2-lim/ wwwoooooo////////ooooooooooo http:///epottt////aatdddll---e vidldooeesogooldddddodddoooeooo
Groeneveld et al. (2024) use an epsilon ("eps") value 1E-05 the AdamW optimizer for training OLMo. A larger eps value leads smaller steps of the optimizer but be stable (Kingma & Ba, 2017).
In Figure B13, find that decreasing eps to the recommended default of 1E-08 (Kingma & Ba, 2017) significantly improves performance while the run remains stable. Thus, we set eps 1E-08 for our final run.
Vm1 11704N11c2N11
35

---

Published as a conference paper at ICLR 2025
B.3 ADAPTATION SETTINGS
Table B1: Adaptation experiments of OLMOOE-B--7B We compare using the pretrained check- point prior to annealing for adaptation, using the checkpoint after the additional 100B tokens of annealing, and using the checkpoint after the additional 100B tokens of annealing and with load balancing loss ($B.1.3) during adaptation. We apply DPO/KTO to the respective SFT model.
Human-
Alpaca-
Task (-)
MMLU
GSM8K
BBH
Eval
Eval 1.0 XSTest
IFEval
Avg O-shot
Setup (-)
O-shot
8-shot CoT
O-shot
0-shor
0-shot
O-shot
0-shot
Metric (-)
EM
EM
EM
Pa 10
Swin
Loose Ace
OLMoE-IB-7B
49.0
2.0
31.5
18.9
62.1
18.5
w/o annealing
+SFT
50.2
43.0
35.6
55.5
68.9
83.8
39.7
53.8
+DPO
50.9
36.0
35.8
58.8
81.7
83.2
47.9
56.3
OLMOE-1B-7B
49.8
3.0
33.6
22.4
59.7
16.6
+SFT
51.4
40.5
38.0
51.6
69.2
84.1
43.3
54.0
+DPO
51.9
45.5
37.0
54,8
84.0
82.6
48.1
57.7
+TTO
51.2
45.5
34.1
57.1
81.6
86.6
47.5
57.7
+SFT
50.9
36.5
35.7
52.4
66.9
84.8
42.3
52.8
(load balancing)
+DPO
51.1
42.5
39.3
55.6
82.9
82.1
46.0
57.1
(load balancing)
We experiment with small design choices for adaptation using our evaluation setup described in Appendix D. (1) Auxiliary losses: Zoph et al. (2022) find that using the auxiliary load balanc- ing loss 1.3. during regular finetuning leads to small performance gains. For instruction tun- ing, however, Shen et al. (2023a) do not find conclusive evidence in favor of using the load balancing router z-loss with only small dif- ferences in performance, both in support of and against the auxiliary losses. In Table B1 we dis- play experiments with the load balancing loss during adaptation and find that not using it leads to better performance (54.0 vs. 52.8 after in- struction tuning (SFT) and 57.7 vs. 57.1 after preference tuning (DPO)). One potential problem of deactivating the load balancing loss is that it may harm balance among experts and turn some into dead weights as observed during pretraining in 1.3.. However, when measuring the load balancing loss in Table B2 on our SFT data ($2), we find that the loss only increases by around 0.01% % after SFT (8.250 vs. 8.249). This likely because which experts certain tokens get routed to is determined early during pretraining, as we find later in the analysis section 1). We also visualize the activation patterns of experts of the model after pretraining, and the models after SFT and DPO trained without load balancing in $H.2 (Figure H6) finding that the dis- tribution remains around the same. Thus, as our models adapted without load balancing perform better and we find it not to impact routing substantially, we do not use load balancing during adaptation (2) Annealing checkpoint: We also experiment with using the checkpoint pre-annealing ($2) for adaptation and find the checkpoint post- post-annealing leads to better performance (53.8 vs. 54.0 af- ter SFT and 56.3 vs 57.7 after DPO), thus we use the post-annealing checkpoint, (3) Preference algorithm: Since the release of DPO (Direct Preference Optimization) (Rafailov et al., 2023), variety of preference algorithms have been proposed (Ethayarajh et al., 2024; Hong et al., 2024; Meng et al., 2024). We experiment with KTO (Ethayarajh et al., 2024) and find that it matches DPO in Table B1 for our setup (Appendix C). While release both models,
Table B2: Load balancing loss (Equation 3) over a subset of the respective corpora prior to scal- ing with the load balancing loss weight a. While we use load balancing loss during pretraining, we do not it during SFT.
ALMOE-1B-7B
Data (1)
After pretraining After SFT
Wikipedia C4
8.331
8.367
8.073
8,076
SFT data
8.249
8.250
DPO for final
36

---

Published as a conference paper at ICLR 2025
OLMOE-IB-7B-INTTKE model, as it scores higher on AlpacaEval, which has smaller chance of data contamination than our other benchmarks (Xu et al., 2024).
C TRAINING CONFIGURATION
Dense LMs (OLMo, Llama...)
OLMoE
Output
Output
Feedforward Network (FFN)
E,
E2
E
E
Es
E
Eg3
Eg4
MoE Module
Router
Norm
Norm
Multi-head Attention
Multi-head Attention
Norm
Norm
N₂
N2 x
Input
Input
Figure C1: Comparison of the architecture of dense LMs and MoE models like OLMoE. The figure excludes some details, e.g., OLMoE- 1B-7B also uses QK-Norm (SB.2.5).
Table Cl: Composition of the pretraining data for OLMOE-IB-7B. StarCoder (Li et al., 2023a; Kocetkov al., 2022), peS2o (Soldaini & Lo, 2023), and Wiki come from Dolma 1.7 (Soldaini et al., 2024). arXiv from Red-Pajama (Computer, 2023), SpenWeeaMtth (Paster et al., 2023) and Algebraic Stack from ProofPile (Azerbayev al., 2023). Links to our data are in Appendix
GPT-NeoX tokens (billions)
UTF-8
Words (billions)
Documents (millions)
Source
Doe Type
bytes (GB)
-eselle
web pages code
3,860
3,380
16,700
2,950
StarCoder
101
63.9
325
78.7
peS2o
STEM papers STEM papers math web pages math proofs code
57.2
51.3
268
38.8
arXiv
21.1
23.5
88.8
1.55
Math
12.7
10.2
42,4
2.91
Algebraic Stack
12.6
9.6
39.3
2.83
English Wikipedia
encyclopedic
3.69
3.16
16.2
6.17
& Wikibooks
Total
4,060
3,530
17,400
3,080
Pretraining We display the pretraining hyperparameter configuration of OLMMOE-IB-7B in Appendix C comparing with other relevant models. We follow Groeneveld et al. (2024) using the AdamW optimizer (Loshchilov & Hutter, 2019) with ZeRO (Rajbhandari al., 2020) via PyTorch FSDP (Zhao et al., 2023) and mixed-precision training (Micikevicius et al., 2018). Our main model settings differing from Groeneveld al. (2024) are: (1) MoE-related changes: OLMoE-IB-7B a sparsely activated decoder-only transformer (Vaswani et al., 2023) using dropless Mixture-of- Experts (Gale et al., 2022). Unlike most prior MoEs,
high granularity (Dai et al., 2024;
37

---

Published as a conference paper at ICLR 2025
Table C2: Adaptation training data for OLMOE-IB-7B. We mix Tulu 2 (Ivison et al., 2023), No Robots (Rajani et al., 2023), CodeFeedback (Zheng et al., 2024), MetaMathQA (Yu et al., 2024) and Daring Anteater (Wang et al., 2024b) for SFT and use a filtered Ultraeeddbcc (Cui et al., 2023; Lin et al., 2022) for preference tuning. Links to our data are in Appendix I.
Source
Domain
Samples
Instruction Tuning
Tulu 2 SFT Mix
Various
326,154
No Robots
Various
9,500
sndefeedbaat-Fiileeed--nsnneee MetaMathQA
Coding
156,526
Math
98,750
Advanced (non-chat) subset of Daring Anteater
Various
17,082
Preference Tuning (DPO (Rafailov el al., 2023))
UltraFeedback binarized and filtered for TruthfulQA contamination
Various
60,800
Krajewski et al., 2024) with 64 small experts with an FFN dimension of just 1,024 rather than a few large experts. We further two auxiliary losses: router z-loss (Zoph al., 2022) and load balancing loss (Shazeer et al., 2017). (2) Stability improvements: (a) We use a truncated normal initialization with standard deviation of 0.02 and minimum (maximum) cut-off of -0.06 (0.06) corresponding to three standard deviations. (b) We use QK normalization (Team, 2024a; Mehta et al., 2024; Dehghani et al., 2023). (c) We use RMSNorm (Zhang& & Sennrich, 2019) instead of the non- parametric LayerNorm used in Groeneveld et al. (2024). (3) Performance improvements: Besides some of the stability improvements which also impact performance, we also reduce the AdamW epsilon to 1.0E-08 from the 1.0E-05 used in Groeneveld et al. (2024) to speed up convergence. Finally, we train OLMOE-IB-7B for significantly longer than all prior OLMo models amounting to 5T tokens and thus more than epoch (1.3) following Muennighoff et al. (2023b). We shuffle the pretraining dataset before starting the second epoch. To all data sources (Table CI), we apply a filter that removes all documents with sequence of 32 or more repeated n-grams, where n-gram is any span of to 13 tokens. For the StarCoder subset, we also remove any document from a repository with fewer than 2 stars on GitHub, whose most frequent word constitutes over 30% of the document, or whose top-2 most frequent words constitute over 50% of the document. We shuffle all samples randomly the beginning of each epoch and train for a total of 5. 133T tokens. During our annealing phase (final 100B tokens), we reshuffle the entire dataset and then linearly decay the learning rate from 5.0E-04 to 0, following prior work (Groeneveld et al., 2024: Li al., 2024a).
Adaptation For finetuning we use Open Instruct (Wang et al., 2023; Ivison et al., 2023).? We filter all SFT samples length of fewer than 4096 tokens match the sequence length of the model. Following Muennighoff et al. (2024), we aggregate loss at the token level during SFT to improve performance on long generative tasks, such as AlpacaEval. We finetune in BF16 with global batch size of 128 (4 H100 nodes with 8 GPUs each, per device batch size of 2, and 2 gradient accumulation steps). We train for 2 epochs with a constant learning rate of 2.0E-5. For DPO (Rafailov et al., 2023), we reduce the global batch size to 32 (4 H100 nodes with 8 GPUs each and a per device batch size of 1). We train for epochs with learning rate of 5.0E-7 and a DPO beta of 0. Our adapted models are built on top of our annealed checkpoint, and we include the load balancing loss during both SFT and DPO based on our experiments in $B.3. Our preference tuning recipe heavily optimized for DPO based on extensive experiments by Ivison et al. (2023), thus for KTO (Ethayarajh et al., 2024) we experiment with a few settings in Appendix G. Our final KTO adaptation uses the same hyperparameters
DPO, except that use the RMSProp optimizer instead of Adam, which we use for SFT and DPO, and that we reduce the training duration to 1.3 epochs (5,000 steps) for KTO instead of the 3 epochs used for DPO.
We pretrain DLMOE-1B-7B on 256 H100 GPUs for approximately 10 days with NV-link intereonnect across GPUs and InfiniBand interconnect across nodes. We also use H100 GPUs for all our experiments but some use a cluster with GCP TCPx interconnect across nodes
Hardware
Code: https:/ /ttp::///ttttt com/al lenai /open- - rustrct..
38

---

Published as a conference paper at ICLR 2025
Table C3: Pretraining hyperparameters of OLMOE-IB-7B and comparable models trained from scratch. We highlight rows where OLMoE-IB-7B differs from OLMo- 1B. Active params include vocab params. "?" undisclosed settings, FFN feed-forward network, Attn Attention, IR = learning rate, WSD = Weight-Stable-Decay (Hu et al., 2024), LBI. = load balancing loss, Inv Sq Root Inverse Square Root decay (Shazeer & Stern, 2018), trunc truncation, std standard deviation, "varies" = stds that are layer or weight-dependent.
OLMoE 1B-7B
JetMoE
OpenMoE
OLMo- (0724)
Dimension
2,048
2,048
2,048
2,048
Activation
SwiGLU
SwiGLU
SwiGLU
SwiGLU
FFN dimension
1,024
5,632
8,192
8,192
Vocab size
50,304
32,000
256,384
50,304
Attn heads
16
16
24
16
Num layers
16
24
32
16
Layer norm type
RMSNorm
RMSNorm
RMSNorm
non non-parametric
Layer norm eps
1.0E-05
1.0E-05
1.0E-06
1.0E-05
OK-Norm
yes
no
no
no
Pos emb.
RoPE
RoPE
RoPE
RoPE
RoPE 0
10,000
10,000
10,000
10,000
Attention variant
full
MoA
full
full
Biases
MLP & Attn
Weight tying
no
yes
no
no
Init dist
trunc normal
?
normal
Init std
0.02
0.02
varies
varies
Init trunc
3xstd
MoE layers
Every
Every
Every 6th
MoE layer type
dMoE
dMoE
ST-MoE
# Experts
64
32
Activated
2
Vocab params
103M
66M
525M
103M
# Active params
1.3B
2.2B
2.6B
1.3B
Total params
6.9B
8.5B
8.7B
1.3B
Sequence length
4,096
4,096
2,048
4,096
Batch size (samples)
1,024
1.024
2,048
512
Batch size (tokens)
-4M
-4M
~4M
~2M
warmup steps
2,500
2,500
10,000
2,000
peak LR
4.0E-04
5.0E-04
0.01
4.0E-04
minimum LR
4.0E-05
5.0E-05
4.0E-05
optimizer
AdamW
AdamW
Adafactor
AdamW
weight decay
0.1
0.1
0.0
0.1
betal
0.9
?
0.9
0.9
beta2
0.95
0.95
AdamW epsilon
1.0F-08
1.0E-05
LRR schedule
cosine
WSD
Inv Sq Root
cosine
gradient clipping
global 1.0
global 1.0
global 1.0 ?
global 1.0
gradient reduce dtype
FP32
FP32
optimizer state dtype
FP32
2
FP32
LBL weight
0.01
0.01
0.01
Router z-loss weight
0.001
0.001
0.0001
Pretraining tokens
5,033B
1,000B
1.100B
2,000B
Annealing tokens
100B
250B
50B
Annealing schedule
lincar
linear
Annealing min LR
39

---

Published as a conference paper at ICLR 2025
instead. For adaptation, use 32 H100 GPUs for 33 hours to instruction tune and for another 14 hours to preference tune via DPO. For KTO adaptation we use H100 GPUs for 30 hours instead.
D EVALUATION SETUP
Table DI: Summary of downstream evaluation during and after pretraining (OLMES). ARC-C and ARC-E refer ARC-Challenge and (Clark al., 2018), CSQAA=Commoneeneeek (Talmor et al., 2019), OBQA=OpenBooKQA (Mihaylov et al., 2018), other benchmarks are named as in their original works (Clark et al., 2019; Gordon al., 2012: Zellers al., 2019; Hendrycks et al., 2021a; Bisk et al., 2019 Welbl et al., 2017; Sap et al., 2019; Sakaguchi et al., 2019). CF=Completion/CCoon formulation, MCF=Multiple-choiiee referring to the use of few-shots varying from 0-5.
formulation, pmizpointwise-munne www.---uuuaaaaaaaaaaaiee Var=variants
During pretraining
After pretraining (OLMES)
Dataset (J)
Format
Shot Norm Split
Format
Shot CF Norm Split
ARC-C
CF
token val 0
max(MCF.F)
pmi character
test
ARC-E
CF
none val max(MCF,CF)
test
BoolQ
CF
none
max(MCF,CF)
none
CОРA
CF
none val 0
CSQA
CF
token val maX(MCF,CF)
pmi val character
HellaSwag
CF
O
token val max(MCF,C))
val
MMLU
MCF
5
none val max(MCF,CF)
character test
MMLU Var
CF
0-5
token val
OBQA
CF
0
token val
max(MCF.CF)
pmi test character val
PIQA
CF
token val 0
max(MCF.CF)
SciQ
CF
0
none val
SocialIQA
CF
token val
max(MCF.CF)
character
val
Winogrande
CF
0
none val
max(MCF.CF)
none
val
During pretraining We evaluate using a similar in-loop evaluation setup as Groeneveld et al. (2024), with the addition of more tasks such as CommonsenseQA, PIQA, and different implementations of MMLU. Following Groeneveld et al. (2024), for the majority of the tasks, we perform O-shot evaluation using the Completion/Clozz formulation (CF), ranking each answer string using language model probabilities. terms of probability normalization, there is either no normalization (none) or normalization by the number of tokens in the answer (token) when ranking solely based probability may heavily favor shorter answers (Brown et al., 2020). For MMLU, the in-loop evaluation also includes a setup where we increase the total number of instances by including range of 0-shot to 5-shot setups together as we found this provides smoother trends as the training proceeds ("MMLU Var"). We also include the Multiple-choice formulation (MCF) version of MMLU, scoring prediction of answer labels like A/B/C/D, which generally starts to rise only later in training as models only gain the multiple-chote
capability later (at around IT tokens for OLMoE-1B- 7 in Figure F3). We also evaluate perplexity selected validation sets from Paloma (Magnusson et al., 2023; Reid al., 2022; Gao et al., 2020; Soldaini al., 2024; Liang et al., 2023; Merity et al., 2016). All code used for evaluation during pretraining is at https::t///// /ttpt:::///gttubs com/ weleeaai/MMo//eee 6lac1o 04461 64c54 350ec5455dbb 79777776657333 (55552225555555333222053555
After pretraining OLMES We perform evaluations following the OLMES evaluation stan- dard (Gu et al.. 2024), with the suite of tasks in the original paper. OLMES (Open Language Model Evaluation Standard) a standard for reproducible LM evaluations that is open, practical, and docu- mented, providing recommendations guided by experiments and results from the literature (Biderman et al., 2024; Gao et al., 2021; Groeneveld et al., 2023). It is designed to support comparisons between smaller base models that require the Cloze formulation of multiple-choice questions against larger models that can utilize the Multiple-choice formulation. To make our evaluations reproducible, we follow OLMES in prompt formatting, choice of in in-context examples, probability normalization, task formulation, as well as all other details. We summarize this setup in Table 2 and refer to Gu et al. (2024) for more details.
40

---

Published as a conference paper at ICLR 2025
After pretraining DCLM For results the DCLM tasks (Li et al., 2024a) in Table F2, precisely follow their setup using the evaluation code released by the authors at https: /qithub. com/ml /// foundations/ de 1m. "Core" results are the low variance tasks in their evaluation code, while "Extended" corresponds to the heavy tasks.
After adaptation After supervised finetuning and direct preference optimization, evaluate models using subset of the evaluations and the same overall setup used in Ivison et al. (2023) and Wang et (2023). We cover wide range of model capabilities in our evaluation suite including coding (HumanFval Chen et al. (2021)), general and mathematical reasoning (Big Bench Hard Suzgun et (2022), GSM8k Cobbe et al. (2021)); world knowledge (MMLU), general instruction following (AlpacaEval 1.0 Li et al. (2023b), not the length-controlled variant (Dubois et al., 2024)), precise instruction following (IFEval Zhou et al. (2023b)) and safety (XSTest Röttger al. (2024)). We refer to Wang et al. (2023) for more details on each benchmark.
E OPENNESS OF MODELS
We list the openness of various models summarized in Figure 1. We exclude Switch Transform- ers (Fedus et al., 2022), as was published over three years ago and very different from more recent MoE models (MLM objective, Encoder-decoder, etc.).
Grok-86B-314B (xAI, 2024)
Model: Their model is licensed under the open-source Apache 2.0 license.
X Data: Unavailable.
X Code:
Unavailable.
X Logs: Unavailable.
Mixtral-39B-141B and Mixtral-13B-42B (Jiang et al., 2024)
Model:
Their model is licensed under the open-source Apache 2.0 license.
X Data: Unavailable.
X Code:
Unavailable.
X Logs: Unavailable.
DBRX-36B-132B (Databricks, 2024)
The model is licensed under custom non-open- source license with additional use-case restrictions."
Model:
Data: Unavailable.
X
Code: They use closed-source custom adaptations of their public libraries LLM-foundry. composer, and megablocks." 10
X Logs: Unavailable.
Skywork-MoB-22B-144 (Wei et al., 2024)
Model: The model licensed under custom non-open-sourc license.
X Data: Unavailable.
.
"1 /
201. colicense..df
41

---

Published as a conference paper at ICLR 2025
X Code: Unavailable.
X Logs: Unavailable.
DeepSeckV2-21B-2366 (DeepSeck-Al al., 2024b) and DeepSeckMoE-3B-1455 (Dai et al., 2024)
Model:
The models are licensed under custom non-open-source licenses."
X Data: Unavailable.
X Code:
Unavailable.
X Logs: Unavailable.
Aretic-17B-480B (Snowflake, 2024a)
Model:
The model licensed under the open-source Apache 2.0 license.
Data:
They describe their mixture but do not release it.
Code: Unavailable.
X Logs: Unavailable.
Qwen2-14B-57B (Team, 2024b)
Model: The model is licensed under the open-source Apache 2.0 license.
X Data: Unavailable.
X Code: Unavailable.
X Logs: Unavailable.
Jamba-12B-52B (Lieber et al., 2024)
Model: The model licensed under the open-source Apache 2.0 license.
X Data: Unavailable.
X Code: Unavailable.
X Logs: Unavailable.
Qwen1.5-3B-14B (Team, 2024b)
!! Model: The model is licensed under custom non-open-source license."
X Data: Unavailable.
X Code: Unavailable.
X Logs: Unavailable.
JetMoE-2B-9B (Shen et al., 2024)
Model: The model is licensed under the open-source Apache 2.0 license.
Data: They describe their mixture but do not release it.
Code: They make their fork of megablocks publicly available," however, their Megatron-
LM training code not available." 16
X Logs: Unavailable.
Phttps://githun
http:////w/////s////////////aaaa and
http:://hhaa.cccccceereeestsstts 11
httteer//thhhuoooo//y//agggagggs hhtppp///fffco////meee////eeeeea ona ee52c0325 697a0bl 55cc
42

---

Published as a conference paper at ICLR 2025
OpenMoE-2B-9B (Xue et al., 2024)
Model: The model licensed under the open-source Apache 2.0 license.
Data: They make scripts for recreating their data available.
Code: They make their code available."
X Logs: Unavailable.
OLMoF-1B-11
Model: The model is licensed under the open-source Apache 2.0 license.
Data: The data is licensed under the open-source ODC-By 1.0 license.
Code: The code is licensed under the open-source Apache 2.0 license.
Logs: Logs are available with the same open-source license as the code (Apache 2.0).
F ADDITIONAL EVALUATION
HellaSwag
MMLU
ARC-Challenge
70
60
40
Downstream accuracy (%)
40
30
30
PIQA
COPA
WinoGrande
80
65
70
60
OLMoE-1B-78
OLMo-1B (0724)
OLMo-7B (0724)
1x10 4x10a2 7x10²2 1x1023
1x102 4x 7x10 1x1021
1x1022 4x1022 7x1022 1x1021
Training FLOPs
Figure Fl: Evaluation of OLMOE-IB-7B and the current best OLMo models dur- ing pretraining. OLMOE-IB-7B differs from the OLMo models in its MoE architec- ture, several training hyperparameters, and its training dataset, see $2. A version of this plot with tokens as the x-axis and markers where annealing starts is Appendix More results, logs, and configurations: https:/ http:://wwndd..oeo wtw.andbaaaaiiin1i//a htt////momeeeee/eee/eee rrportss Plot-OIMoF B- 7B-vs 1ee-vs--Oi 1B--v-OLM-- 7B Vee-vs-OLLN OLMo- BB- 1zo000oyMM2
During pretraining In Figure F1 benchmark the performance of OLMoE-1B-7B during pre- training with the current best OLMo models (Groeneveld et al., 2024) on commonly used downstream reaches better performance with less compute (FLOPs) than the dense OL models. OLMOF--B--
tasks. We find that across all tasks OLMoF- 11--BB-BB
matches or outperforms OLMo-7B at the end of training despite OLMoE-1B-77 having used less than half as many FLOPs for training and using only IB active parameters. This is likely result of the dataset and modeling changes we make the OLMo setup including MoE-related changes, stability, and performance improvements, outlined in Appendix C. Appendix F contains training and validation loss plots showing very smooth loss curves without major loss spikes during the 5T tokens of our pretraining.
"htpps: / / thub. com/ www.///weerrrhoo/eeeeeeeeeeeeeee htw.://ww...eeeeeeaaaeaarrrreeer fiee
training titutpuppu
43

---

Published as a conference paper at ICLR 2025
Training loss
Validation loss (C4)
Validation loss (The Pile)
5.0
2.4
3.0
4.0
2.2
2.8
3.0
2.0
Performance
2.6
2.0
Validation loss (Books)
Validation loss (Reddit)
Validation loss (Stack)
3.0
3.2
1.2
2.8
3.0
2.6
1.0
2.4
2.8
4
0.1
1
0.1
5
Training tokens (T)
Figure F2: Losses of OLMoE-1B--7B during training. The Books, Reddit, and Stack (Kocetkov al., 2022) datasets are from Dolma (Soldaini al., 2024) via Paloma (Magnusson et al., 2023). More results, logs, and configurations: https: h/tp://ww//////////////// aa/aalaaaaa-//a 1ww.mmlmmmmmmmmmmmmmmmmmmmmmmmmm httpm//mmmmmmmmmeeeeeeeeem
Plot-OLMoE- 1B- 7B- - -
HellaSwag
MMLU
ARC-Challenge
45
60
40
Downstream accuracy (%)
50
35
www
30
PIQA
COPA
WinoGrande
70
80
80
65
70
70
OLMoE-1B-7B
60
OLMo-18 (0724)
OLMo-7B (0724)
50:
1
4
4
Training tokens (T)
Figure F3: Evaluation of OLMOE-IB-7B and the current best OLMo models dur- ing pretraining. Grey vertical lines correspond to where the respective run enters an- nealing with the 1st line being for OLMo-7B, the 2nd for OLMo-1B, and the third for OLMoE-1B-7BB Figure Fl is a version of this plot with training FLOPs as the x-axis. More results, logs, and configurations: https://vendd.ai//i/22 http://2221mmmmooo//rrrmrssss Plot-OrMo--10 1B- 7B vs OLMo- 7B-vs- -m 100
44

---

Published as a conference paper at ICLR 2025
Table F1: More results OLMES. iiniicates of the MCF score, Appendix D. See Table for details on naming and a summary of these results.
Model
ARC C ARC BoolQ CSQA HSwag MMLU OBQA PIQA SIQA WinoG Avg
LMs with -7-9B active parameters
Mistral-7B
78.6*
90.8"
89.3
72.4"
83.0
64.0
80.6"
82.8
71.3
77.9
79.1
OLMo-7B (0724)
68.0"
85.7"
85.3
85.4"
80.5
54.9
67.6"
79.3
76.1
73.2
75.6
DCLM-7B
79.8*
92.3"
87.0
77.0
82.3
64.4"
79.6"
80.1
71.2
77.3
79.1
Llama2-7B
54.2
84.0
86.1
74.2
78.9
46.2
57.8
77.5
59.6
71.7
69.0
Llama3.1-80
79.5*
91.7"
88.5
74.37
81.6
66.9
78.6"
81.1
71.4"
79.0
Gemma2-9B
89.5'
95.5
89.4
78.8
87.3
70.6'
88.4*
86.1
76.0'
78.8
84.0
LMs with ~2-3B active parameters
StableLM-2B
50.6*
75.3
82.3
70.4*
70.3
40.4"
56.6"
75.6
64.3
65.1
Gemma2-3B
67.5*
84.3"
83.6
66.4"
74.6
53.3
68.87
78.5
64.7"
71.8
71.4
JetMoF-2B-9B
61.4*
81.9
85.7
75.3
81.7
49.1"
68.0"
80.3
71.3"
70.7
72.5
OpenMoE-3B-99
29.3
50.6
63.2
21.5
44.4
27.4
34.6
63.3
42.9
51.9
42.9
DeepSeck-3B-16B
53.4
82.7
81.9
72.7
80.4
45.5"
58.4
80.1
59.9
73.2
68.8
Llama3.2-3B
69.6'
85.1
78.3
69.0
77.0
57.87
67.2
77.4
64.9"
69.9
71.6
Qwenl.5-30 3B-B4B
77.4
91.6
85.0
81.4
80.0
62.4
80.6'
81.0
74.1
72.3
78.6
LMs with ~ IB active parameters
OLMo-1B (0724)
36.4
53.5
66.8
42.4
67.5
32.1
44.2
74.0
45.2
62.9
52.5
TinyLlama-1 IB
38.1
69.5
63.6
61.1
60.8
33.6
45.0
71.7
50.4
60.1
55,4
Pythia- 1B
31.4
63.4
56.8
50.9
48.0
31.1
40.4
68.9
46.4
52.7
49.0
Llama3.2-1B
43.5
71.6
69,4
59.6
67.3
38.2
42.0
73.7
52.0
62.5
58.0
7amba2-1BB
55.0"
85..4
76.1
70.1
73.4
44.73
59.87
76.6
58.4
67.2
66.7
DCLM-
57.6"
79.5
80.9
71.3
75.1
48.5"
60.0"
76.6
60.5"
67.8
OLMOF-11 BB-7B
62.1
84.2
79.2
72.9
80.0
54.1
65.4"
79.8
63.0"
70.2
71.1
45

---

Published as a conference paper at ICLR 2025
Table F2: DCLM evaluation metrics the Core and Extended task subsets (Li et al., 2024a). =Core tasks. "annealed" is the final pretraining checkpoint we use for OLMoE-1B-7E and was annealed from the checkpoint step 1,200,000. We left the non-annealing pretraining run train little longer resulting in the 1,220,000 checkpoint.
OLMoE-1B-77 checkpoint (-)
step 1,200,000
step 1,220,000
annealed
OLMo-IB OLMo-7B
AGI Eval LSAT-AR'
24.3
26.5
28.7
28.3
28.3
AGI Eval LSAT-LR
40.2
38.6
37.3
30.2
42,9
AGI Eval LSAT-RC
47.4
43.7
46.6
23.5
61.6
AGI Eval SAT-En
55.3
54.9
52.9
28.2
73.8
AGI Eval SAT-Math CoT
5.5
4.1
6.4
1.8
6.8
AQuA CoT
2.4
ARC Challenge
53.3
53.4
53.8
34,6
48.1
ARC Easy"
77.1
78.5
77.7
64.4
BBQ
48.3
50.6
45.8
67.2
BigBench CS Algorithms"
47.1
50.2
47.2
47.5
53.6
BigBench Conceptual Combinations
$1.5
50.5
56.3
31.1
68.0
BigBench Conlang Translation
3.7
6.1
7.3
BigBench Dyck Languages
19.3
15.9
21.5
26.6
22,2
BigBench Elementary Math QA
26.2
27.0
26.9
26.2
30.4
BigBench Language Identification"
31.9
34.0
31.0
27.0
BigBench Logical Deduction
26.6
25.3
24.6
23.6
27.3
BigBench Misconceptions
59.8
55.3
62.6
55.7
58.0
BigBench Novel Concepts
62.5
62.5
65.6
43,8
53.1
BigBench Operators
36.2
34.3
33,8
23.8
45.2
BigBench QA Wikidata"
68.2
68.8
69.2
67.0
69.9
BigBench Repeat Copy Logic"
15.6
18.8
BigBench Strange Stories
66.7
68.4
69.5
53.4
66.1
BigBench Strategy QA
56.2
58.1
57.0
51.5
68.6
BigBench Understanding Fables
47.1
44.4
47.6
28.0
61.4
BoolQ"
73.3
72.8
73.2
63.7
83.9
COPA
81.0
80.0
78.0
75.0
77.0
CoQA* *
44.4
43.7
45.4
Commonsense@A"
67.2
67.0
69.3
19.6
86.0
Enterprise PIT Classification
$2.3
53.7
52.2
57.3
50.6
GPQA Diamond
22.2
21.2
19.7
19.7
20.2
GPQA Main
24.8
22,3
22.5
20.3
23,0
GSM8K CoT
6.4
7.4
7,4
4.9
30.6
HellaSwag wag O-shot"
76.0
77.0
65.8
HellaSwag 10-shot"
77.6
77.5
78.6
66.3
78.9
Jeopardy"
48.8
48.7
50.3
22.6
46.5
LAMBADA
72.7
72,2
73.3
61.1
71.8
LogiQA
34.9
34.3
34.6
28.7
31.0
MMLU Few-shot
52.2
51.9
53.3
28.4
55.1
MMLU Zero-shot
41.6
42.7
43.3
26.2
Math QA
27.1
27.5
24.1
29.8
OpenBookQA"
41.4
44.0
44.8
36.6
43.4
PIQA
81.2
82.0
76.4
81.7
PubMedQA
56.1
46.6
57.9
0.2
$7.9
SQuAD*
$2.9
52.4
$2.4
0.0
65.5
SVAMP CoT
30.0
28.0
33.0
14.3
44.7
Simple Arithmetic, no spaces
17.6
18.1
20.1
1.2
15.3
Simple Arithmetic, with spaces
19.5
20.6
22.1
1.8
16,0
Social IQA
71.5
70.7
69.3
69.5
84,4
Trivia QA
$4.2
53.0
$5.9
25.1
$1.8
Winogender Female
46.7
50.0
41.7
Winogender Male
55.0
58.3
60.0
63.3
58.3
Winograd"
82.8
83.2
84.6
79,9
83.2
Winogrande"
68.0
68.5
69.0
61.8
67.6
Core
46.3
46.5
47.2
30.2
49,8
Extended
31.3
30.9
32.5
16.9
37.0
46

---

Published as a conference paper at ICLR 2025
G
OTHER EXPERIMENTS
Training loss
Validation loss (C4)
Validation loss (The Pile)
4.0
3.5
3.5
3.0
3.5
3.0
2.5
2.5
3.0
HellaSwag
MMLU Var
ARC-Challenge
35
32
50
30
30
40
28
Performance
30
26
ARC-Easy
PIQA
WinoGrande
60
70
58
55
50
65
60
40
50
55
BoolQ
COPA
SocialIQA
60
80
70
50
42
OLMoE-Mix
+Reddit
60
40
+FLAN
40
10
40
70
100
130
10
40
70
100
130
10
40
70
100
130
Tokens (B)
Figure
G1: logs, and
Adding Reddit
or
FLAN to OLMOE-MIX.
More results,
configurations:
https: /ww//b.anbaaaa.//....... a aaaaaiialii 1esii-11m/// 1tw.1///mmmmm/mreteeaeeeee 10zo11144gggigg
Blot-Adding
Adding Reddit FLAN OLMOE-MIX In Figure G1 benchmark adding the Reddit FLAN (Wei et al., 2022) subsets Dolma 1.7 (Soldaini et al., 2024) to our pretraining data mix (&2). Overall, we do not find either one lead consistent gains, thus do use them in our final data mix.
Load balancing precision Fedus et al. (2022) selectively perform operations related routing in full precision (FP32) to improve stability. In Figure G2 we test whether computing the load balancing loss in full precision improves stability, but not find reduce spikes. Thus, we stick with bfloat16 (BF16).
Noise upcycling For the creation of Qwen2-MoE (Yang et al., 2024a: Team, 2024b; Bai et al., 2023a), the authors add 50% of gaussian noise to feedforward networks before continuing training in an upcycled setup (Komatsuzaki et al., 2023). Komatsuzaki et al. (2023) also report that they experimented with adding noise but did not find it beneficial. In Figure G3, experiment with
47

---

Published as a conference paper at ICLR 2025
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
34
2.9
Performance
32
4.0
2.8
60
30
BF16
28
FP32
55
10 40
100130
40
70
100
130
40
70
100
130
10
70
100
130
Tokens (B)
Figure G2: Load balancing precision, More results, logs, and configurations: https: /wndb. http:///wwm..mmmeerrrre////ooo-- Www.-/----WV111l0000000A0A
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
35
8
60
Performance
3.25
3.00
30
No
40
noise
2.75
Noise
210
250
500
750
10
250
500
750
10
250
500
750
10
250
500
750
Tokens (B)
Figure G3:
Adding noise
to
the upcycled checkpoint.
More results,
logs,
and
configurations:
https:/ http:://w/ddd... 1ama222LLmm/mmmmm/reprttttt
regular upcycling versus adding noise by randomly replacing 50% each MLP with numbers drawn from a normal distribution with a standard deviation of 0.02 following. We find that after 700 billion tokens, the no noise variant still performs slightly better but both appear to converge the same performance. If training further, it is possible that the noise variant eventually outperforms the no noise variant, but that point, may make more sense just train the MoE from scratch 1.2.
Training loss
Validation loss (C4)
HellaSwag
MMLU Var
30
8
3.4
50
Performance
6-
40
Dense
Layer-
4-
3.0
shared
26
30
MoE
50
150
250
50
150
250
50
150
250
50
150
250
Tokens (B)
Figure G4: Sharing the same MoE across layers versus regular dense LM. The num- ber of experts in the MoE. is equivalent to its number of layers. Thus, because the MoE is shared across layers, it has the same number of total and active parameters as the dense model. More results, logs, and configurations: https:/ /ttpo:://wwnd.../// wandb. ai/aiiaaaaaeaea 11m/ hww/ooommooeeeeeeee olmoe/reports/ Plot- Ptot-Sharee rew-rr-dd---e-eeeee -eteee-mmllliooo -W1112044DD000T
Shared Layer Some work has investigated Mixture-of-Experts with weights shared across layers in the context of Universal Transformers (Tan et al., 2023; Csordás et al., 2024; Dehghani et al., 2019). We test whether layer-shared Mixture-of -x-eressss can beat non-shared dense models Figure G4. The layer-shared MoE uses load balancing loss that is applied the model level rather than the layer level. This gives the model more flexibility by allowing it to completely deactivate certain experts for some layers and even emulate dense model by always activating separate expert for
48

---

Published as a conference paper at ICLR 2025
each layer. This makes it generalization of the dense model which motivated hypothesis that it may perform better than the dense model. However, in practice, we find that both perform similarly with the regular dense models even maintaining small advantage validation loss and HellaSwag. One possible advantage of layer-shared MoEs is that they can allow for better load balancing at inference. If prompts come in continuously, then newly incoming prompts previous prompts that have already passed through several layers and sent through the MoE module together, the MoE module is the same regardless of whether is the first last layer. Sharing also reduces throughput by around 20% during training, which further motivates our decision not to use it for OLMoE-1B-77
be batched with
KTO experiments In Table G1 we experiment with the number of steps (5,000 vs. 10,000) and the optimizer (Adam (Kingma & Ba, 2017) vs. RMS) used for KTO (Ethayarajh al., 2024). Based these experiments we use the RMS optimizer and the checkpoint at 5,000 steps in $B.3.
Table GI: KTO adaptation experiments. 5,000 and 10,000 steps correspond 1.3 and 2.6 epochs on our adaptation dataset ($2), respectively.
Human-
Alpaca-
Task (-)
MMLU
GSM8k
BBH
Eval
Eval 1.0
XSTest
IFEval
Avg
Setup (-)
O-shot
shot CoT
O-shot
O-shot
Sbohot
O-shot
O-shot
Shshot
Metric (-)
Pass
KTO, 5,000 steps, RMS
51.2
45.5
34.1
57.1
81.6
86.6
47.5
$7.7
KTO, 10,000 steps, RMS
51.0
41.0
34.7
53.8
81.0
62.3
47.5
54.2
KTO, 5,000 steps, Adam
51.2
42.0
35.3
55.6
81.0
84.5
46.6
56.0
KTO, 10,000 steps, Adam
51.0
43.0
34.1
54.9
79.7
62.7
47.5
53.3
H ANALYSIS
H.1 DETAILS OF ANALYSIS IN $5
Router saturation We define router saturation the proportion expert activations at some intermediary checkpoint at time that matches the expert IDs activated at some final checkpoint over the same dataset:
N
|
E
(T)
Router Saturation
(5)
N
where:
N: The total number of tokens in the dataset.
k: The number of top-k experts activated per input token. While we train with k: = 8 ($2), also analyze i 1 by only looking the expert with the highest routing probability.
e0, The set experts activated for the ith token at the fth checkpoint.
e(T) The experts activated for the ith token at the final checkpoint T'.
|e(") E [: The number of common experts activated for the ith token between the äth and final checkpoints.
Router saturation thus corresponds to whether the router weights are still learning which expert will process certain data. A value of 100% indicates that the router at the intermediate checkpoint will route to the same experts as the final checkpoint router. However, even at 100% saturation the router weight still change and adapt the exact router probability for each expert. These probabilities are used to scale the output of the respective expert the model. For OLMoEE1B. 1B-B with its 64 experts, random routing equals saturation of 1/64 1.6% for k and 8/64 12.5% for 8.
49

---

Published as a conference paper at ICLR 2025
Expert co-activation We define expert co-activation the proportion of times specific experts, Ej and Ej. are simultaneously activated out of the total number of activations of one those experts:
E,
Expert co-activation( E,, F.) Ne.
(6)
where:
E; The first expert.
Ey: The second expert.
Ey.
= The number of times experts E, and Ej are activated together.
Ng. The total number of times expert Eg is activated.
co-activation of 100% indicates that if E, activated, E, also always activated. A value of 0% indicates that the experts never co-occur. If multiple expert pairs have high co-activation, may suggest that these experts could be merged, benefiting less from keeping them separate. In a distributed setup, could place highly co-activated experts the same device to reduce communication costs during model inference.
Domain specialization We define domain specialization as the proportion of tokens from particu- lar domain that get routed a particular expert E,:
(k) N
F..D
Domain specialization( E,, D) ND
(7)
where:
Eg: The ith expert in the model.
D: The domain from which the data originates.
k: The number experts considered (e.g., 8 means considering the top 8 experts with the highest routing probabilities).
(M(A) :E.D: The number of tokens from domain D for which E is among the top-k selected experts.
Np: The total number of tokens from domain processed by the MoE.
Domain specialization thus refers to the specialization of expert E; domain D. A value of 100% indicates that all data from that domain is routed to E,, whereas 0% indicates the expert is never used for that domain and can be removed from the model without affecting performance in that domain.
Vocabulary specialization define vocabulary specialization as the proportion of tokens with token ID z (also called vocabulary element) that are routed to one particular expert E, out of all experts that layer:
((K)
R.
Vocabulary specialization 24, = N
(8)
where:
Es The ith expert in the model.
x: The token ID being analyzed.
k: The number of experts considered (e.g., 8 means considering the top 8 experts with the highest routing probabilities).
E. The number of times input data is routed to E, for
N The total number of times input data is routed across all experts for
50

---

Published as a conference paper at ICLR 2025
Vocabulary specialization thus refers to how specialized particular expert is item. We distinguish imput and output variants of this specialization, where z either the input token ID or the next output token ID (either the ground-truth next token ID the token ID predicted by the model). value of 100% indicates that for all occurrences of that vocabulary element, input data is routed to F,, whereas 0% indicates expert that is fully irrelevant for that vocabulary element and can be effectively removed from the model without affecting performance whenever the token ID appears.
vocabulary
H.2 ADDITIONAL ANALYSIS
Layer
Layer 15
40
10
60
20
13
43
46
7
58
53
60
5
21
41
7
18
62
31
15
30
56
26
29
4.
43
8
0
155
46
31
50
47.
9
-0
40
20
.43
33
S
tP
18
46
85
00
T2
15
22
TOWNN
Figure HI: Co-activation among experts of OLMOE-IB-78 on random 0.5% of the C4
validation data. We display the 32 experts with the highest maximum co-activation score via their expert IDs on the x- and y-axis. See Figure for layer 7.
Per layer
Per expert in layer 7
Vocabulary ppecialization
60
taken ID Predicted output token ID Ground-truth output
20
101112131415
Layer
Expert
Vocabulary specialization of OLMoE-1B-7B across layers and experts. To compute vocabulary specialization per layer (left) average the specialization of each expert in that layer. Dashed lines (right) correspond to the average of layer as depicted left. We display the first 32 experts of 64. This plot when & in Equation 8.
Figure H2:
Vocabulary specialization (96)
1100
layer
100
expert layer
95
90
85
Input taken ID
80
20
Predicted output
Ground-truth output token ID
0011122311115
2
6 Expert ID
27
58
Layer ID
Figure H3: Vocabulary specialization for OLMOE-1B-7B when considering all 8 activated experts. Equivalent to k = 8 in Equation 8.
51

---

Published as a conference paper at ICLR 2025
Table HI: Vocabulary specialization in the 7th layer of OLMOE-IB-78. We use l = 1 (Equa- tion 8) and random 0.5% of the C4 validation data excluding token IDs with <10 appearances. See
Table 4 for more.
Expert ID Input token IDs
Predicted output token IDs
58
("
(100%)
(100%)
(94%)
(92%)
such (100%)
(100%)
see
(95%)
(92%)
(92%)
(90%)
(89%)
which (91%)
driving (91%) UK
(90%)
(88%)
(87%)
(87%)
(86%)
(88%)
including (88%)
normal (88%)
7
Him (100%)
inde (100%)
Jesus (98%)
rella (100%)
Him (94%)
sin
(90%)
God
(90%)
pray
(81%)
Holy (80%
prince (80%)
glory (72%)
Jesus (69%)
Quran
(80%)
God (77%)
Lord (76%)
Lord
(68%)
Christ
(65%)
Spirit (55%)
glory (75%)
Spirit (66%)
Christ
(65%)
Holy
(53%)
God
(50%)
Prayer (50%)
37
Sunday
(100%)
Tuesday
(100%)
days
(91 (1%)
anniversary (90%)
month
Thursday
(100%)
Olympic
(100%)
(88%)
week
(84%)
mpi (83%) semester
Christmas
(100%)
rugby
(100%)
(81%) %))
mand
(80%)
Olympics (78%) cent
Championship
(100%)
weekends
(100%)
(76%)
season (76%)
pemm (75%)
0
ESM
(100%)
icillin (100%)
agra
(98%)
(100%) sil (96%) pills (91%) vi
aust
(96%) asa (93%) pills (92%)
mg
(90%)
xen (87%)
pharmacy (87%)
gener
(85%)
(82%)
login (82%)
doc
(81%)
(85%)
aust
(82%)
mg (75%)
Content
generic
(81 16)
cd (81
Essay
(81%) (%)
(75%)
(73%)
THAT
(73%)
dispens
password (81%) Content (80%)
(68%)
icillin (68%)
generic (66%)
3
grandmother (92%)
brother (91%)
Daisy
hood (36%) mother (35%) inde (31%)
(83%) daughter (78%)
(75%)
father
boy (29%) girl (28%) married (27% %
(72%)
wife (70%) husband (70%)
lady tri (21%)
(20%) died (18%) taught
(63%) dad (62%) boy (61%)
(14%) lived (13%) knew (10%)
layer
expert layer
cacabllary specialization
90
80
85:
60
80-
75
40
Input taken ID
70
Predicted oucput token
Ground-truth output token
65
0123456
4
Layer
Expert ID
Figure H4: Vocabulary specialization for Mixtral-8x7B when considering all activated experts.
Equivalent to k = 2 in Equation 8.
52

---

Published as a conference paper at ICLR 2025
GitHub
arXiv
Books
Expert
40
30
20
10
Frequency (normalized
40
Expert 4
30
20
10
Expert
40
30
20
10
50
100
0
50
100
50
100
50
100
Routing Probability (%)
GitHub
arXiv
Books
40
Expert
30
20
10
Frequency (normalized
40
Expert 4
30
20
10
40
Expert
30
20
10
100
50
100
100
50
100
Routing Probability (%)
Figure H5: Vocabulary specialization across domains OLMoE-1B-7B (top) and Mixtral- 8x7B (bottom). We visualize how often token IDs get routed to specific experts. We only include IDs that appear at least times in the various corpora. Vertical gray lines correspond routing (8/64=12.5%) for OLMoE-1B-- for Mixtral as has 8 experts, 2 of which are activated), For example, among all token IDs in GitHub that get routed to Expert 0 at least 8 times for OLMoE-10 1---------B- ~40% of them get routed to Expert with a probability of 100% (upper left) indicating that Expert is specialized on those token IDs. For OLMoE-1BB77 there is much frequency the routing probability extremes (0% 100%) indicating that these experts exclusively focus on certain token IDs, especially for specific domains ($5.3) like GitHub and arXiv.
uniform
as it has 64 experts, 8 of which are activated; 2/8=25%
53

---

Published as a conference paper at ICLR 2025
OLMoE
OLMoE-SFT
OLMoE-DPO
Layer
Layer
Layer
100
Domain specialization (%)
100
0
100
0
16
24
32
40
48
56
0
8
16 24
32
40 48
56
0
8
16
24
32
40
48
56
Expert ID
Figure H6: Load imbalances in selective layers after adaptation. We visualize how often tokens from our instruction tuning dataset ($2) get routed to the 8 active experts out the 64 total experts (k: in Equation 7). Horizontal gray lines correspond to uniform routing (8/64=12.5% per expert). Although we run SFT and DPO without loss balancing loss ($B.3), we observe that the load distribution does not change substantially.
54

---

Published as a conference paper at ICLR 2025
GitHub
arXiv
Wikipedia
Books
C4
Layer
Layer
Layer 15
50
0
Now Bojon
50
Domain specialization (%)
0
50
0
50
0 Ay
50
16
24
32
40
48
56
8
16
24
32
40
48
56
0
8
16
24
32
40
48
56
Expert ID
Layer
Layer 7
Layer 15
100
0
Domain pecialization (%)
100
0
100
0
100
0
100
0
2
4
2 4
2 4
Expert ID
Figure H7: Domain specialization of OLMoF--1B-7 (top) vs. Mixtral-8x7B (bottom) of the top-1 routed expert. We visualize how often tokens from different domains get routed to the 64 (OLMoE) or 8 (Mixtral) experts at the end of pretraining. Unlike in Figure 7, here we only consider tokens routed to the top-1 expert (A in Equation 7). Horizontal gray lines correspond to uniform routing (1/64=1.56% per expert for OLMOE-IB. and 1/8=12.5% for Mixtral).
55

---

Published as a conference paper at ICLR 2025
GitHub
Layer 15
17
24
30
36
44
57
other
Layer 7
2
4
12
17
21
27
35
42
45
58
other
Layer 0
2
15
17
21
27
36
41
47
49
54
other
arXiv
Layer 15
17
34
other
Layer 7
2
4
17
35
36
58
other
Layer 0
17
36
38
54
other
Wikipedia
Layer 15
35
41
43
57
63
other
Layer
27
31
33
42
49
other
Layer 0
3
4
11
17
29
36
42
51
other
Books
Layer 15
2
4
13
20
22
23
39
45 47
52
54
55
56
61
62
63
other
Layer 7
20 23
28
34
42
48
52
54
55
56
61 62
63
other
Layer 0
1
5
16
20
28
31
36 38
41
46
49
50
58
59
62
other
Figure H8: OLMOE-I1 1B-7BB token routing across layers. We visualize how often tokens from different domains get routed to pair of experts across layers under top-1 routing, corresponding to Figure H7. The size of each rectangle is proportional the total number of tokens an expert receives, while the flow between show experts that receive tokens 50% above random chance and use stronger coloring for larger flows. We observe some instances of cross-layer coordination between pairs of experts, e.g., expert 27 in layer 7 and expert 57 in layer 15 process substantial fraction of Wikipedia tokens together. The flows between layers 0 7 and 7 15 independent in this visualization.
experts shows the proportion of tokens routed to both experts. We only
56

---

Published as a conference paper at ICLR 2025
GitHub
Layer 15
5
Layer 7
2
4
Layer 0
1
3
5
arXiv
Layer 15
3
5
6
Layer 7
2
3
6
Layer 0
0
1
2
3
4
5
6
7
Wikipedia
Layer 15
2
3
4
6
7
Layer 7
1
3
5
7
Layer 0
1
2
3
4
5
6
7
Books
Layer 15
6
7
Layer 7
2
3
4
5
6
Layer 0
2
3
4
5
7
Figure H9: Mixtral-8x7B token routing across layers. We visualize how often tokens from different domains get routed pair of experts across layers under top-1 routing, corresponding to Figure H7. The size of each rectangle is proportional to the total number of tokens expert receives, while the flow between two experts shows the proportion of tokens routed both experts. The flows between layers 0 - 7 and 7 - 15 are independent in this visualization.
57

---

Published as a conference paper at ICLR 2025
I ARTIFACTS
Table II: All artifacts released and used in this work. We point from the name used for given
artifact in this work (e.g. Figure 1) to the URL where it can be obtained.
Artifact
Public link
OLMoE- ----7-BBBBB 1B-7B
OLMOE MAMOE-IB-7B---- IB-7B-INSTRUCT
OLMOE-MIX
http:///lk.c//ttaastt////laal/
SFT data
http:///hf.co/aattsetts/llllad_
KTO/DPO data
aitrffescbcckkhnarrreeeeelleeeee
Code
attesr//etthhaacccmm//leeea/aaae
Logs
BLOOM-7B
httes://hf.cc/higgcccemee/////dp
Moo:-eekMME-33B-166
DeppSeckMoE-3B-100-eee
httpss//Efccoepppeeerrr///eeeeee chat
DCLM-IB
#ttp://Ihf..o/RRR--///022111111
DCLM-7B
Falcon-7B
http:://ll.co/tiiua///aooo00.11.
Gemma2-3B
http:://af.co/poolle///mmmm/
Gemma2-9B
http:://hf..ooppooll///mmmmam
CeMMEE-BB999
attp:://E..co/tttnn///mmmm-B
(cMMoE-2B-9B+SP
JetMoE-2B-9B+Chi.
Llama-7B
https///ff.c///uggyy/amm//mmm11
Llama2-7B
Llama3.J-8B
MPT-7B
http:///ll.co//mmaaacc///1111113
Mistral-7B
Mixtral-8x7B
OL IB (0724)
http:://ff.ooaaaaae////0001111-2
OLMo-7B (0724)
OpenMoF-3B-99
Pythia-7B 7B
http:://Ef.o//lottaaa////tt2222
Qwen -5-3B-14B
Qwenl.5-23 1-e.1-5----BB----
StableLMM-2BB
TinyLlama-l
J SELECTING THE NUMBER OF TOTAL AND ACTIVE PARAMETERS
In addition what mention in $4.1, 1, there three key reasons select configuration of 1B active parameters and 7B total parameters for OLMoF-1B--B.
Model training 7B total parameters allow for full-parameter training on single GPU, Specifically, model can be trained on one 80GB VRAM GPU (e.g. A100 H100) it requires around 70GB of memory for training the model 16-bit with an 8-bit optimizer (Anthony et al., 2023). This makes the model significantly more accessible to researchers who are often constrained by single GPU and also bypasses the need for more complicated distributed training across multiple GPUs. A slightly larger model (e.g. JetMoF-2BB-99 may longer fit under this setup.
58

---

Published as a conference paper at ICLR 2025
Model usage on laptops Laptops commonly have around 16GB of RAM, thus 7B parameters corresponding to 14GB in 16-bit precision perfectly fit into most laptop's s RAM. With the speed of IB parameters, the configuration of OLMoE-1D 1---B could make it ideal local assistant.
Model usage on phones We have been able to run OLMOE-1B-7B on an iPhone by quantizing the model to 4-bit after which it requires around 3.5GB (0.5 * 7) of RAM. This is just below the
5GB RAM limit that is commonly imposed for an iOS app 18 leaving 1.5GB of RAM for other functionalities of the app. Thanks to the 1B active params OLMOE-IB-7BB runs very fast on smartphones; we were able to run it at 110 tokens/second on an iPhone 16. This enables applications that might not make sense with larger and slower models, such as having the model quickly read multiple long files and summarize them.
K LIMITATIONS AND FUTURE WORK
We highlight four key limitations with this release of OLMOF-IB-7BB We look forward to addressing these issues in future iterations of OLMoE.
More parameters OLMoE-1B- 1B-7B has 7B total parameters out of which IB are activated for each input token. This small size makes OLMoE-IB-B very cheap use, yet we demonstrate in this work that it outperforms much more expensive models (Figure 1). We provide further reasons for this precise configuration in Appendix However, using only 1B parameters for each input token also limits the capabilities of OLMoE- -B-BBB as seen by its performance compared to models that use >7x more parameters, such as Llama3. in $3. While it may be possible that more parameters are not needed to match 8B models and beyond (Karpathy, 2024), in the short-ters soor-termm adding parameters is an easy way to improve the performance of OLMOE. Significantly adding parameters may, however, make dropless routing (Gale et al., 2022) as used in this work more challenging and may require expert parallelism (Lepikhin et al., 2020) with token dropping. We note that the DBRX model also uses dropless routing (Databricks, 2024; Gale al., 2022) at scale of 36B active and 132B total parameters. A different approach to more parameters could be allowing the model to utilize more than IB parameters per input, possibly via recursion (Dehghani et al., 2019) agentic workflows (Wang et al., 2024a; Yang et al., 2024b). Relatedly, changing the allocation of parameters to e.g. vocabulary versus -occbucayy parameters is another avenue for improvement (Tao et al., 2024).
More data We train OLMOE-IB-7B for 5 trillion tokens, however, some recent dense models train significantly longer, such as Llama 3 with 15 trillion tokens (Dubey et al., 2024). To the best of our knowledge, there has been no large MoE that has been overtrained (Gadre et al., 2024) as much as OLMOE- B-7B. Specifically, taking the active parameters of OLMoE- B-7B our token multiplier (Gadre et al., 2024) around 5,000 (5T/ IB). There are likely benefits to training even longer, but to what degree overtraining is effective for MoEs and how it differs from dense models still requires more research (Allen-Zhu & Li, 2024).
Multimodal OL 1B-7B is text-only large language model, thus cannot take inputs or produce outputs in other modalities like images or audio. This limits its utility for the large variety of multimodal use cases of such models (Huang et al., 2018; Su et al., 2020; Chen et al., 2020; Kiela et al., 2021; Muennighoff, 2020; Radford et al., 2022: Bai et al., 2023b: Driess et al., 2023; Dubey et al., 2024). There has been early work on open multimodal MoEs (Mustafa et al., 2022; Lin et al., 2024a; Li et al., 2024b; Shen et al., 2023b; McKinzie et al., 2024; Wu et al., 2024a) and we look forward to making future versions of OLMoE a part of that.
Multilingual We pretrain OLMoE-IB-77 on predominantly English corpus and exclusively evaluate on English tasks. This may severely limit the usefulness of our model for research on non-English language models (Lovenia et al., 2024; Singh et al., 2024; Üstün et al., 2024; Enevoldsen al., 2024; Son et al., 2024; Xiao et al., 2023). While there has been work on training language- specific LMs (Laukkonen et al., 2023; Faysse et al., 2024), it is more likely that as we add more data to build better future iterations of OLMOE we will mix more non-Englin non-English data due to data
phttps://gtt www.bb..../theeeeeeeeeeeeeeeeeee ab/ httpi//appaaaa//cc/////a/a/a//as /ssps//ppssuuss/999
59

---

Published as a conference paper at ICLR 2025
constraints (Muennighoff al., 2023b). This may make future OLMOE models perform better in non-English languages.
L OLMOE-IB-7B-OP
We introduced OLMOE-IB-7B in September 2024. In January 2025, we released a better model, OLMOE-1B-7B- which we discuss here.
Table Ll: DOLMINO composition and sampling distribution used for OLMoE-IB-7B--D
Source
Total tokens
Source %
Mix %
Filtered DCLM
752B
6.85
50.2
Decontaminated FLAN
17.0B
100
16.7
StackExchange Q&A
1.26B
200
2.47
peS2o
58.6B
16.7
9.52
Wikipedia/Wikibooks
3.70B
100
3.57
Dolmino Math
10.7B
200
17.5
For pretraining, BLMMO-IBB-BB-1111 uses the data mix for the first stage of training. Following OLMo (OLMo et al., 2024), we anneal this new model on a curated mix of high-quality sources. We sample this mix from the DOLMINO dataset, 19 collection of high-quality web pages, academic content, question answering pairs, instruction data, and math problems. We use the same 100B tokens sample of DOLMINO used to anneal OLMo 2 13B; a summary of this dataset is in Table LI.
Table L2: OLMOE-1B-7B-0924 and OLMOE-IB-7B-011 on OLMES, We bold the best performance, indicates use of the MCF score, see Appendix for evaluation details.
OLMoE release
|RC C ARC E BoolQ CSQA HSwag MMLU OBQA PIQA STQA WinoG Avg
Sep 2024 (0924)
62.1
84.2
79.2
72.9
80.0
54.1
65.4
79.8
63.0
70.2
71.1
Jan 2025 (0125)
67.5
84.4
80.6
70.8
81.7
56.3"
69.6"
78.7
66.8
70.6
72.7
We compare OLMOE-IB-7B-012 with OLM0E-1B-7B In Table L2. Overall, the new model is notable improvement over the previous iteration being better on average (+ (.66) and notable datasets like MMLU (+2.1).
Following this improved annealing setup, adapt OLMOE-IB-7B. from Tülu 3 (Lambert et al., 2025). This recipe represents an updated version of the one originally used for OLMOE. features an improved SFT mix, better sampled DPO data, and PPO step that leverages verifiers for the model reward. We compare this new iteration using the evaluation setup from Tülu (which differs from other evaluations this paper) in Table L.3. After adaptation, new model is significantly better, with 10-point gain the benchmark average.
using the post-training
The new models and datasets freely available the Hugging Face hub." about this release, we refer to its announcement on Ai2's website."
20 For more information
"wwgginggace..c httgg//...ee.oootttte//////e///a
"hf http:///eeecctooo/a/aaaeee/aeeee 4o0 21 "allenaa wtgg//lgg//looeeeeeeeeooo
60

---

Published as a conference paper at ICLR 2025
Table L3: OLMOE-1B-7B-0924 and OLMOE-1B-7B-0125 after adaptation. We bold the best
performance.
OLMOB-1B-7B-0924
OLMoE-1B-7B-012:
Skill
Benchmarkjev))
+SFT
+DPO
+SFT
+DPO
+RLVR
Avg.
39.7
39.8
46.6
49.3
49.8
Knowledge
MMLU (0 shot, CoT)
54.3
54.6
55.3
54.9
55.1
PopQA shot)
21.0
20.6
20.1
19.7
19.8
TruthfulQA shot)
44.7
49.1
45.5
50.0
50.6
Reasoning
BigBenchHard,3 shot, CoT)
36.6
36.8
37.3
37.4
38.6
DROP shot)
34.7
34.5
48.6
48.4
47.9
Math
MATH (4 sbot CoT, Hex)
8.2
8.2
21.4
20.4
21.4
GSMSK, (8 shot, CoT)
42.5
47.4
55.7
64.6
72.4
Coding
Human HumanEval HumanEEval
63.7
63.0
62.6
61.9
62.3
57.4
58.9
55.7
57.6
54.4
IF & chat
IFEval loose)
41.2
45.3
56.6
65.6
66.4
A AlpacaEvval 2LC win)
6.4
7.5
5.8
19.5
18.0
Safety
Safety (6 task avg.)
65.8
51.4
94.5
91.4
90.4
61


Reviews:
[{"review_id": "wsaqtgZmFV", "reviewer": "Reviewer_7Ufw", "summary": "This paper introduces OLMoE, a fully open, state-of-the-art language model built on a sparse Mixture-of-Experts (MoE) architecture. The authors conducted extensive experiments to validate the effectiveness of the proposed method, including evaluations after pre-training and adaptation phases. Additionally, they explored key design choices within the MoE framework, examining factors like expert granularity, routing strategies. Their analyses provided valuable insights into MoE, including router saturation, expert co-activation, and domain/vocabulary specialization.", "questions": "See Above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The writing in this paper is clear and easy to follow.\n- The paper advances MoE research by providing a fully open-sourced, state-of-the-art MoE architecture, which is beneficial for the research community.\n- The paper presents a thorough analysis of key design choices in MoE, offering valuable guidance on building high-performance MoE models.\n- The analysis is insightful, with discussions on phenomena such as router saturation and expert co-activation providing fresh perspectives and meaningful implications for the field.", "weaknesses": "I have a question regarding the experimental results: were the model parameters quoted directly from the original paper for the results shown in Table 2? For instance, in the original paper, OpenMOE’s activation parameter count is reported as 2.1B, whereas Table 2 shows an activation parameter count of 2.9B for OpenMOE. I recommend that the authors carefully verify the accuracy of these values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOI4jeOedV", "reviewer": "Reviewer_PJU8", "summary": "This paper presents a mixture-of-experts (MoE) LLM model called OLMoE that has 1B active parameters and 7B total parameters. The OLMoE model Pareto dominates many state-of-the-art models in the performance vs. active parameters space. The paper explores and presents insights on what is optimal in the design-space of MoE parameters and present analysis of routing behavior in MoEs.", "questions": "The work is well presented and possible suggestions for improvements are addressed in the future work section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) Strong empirical results with state-of-the-art performance for 1B active parameters.\n2) Good exploration of the MoE design space which forms a good guide for MoE model design.\n3) Novel analysis of routing behavior in MoE models during training and inference.\n4) This is the only MoE model where the model weights, code, data and checkpoints are openly available and thus the work is entirely reproducible.", "weaknesses": "1) Other state-of-the art MoE models in related works are not exactly in the same parameter count configuration (1B/7B) so an exact comparison cannot be made to this model's performance.\n2) Most of the design choices and training choices are based on prior work and the novelty is more in the design space exploration and analysis of routing behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRHBIfCcem", "reviewer": "Reviewer_vUvg", "summary": "This work is devoted to sharing the insights, data, and checkpoints of a series of MoE LLMs. The model achieved promising results on various benchmarks as a fully open model family.", "questions": "1) What do you think about the necessity of expert parallelism? This model used dropless MoE, so it anyway will be unbalanced when using expert parallelism during training and inference. Without expert parallelism, it is still okay when the model is small. However, if we are aiming at a very large model, which has very large experts even if we are using the \"fine-grained MoE\", the expert parallelism would still be required? So how can we handle the token drop problem in this case?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) There is no doubt that training MoE LLMs is challenging. This work offers a couple of important takeaways about how to train a good MoE LLMs, which is very helpful to the community.\n2) The presentation is very clear. For instance, the Table 1 delivers many key designs clearly at the early section of the paper.\n3) The model performance is good as well. As shown in Table 2 and 3, the model performs competitive with dense open models and partially open models (e.g. Qwen, Deepseek).\n4) The Analysis in Section 5 is informative, which greatly help readers and authors to understand how is the model working. This can also greatly speedup the growth of the community.", "weaknesses": "1) Although the model has been relatively large, it is still much smaller than the SoTA MoE LLMs. I understand it is hard to get enough training resource for a fully open projects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)