OLMoE: Open Mixture-of-Experts Language Models

Authors: Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, Hannaneh Hajishirzi
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, mixture-of-experts, open-source

Abstract:
We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present novel findings on MoE training, define and analyze new routing properties showing high specialization in our model, and open-source all our work: model weights, training data, code, and logs.

Decision: accept (oral)
Reviewer scores: 8, 8, 10

Reviews and discussion:
<thread>
creation date:
1730219345553

thread_type:
official_review

<thread_object>
creator: reviewer_vuvg

<content>
summary:
This work is devoted to sharing the insights, data, and checkpoints of a series of MoE LLMs. The model achieved promising results on various benchmarks as a fully open model family.

strengths:
1) There is no doubt that training MoE LLMs is challenging. This work offers a couple of important takeaways about how to train a good MoE LLMs, which is very helpful to the community.
2) The presentation is very clear. For instance, the Table 1 delivers many key designs clearly at the early section of the paper.
3) The model performance is good as well. As shown in Table 2 and 3, the model performs competitive with dense open models and partially open models (e.g. Qwen, Deepseek).
4) The Analysis in Section 5 is informative, which greatly help readers and authors to understand how is the model working. This can also greatly speedup the growth of the community.

weaknesses:
1) Although the model has been relatively large, it is still much smaller than the SoTA MoE LLMs. I understand it is hard to get enough training resource for a fully open projects.

questions:
1) What do you think about the necessity of expert parallelism? This model used dropless MoE, so it anyway will be unbalanced when using expert parallelism during training and inference. Without expert parallelism, it is still okay when the model is small. However, if we are aiming at a very large model, which has very large experts even if we are using the "fine-grained MoE", the expert parallelism would still be required? So how can we handle the token drop problem in this case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595485046

thread_type:
official_review

<thread_object>
creator: reviewer_pju8

<content>
summary:
This paper presents a mixture-of-experts (MoE) LLM model called OLMoE that has 1B active parameters and 7B total parameters. The OLMoE model Pareto dominates many state-of-the-art models in the performance vs. active parameters space. The paper explores and presents insights on what is optimal in the design-space of MoE parameters and present analysis of routing behavior in MoEs.

strengths:
1) Strong empirical results with state-of-the-art performance for 1B active parameters.
2) Good exploration of the MoE design space which forms a good guide for MoE model design.
3) Novel analysis of routing behavior in MoE models during training and inference.
4) This is the only MoE model where the model weights, code, data and checkpoints are openly available and thus the work is entirely reproducible.

weaknesses:
1) Other state-of-the art MoE models in related works are not exactly in the same parameter count configuration (1B/7B) so an exact comparison cannot be made to this model's performance.
2) Most of the design choices and training choices are based on prior work and the novelty is more in the design space exploration and analysis of routing behavior.

questions:
The work is well presented and possible suggestions for improvements are addressed in the future work section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731176691504

thread_type:
official_review

<thread_object>
creator: reviewer_7ufw

<content>
summary:
This paper introduces OLMoE, a fully open, state-of-the-art language model built on a sparse Mixture-of-Experts (MoE) architecture. The authors conducted extensive experiments to validate the effectiveness of the proposed method, including evaluations after pre-training and adaptation phases. Additionally, they explored key design choices within the MoE framework, examining factors like expert granularity, routing strategies. Their analyses provided valuable insights into MoE, including router saturation, expert co-activation, and domain/vocabulary specialization.

strengths:
- The writing in this paper is clear and easy to follow.
- The paper advances MoE research by providing a fully open-sourced, state-of-the-art MoE architecture, which is beneficial for the research community.
- The paper presents a thorough analysis of key design choices in MoE, offering valuable guidance on building high-performance MoE models.
- The analysis is insightful, with discussions on phenomena such as router saturation and expert co-activation providing fresh perspectives and meaningful implications for the field.

weaknesses:
I have a question regarding the experimental results: were the model parameters quoted directly from the original paper for the results shown in Table 2? For instance, in the original paper, OpenMOE’s activation parameter count is reported as 2.1B, whereas Table 2 shows an activation parameter count of 2.9B for OpenMOE. I recommend that the authors carefully verify the accuracy of these values.

questions:
See Above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731707068248

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their detailed reviews. We have made the following changes to the paper and uploaded a new revision:

1. We have fixed a typo in the active parameter count of OpenMoE in Table 2 thanks to feedback from reviewer 7Ufw.
2. We have added a discussion of the advantages of the specific OLMoE configuration (1B/7B) in a new Appendix section (J), including quantization experiments we ran achieving 110 tokens/second generation speed on an iPhone with OLMoE, in response to reviewer PJU8.
3. We have added notes on expert parallelism and how the community can scale up OLMoE in Appendix K to address the question raised by reviewer vUvg.

Overall, we are glad reviewers have found the paper to provide fresh and novel insights. Reviewers have also pointed to the open nature and reproducibility of the work, as well as its artifacts. Building on these, we hope that together with the broader community, we can further advance the state of fully open language models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a state-of-the-art language model leveraging a sparse Mixture-of-Experts (MoE) architecture. It presents novel findings on MoE training, defines and analyzes new routing properties showing high specialization in their model, and open-sources all aspects of their work, including model weights, training data, code, and logs.

The paper provides significant contributions to the field of MoE models, offers valuable insights, and is fully open-source, making it a valuable resource for the research community. Based on the reviews and the authors' thorough responses, I recommend acceptance.