OLMoE: Open Mixture-of-Experts Language Models
Authors: name: Niklas Muennighoff; name: Luca Soldaini; name: Dirk Groeneveld; name: Kyle Lo; name: Jacob Morrison; name: Sewon Min; name: Weijia Shi; name: Pete Walsh; name: Oyvind Tafjord; name: Nathan Lambert; name: Yuling Gu; name: Shane Arora; name: Akshita Bhagia; name: Dustin Schwenk; name: David Wadden; name: Alexander Wettig; name: Binyuan Hui; name: Tim Dettmers; name: Douwe Kiela; name: Ali Farhadi; name: Noah A. Smith; name: Pang Wei Koh; name: Amanpreet Singh; name: Hannaneh Hajishirzi
Published: 2024-09-03 17:08:20

Abstract: We introduce OLMoE, a fully open, state-of-the-art language model leveraging
sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
uses only 1B per input token. We pretrain it on 5 trillion tokens and further
adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
models with similar active parameters, even surpassing larger ones like
Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
training, analyze routing in our model showing high specialization, and
open-source all aspects of our work: model weights, training data, code, and
logs.