OLMoE: Open Mixture-of-Experts Language Models
Authors: name: Niklas Muennighoff, name: Luca Soldaini, name: Dirk Groeneveld, name: Kyle Lo, name: Jacob Morrison, name: Sewon Min, name: Weijia Shi, name: Pete Walsh, name: Oyvind Tafjord, name: Nathan Lambert, name: Yuling Gu, name: Shane Arora, name: Akshita Bhagia, name: Dustin Schwenk, name: David Wadden, name: Alexander Wettig, name: Binyuan Hui, name: Tim Dettmers, name: Douwe Kiela, name: Ali Farhadi, name: Noah A. Smith, name: Pang Wei Koh, name: Amanpreet Singh, name: Hannaneh Hajishirzi

We introduce OLMoE, a fully open, state-of-the-art language model leveraging
sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
uses only 1B per input token. We pretrain it on 5 trillion tokens and further
adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
models with similar active parameters, even surpassing larger ones like
Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
training, analyze routing in our model showing high specialization, and
open-source all aspects of our work: model weights, training data, code, and
logs.