LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in
  Large Language Models
Authors: name: Nam V. Nguyen; name: Thong T. Doan; name: Luong Tran; name: Van Nguyen; name: Quang Pham
Published: 2024-11-01 14:04:36

Abstract: Mixture of Experts (MoEs) plays an important role in the development of more
efficient and effective large language models (LLMs). Due to the enormous
resource requirements, studying large scale MoE algorithms remain in-accessible
to many researchers. This work develops LibMoE, a comprehensive and
modular framework to streamline the research, training, and evaluation of MoE
algorithms. Built upon three core principles: (i) modular design, (ii)
efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs
more accessible to a wide range of researchers by standardizing the training
and evaluation pipelines. Using LibMoE, we extensively benchmarked five
state-of-the-art MoE algorithms over three different LLMs and 11 datasets under
the zero-shot setting. The results show that despite the unique
characteristics, all MoE algorithms perform roughly similar when averaged
across a wide range of tasks. With the modular design and extensive evaluation,
we believe LibMoE will be invaluable for researchers to make meaningful
progress towards the next generation of MoE and LLMs. Project page:
https://fsoft-aic.github.io/fsoft-LibMoE.github.io.