MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model
  Series
Authors: name: Ge Zhang; name: Scott Qu; name: Jiaheng Liu; name: Chenchen Zhang; name: Chenghua Lin; name: Chou Leuang Yu; name: Danny Pan; name: Esther Cheng; name: Jie Liu; name: Qunshu Lin; name: Raven Yuan; name: Tuney Zheng; name: Wei Pang; name: Xinrun Du; name: Yiming Liang; name: Yinghao Ma; name: Yizhi Li; name: Ziyang Ma; name: Bill Lin; name: Emmanouil Benetos; name: Huan Yang; name: Junting Zhou; name: Kaijing Ma; name: Minghao Liu; name: Morry Niu; name: Noah Wang; name: Quehry Que; name: Ruibo Liu; name: Sine Liu; name: Shawn Guo; name: Soren Gao; name: Wangchunshu Zhou; name: Xinyue Zhang; name: Yizhi Zhou; name: Yubo Wang; name: Yuelin Bai; name: Yuhan Zhang; name: Yuxiang Zhang; name: Zenith Wang; name: Zhenzhu Yang; name: Zijian Zhao; name: Jiajun Zhang; name: Wanli Ouyang; name: Wenhao Huang; name: Wenhu Chen
Published: 2024-05-29 17:57:16

Abstract: Large Language Models (LLMs) have made great strides in recent years to
achieve unprecedented performance across different tasks. However, due to
commercial interest, the most competitive models like GPT, Gemini, and Claude
have been gated behind proprietary interfaces without disclosing the training
details. Recently, many institutions have open-sourced several strong LLMs like
LLaMA-3, comparable to existing closed-source LLMs. However, only the model's
weights are provided with most details (e.g., intermediate checkpoints,
pre-training corpus, and training code, etc.) being undisclosed. To improve the
transparency of LLMs, the research community has formed to open-source truly
open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training
corpus and training code) are being provided. These models have greatly
advanced the scientific study of these large models including their strengths,
weaknesses, biases and risks. However, we observe that the existing truly open
LLMs on reasoning, knowledge, and coding tasks are still inferior to existing
state-of-the-art LLMs with similar model sizes. To this end, we open-source
MAP-Neo, a highly capable and transparent bilingual language model with 7B
parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the
first fully open-sourced bilingual LLM with comparable performance compared to
existing state-of-the-art LLMs. Moreover, we open-source all details to
reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning
pipeline, checkpoints, and well-optimized training/evaluation framework are
provided. Finally, we hope our MAP-Neo will enhance and strengthen the open
research community and inspire more innovations and creativities to facilitate
the further improvements of LLMs.