Baichuan 2: Open Large-scale Language Models
Authors: name: Aiyuan Yang; name: Bin Xiao; name: Bingning Wang; name: Borong Zhang; name: Chao Yin; name: Chenxu Lv; name: Da Pan; name: Dian Wang; name: Dong Yan; name: Fan Yang; name: Fei Deng; name: Feng Wang; name: Feng Liu; name: Guangwei Ai; name: Guosheng Dong Haizhou Zhao; name: Hang Xu; name: Haoze Sun; name: Hongda Zhang; name: Hui Liu; name: Jiaming Ji; name: Jian Xie; name: Juntao Dai; name: Kun Fang; name: Lei Su Liang Song; name: Lifeng Liu; name: Liyun Ru; name: Luyao Ma; name: Mang Wang; name: Mickel Liu; name: MingAn Lin; name: Nuolan Nie; name: Peidong Guo; name: Ruiyang Sun; name: Tao Zhang; name: Tianpeng Li; name: Tianyu Li; name: Wei Cheng; name: Weipeng Chen; name: Xiangrong Zeng; name: Xiaochuan Wang; name: Xiaoxi Chen; name: Xin Men; name: Xin Yu; name: Xuehai Pan; name: Yanjun Shen; name: Yiding Wang; name: Yiyu Li; name: Youxin Jiang; name: Yuchen Gao; name: Yupeng Zhang; name: Zenan Zhou; name: Zhiying Wu
Published: 2023-09-19 04:13:22

Abstract: Large language models (LLMs) have demonstrated remarkable performance on a
variety of natural language tasks based on just a few examples of natural
language instructions, reducing the need for extensive feature engineering.
However, most powerful LLMs are closed-source or limited in their capability
for languages other than English. In this technical report, we present Baichuan
2, a series of large-scale multilingual language models containing 7 billion
and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.
Baichuan 2 matches or outperforms other open-source models of similar size on
public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan
2 excels in vertical domains such as medicine and law. We will release all
pre-training model checkpoints to benefit the research community in better
understanding the training dynamics of Baichuan 2.