Baichuan 2: Open Large-scale Language Models
Authors: name: Aiyuan Yang, name: Bin Xiao, name: Bingning Wang, name: Borong Zhang, name: Chao Yin, name: Chenxu Lv, name: Da Pan, name: Dian Wang, name: Dong Yan, name: Fan Yang, name: Fei Deng, name: Feng Wang, name: Feng Liu, name: Guangwei Ai, name: Guosheng Dong Haizhou Zhao, name: Hang Xu, name: Haoze Sun, name: Hongda Zhang, name: Hui Liu, name: Jiaming Ji, name: Jian Xie, name: Juntao Dai, name: Kun Fang, name: Lei Su Liang Song, name: Lifeng Liu, name: Liyun Ru, name: Luyao Ma, name: Mang Wang, name: Mickel Liu, name: MingAn Lin, name: Nuolan Nie, name: Peidong Guo, name: Ruiyang Sun, name: Tao Zhang, name: Tianpeng Li, name: Tianyu Li, name: Wei Cheng, name: Weipeng Chen, name: Xiangrong Zeng, name: Xiaochuan Wang, name: Xiaoxi Chen, name: Xin Men, name: Xin Yu, name: Xuehai Pan, name: Yanjun Shen, name: Yiding Wang, name: Yiyu Li, name: Youxin Jiang, name: Yuchen Gao, name: Yupeng Zhang, name: Zenan Zhou, name: Zhiying Wu

Large language models (LLMs) have demonstrated remarkable performance on a
variety of natural language tasks based on just a few examples of natural
language instructions, reducing the need for extensive feature engineering.
However, most powerful LLMs are closed-source or limited in their capability
for languages other than English. In this technical report, we present Baichuan
2, a series of large-scale multilingual language models containing 7 billion
and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.
Baichuan 2 matches or outperforms other open-source models of similar size on
public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan
2 excels in vertical domains such as medicine and law. We will release all
pre-training model checkpoints to benefit the research community in better
understanding the training dynamics of Baichuan 2.