DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
Authors: name: DeepSeek-AI, name: Xiao Bi, name: Deli Chen, name: Guanting Chen, name: Shanhuang Chen, name: Damai Dai, name: Chengqi Deng, name: Honghui Ding, name: Kai Dong, name: Qiushi Du, name: Zhe Fu, name: Huazuo Gao, name: Kaige Gao, name: Wenjun Gao, name: Ruiqi Ge, name: Kang Guan, name: Daya Guo, name: Jianzhong Guo, name: Guangbo Hao, name: Zhewen Hao, name: Ying He, name: Wenjie Hu, name: Panpan Huang, name: Erhang Li, name: Guowei Li, name: Jiashi Li, name: Yao Li, name: Y. K. Li, name: Wenfeng Liang, name: Fangyun Lin, name: A. X. Liu, name: Bo Liu, name: Wen Liu, name: Xiaodong Liu, name: Xin Liu, name: Yiyuan Liu, name: Haoyu Lu, name: Shanghao Lu, name: Fuli Luo, name: Shirong Ma, name: Xiaotao Nie, name: Tian Pei, name: Yishi Piao, name: Junjie Qiu, name: Hui Qu, name: Tongzheng Ren, name: Zehui Ren, name: Chong Ruan, name: Zhangli Sha, name: Zhihong Shao, name: Junxiao Song, name: Xuecheng Su, name: Jingxiang Sun, name: Yaofeng Sun, name: Minghui Tang, name: Bingxuan Wang, name: Peiyi Wang, name: Shiyu Wang, name: Yaohui Wang, name: Yongji Wang, name: Tong Wu, name: Y. Wu, name: Xin Xie, name: Zhenda Xie, name: Ziwei Xie, name: Yiliang Xiong, name: Hanwei Xu, name: R. X. Xu, name: Yanhong Xu, name: Dejian Yang, name: Yuxiang You, name: Shuiping Yu, name: Xingkai Yu, name: B. Zhang, name: Haowei Zhang, name: Lecong Zhang, name: Liyue Zhang, name: Mingchuan Zhang, name: Minghua Zhang, name: Wentao Zhang, name: Yichao Zhang, name: Chenggang Zhao, name: Yao Zhao, name: Shangyan Zhou, name: Shunfeng Zhou, name: Qihao Zhu, name: Yuheng Zou
Organization: DeepSeek

The rapid development of open-source large language models (LLMs) has been
truly remarkable. However, the scaling law described in previous literature
presents varying conclusions, which casts a dark cloud over scaling LLMs. We
delve into the study of scaling laws and present our distinctive findings that
facilitate scaling of large scale models in two commonly used open-source
configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
LLM, a project dedicated to advancing open-source language models with a
long-term perspective. To support the pre-training phase, we have developed a
dataset that currently consists of 2 trillion tokens and is continuously
expanding. We further conduct supervised fine-tuning (SFT) and Direct
Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
creation of DeepSeek Chat models. Our evaluation results demonstrate that
DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
the domains of code, mathematics, and reasoning. Furthermore, open-ended
evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
compared to GPT-3.5.