DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
Authors: name: DeepSeek-AI; name: Xiao Bi; name: Deli Chen; name: Guanting Chen; name: Shanhuang Chen; name: Damai Dai; name: Chengqi Deng; name: Honghui Ding; name: Kai Dong; name: Qiushi Du; name: Zhe Fu; name: Huazuo Gao; name: Kaige Gao; name: Wenjun Gao; name: Ruiqi Ge; name: Kang Guan; name: Daya Guo; name: Jianzhong Guo; name: Guangbo Hao; name: Zhewen Hao; name: Ying He; name: Wenjie Hu; name: Panpan Huang; name: Erhang Li; name: Guowei Li; name: Jiashi Li; name: Yao Li; name: Y. K. Li; name: Wenfeng Liang; name: Fangyun Lin; name: A. X. Liu; name: Bo Liu; name: Wen Liu; name: Xiaodong Liu; name: Xin Liu; name: Yiyuan Liu; name: Haoyu Lu; name: Shanghao Lu; name: Fuli Luo; name: Shirong Ma; name: Xiaotao Nie; name: Tian Pei; name: Yishi Piao; name: Junjie Qiu; name: Hui Qu; name: Tongzheng Ren; name: Zehui Ren; name: Chong Ruan; name: Zhangli Sha; name: Zhihong Shao; name: Junxiao Song; name: Xuecheng Su; name: Jingxiang Sun; name: Yaofeng Sun; name: Minghui Tang; name: Bingxuan Wang; name: Peiyi Wang; name: Shiyu Wang; name: Yaohui Wang; name: Yongji Wang; name: Tong Wu; name: Y. Wu; name: Xin Xie; name: Zhenda Xie; name: Ziwei Xie; name: Yiliang Xiong; name: Hanwei Xu; name: R. X. Xu; name: Yanhong Xu; name: Dejian Yang; name: Yuxiang You; name: Shuiping Yu; name: Xingkai Yu; name: B. Zhang; name: Haowei Zhang; name: Lecong Zhang; name: Liyue Zhang; name: Mingchuan Zhang; name: Minghua Zhang; name: Wentao Zhang; name: Yichao Zhang; name: Chenggang Zhao; name: Yao Zhao; name: Shangyan Zhou; name: Shunfeng Zhou; name: Qihao Zhu; name: Yuheng Zou
Published: 2024-01-05 18:59:13

Abstract: The rapid development of open-source large language models (LLMs) has been
truly remarkable. However, the scaling law described in previous literature
presents varying conclusions, which casts a dark cloud over scaling LLMs. We
delve into the study of scaling laws and present our distinctive findings that
facilitate scaling of large scale models in two commonly used open-source
configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
LLM, a project dedicated to advancing open-source language models with a
long-term perspective. To support the pre-training phase, we have developed a
dataset that currently consists of 2 trillion tokens and is continuously
expanding. We further conduct supervised fine-tuning (SFT) and Direct
Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
creation of DeepSeek Chat models. Our evaluation results demonstrate that
DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
the domains of code, mathematics, and reasoning. Furthermore, open-ended
evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
compared to GPT-3.5.