DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code
  Intelligence
Authors: name: DeepSeek-AI; name: Qihao Zhu; name: Daya Guo; name: Zhihong Shao; name: Dejian Yang; name: Peiyi Wang; name: Runxin Xu; name: Y. Wu; name: Yukun Li; name: Huazuo Gao; name: Shirong Ma; name: Wangding Zeng; name: Xiao Bi; name: Zihui Gu; name: Hanwei Xu; name: Damai Dai; name: Kai Dong; name: Liyue Zhang; name: Yishi Piao; name: Zhibin Gou; name: Zhenda Xie; name: Zhewen Hao; name: Bingxuan Wang; name: Junxiao Song; name: Deli Chen; name: Xin Xie; name: Kang Guan; name: Yuxiang You; name: Aixin Liu; name: Qiushi Du; name: Wenjun Gao; name: Xuan Lu; name: Qinyu Chen; name: Yaohui Wang; name: Chengqi Deng; name: Jiashi Li; name: Chenggang Zhao; name: Chong Ruan; name: Fuli Luo; name: Wenfeng Liang
Published: 2024-06-17 13:51:35

Abstract: We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code
language model that achieves performance comparable to GPT4-Turbo in
code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained
from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion
tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially
enhances the coding and mathematical reasoning capabilities of DeepSeek-V2,
while maintaining comparable performance in general language tasks. Compared to
DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in
various aspects of code-related tasks, as well as reasoning and general
capabilities. Additionally, DeepSeek-Coder-V2 expands its support for
programming languages from 86 to 338, while extending the context length from
16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves
superior performance compared to closed-source models such as GPT4-Turbo,
Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.