DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code
  Intelligence
Authors: name: DeepSeek-AI, name: Qihao Zhu, name: Daya Guo, name: Zhihong Shao, name: Dejian Yang, name: Peiyi Wang, name: Runxin Xu, name: Y. Wu, name: Yukun Li, name: Huazuo Gao, name: Shirong Ma, name: Wangding Zeng, name: Xiao Bi, name: Zihui Gu, name: Hanwei Xu, name: Damai Dai, name: Kai Dong, name: Liyue Zhang, name: Yishi Piao, name: Zhibin Gou, name: Zhenda Xie, name: Zhewen Hao, name: Bingxuan Wang, name: Junxiao Song, name: Deli Chen, name: Xin Xie, name: Kang Guan, name: Yuxiang You, name: Aixin Liu, name: Qiushi Du, name: Wenjun Gao, name: Xuan Lu, name: Qinyu Chen, name: Yaohui Wang, name: Chengqi Deng, name: Jiashi Li, name: Chenggang Zhao, name: Chong Ruan, name: Fuli Luo, name: Wenfeng Liang
Organization: DeepSeek

We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code
language model that achieves performance comparable to GPT4-Turbo in
code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained
from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion
tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially
enhances the coding and mathematical reasoning capabilities of DeepSeek-V2,
while maintaining comparable performance in general language tasks. Compared to
DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in
various aspects of code-related tasks, as well as reasoning and general
capabilities. Additionally, DeepSeek-Coder-V2 expands its support for
programming languages from 86 to 338, while extending the context length from
16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves
superior performance compared to closed-source models such as GPT4-Turbo,
Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.