OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Authors: name: Siming Huang; name: Tianhao Cheng; name: Jason Klein Liu; name: Jiaran Hao; name: Liuyihan Song; name: Yang Xu; name: J. Yang; name: J. H. Liu; name: Chenchen Zhang; name: Linzheng Chai; name: Ruifeng Yuan; name: Zhaoxiang Zhang; name: Jie Fu; name: Qian Liu; name: Ge Zhang; name: Zili Wang; name: Yuan Qi; name: Yinghui Xu; name: Wei Chu
Published: 2024-11-07 17:47:25

Abstract: Large language models (LLMs) for code have become indispensable in various
domains, including code generation, reasoning tasks and agent systems.While
open-access code LLMs are increasingly approaching the performance levels of
proprietary models, high-quality code LLMs suitable for rigorous scientific
investigation, particularly those with reproducible data processing pipelines
and transparent training protocols, remain limited. The scarcity is due to
various challenges, including resource constraints, ethical considerations, and
the competitive advantages of keeping models advanced. To address the gap, we
introduce OpenCoder, a top-tier code LLM that not only achieves performance
comparable to leading models but also serves as an ``open cookbook'' for the
research community. Unlike most prior efforts, we release not only model
weights and inference code, but also the reproducible training data, complete
data processing pipeline, rigorous experimental ablation results, and detailed
training protocols for open scientific research. Through this comprehensive
release, we identify the key ingredients for building a top-tier code LLM: (1)
code optimized heuristic rules for data cleaning and methods for data
deduplication, (2) recall of text corpus related to code and (3) high-quality
synthetic data in both annealing and supervised fine-tuning stages. By offering
this level of openness, we aim to broaden access to all aspects of a top-tier
code LLM, with OpenCoder serving as both a powerful model and an open
foundation to accelerate research, and enable reproducible advancements in code
AI.