OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Authors: Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, JIARAN HAO, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, Qian Liu, Houyi Li, Ge Zhang, Yuan Qi, Xu Yinghui, Wei Chu, Zili Wang
Accepted at: acl25

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

Siming Huang1,* Tianhao Cheng1,* Jason Klein Liu Weidi Xu4
Jiaran Hao4 Liuyihan Song4 Yang Xu4 Jian Yang5 Jiaheng Liu2
Chenchen Zhang5 Linzheng Chat3 Ruifeng Yuan Xianzhen Luo3
Qiufeng Wang Yuantao Fan Qingfeng Zhu3 Zhaoxiang Zhang6
Yang Gao2 Jie Fu Qian Liu Houyi Li1 Ge Zhang5
Yuan Qi4 Yinghui Xu1,4,† Wei Chu4,† Zili Wang4,5,†

1 Fudan University 2 Nanjing University
3 Harbin Institute of Technology
4 INF 5 M-A-P 6 CASIA

Abstract

Code LLMs have been widely used in various domains, including code generation, logical reasoning, and agent systems. However, open-access code LLMs mostly only release weights, lacking key features such as reproducible data pipelines and transparent training protocols, which are crucial for advancing deeper and more reliable investigations. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to industrial leading models but also serves as an “open cookbook” for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Our work identifies the key ingredients for building a top-tier code LLM are: language-specific filtering rules, file-level deduplication , high-quality synthetic data and two-stage supervised fine-tuning strategy. By offering high level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, enabling reproducible advancements in code intelligence. The released resource is available at https://opencoder-llm.github.io.

Figure 1: OpenCoder surpasses all previous fully open models (i.e., with open model weights and reproducible datasets) and other open-access models (i.e., with open model weights only) at the 6B+ parameter scale, pushing the frontier of fully open models to new heights.

1 Introduction

Large Language Models (LLMs) have achieved significant success in various domains (Wang et al., 2023; Que et al., 2024; Liu et al., 2024a,c; Wu et al., 2024), particularly in code-related tasks, revolutionizing the current paradigm of software development (Qian et al., 2024; Wang et al., 2024). Code-specific LLMs have emerged as a critical area within LLM research, with tools such as ChatGPT, Copilot, and Cursor reshaping the workflows of
developers. Despite this, the performance of open-source LLMs focused on code (Li et al., 2023b; Tao et al.; Lozhkov et al., 2024a) still falls short compared to state-of-the-art LLMs (Hui et al., 2024; Zhu et al., 2024), largely because these leading models keep their training datasets—an essential factor in LLM development—proprietary. This lack of transparency hinders the research community from establishing strong baselines and gaining deeper insights into top-tier code LLMs.

To remedy the gap, we set three primary goals by releasing OpenCoder and its development materials: (i) We aim to provide scholars with a meticulously curated and fully transparent strong baseline code LLM for research on mechanical interpretability and the data distribution of code LLMs. (ii) We intend to conduct in-depth investigations into the pretraining and instruction data curation pipeline for the development of stronger code LLMs. (iii) By enabling a detailed review of the development of the models, we hope to unlock more diverse customized solutions based on transparent code LLM. Through OpenCoder, we strive to stimulate and accelerate the growth of the open-source code LLM community.

Our comprehensive set of controlled experiments highlights key design choices for data curation for advanced code LLMs in different training stages. During pre-training Stage: (i) Effective data cleaning is crucial (Zhou et al., 2024), requiring well-designed heuristic rules to process large-scale corpora under limited resources and visualization to perceive data distribution. (ii) The impact of deduplication is significant, with file-level deduplication proving to be more effective than repository-level deduplication by maintaining data diversity and enhancing model performance on downstream tasks (Li et al., 2023b). (iii) The influence of GitHub stars is also examined, revealing that filtering data based on Github star count can possibly reduce data diversity and affect the overall data distribution, contributing to a suboptimal result (Allal et al., 2023). Moreover, in the annealing phase, high-quality data is crucial for further enhancing the model’s capabilities, indicating that data quality is more important than quantity in the later stages of model training. Finally, during instruction tuning phase, a two-stage instruction tuning strategy allows the model to acquire broad capabilities initially and then refine them with code-specific tasks, resulting in improved performance on both theoretical and practical coding tasks.

<table>
  <tr>
    <th>Model</th>
    <th>Pipe</th>
    <th>PT-data</th>
    <th>SFT-data</th>
    <th>Mid-ckpt</th>
    <th>Tokens</th>
    <th>HE</th>
  </tr>
  <tr>
    <th colspan="7">Open Model Weights & Reproducible Datasets</th>
  </tr>
  <tr>
    <td>OpenCoder-8B</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>2.5</td>
    <td>83</td>
  </tr>
  <tr>
    <td>StarCoder-15B</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>4.1</td>
    <td>72</td>
  </tr>
  <tr>
    <td>Crystal-7B</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>1.3</td>
    <td>34</td>
  </tr>
  <tr>
    <th colspan="7">Open Model Weights</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>2.5</td>
    <td>34</td>
  </tr>
  <tr>
    <td>CodeGemma-7B</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>6.5</td>
    <td>56</td>
  </tr>
  <tr>
    <td>DS-Coder-V2-Lite</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>10.2</td>
    <td>81</td>
  </tr>
  <tr>
    <td>Yi-Coder-9B</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>6.0</td>
    <td>85</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-7B</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>23.5</td>
    <td>88</td>
  </tr>
</table>

Table 1: Comparison of open-source resources among code LLMs. Pipe: pretraining data cleaning pipeline; PT-data: reproducible pretraining data; SFT-data: large-scale SFT corpus (>1M samples); Mid-ckpt: intermediate pretraining checkpoints; Tokens: total training tokens(B); HE: HumanEval scores for chat models.

Our contribution is summarized below:

• We present OpenCoder, a top-tier code Ilm archieving competitive performance with leading models across multiple benchmarks.

• We provide an full-stack open cookbook for code LLMs, including pipeline, training sets and middle checkpoints as detailed in Table 1.

• We identify the key ingredients for building a top-tier code LLM, including language-specific heuristic rules, file-level duplication, synthetic-data and two-stage SFT.

2 Pretraining Data

Pretraining data plays a crucial role in the development of LLMs, where the scale, quality, and diversity of the data greatly affect the model’s overall performance. To this end, we present how to process massive datasets with fine-grained heuristic rules under limited computational resources, and analyze the overall data distribution through visualization. This section will comprehensively illustrate the data processing strategies used in the general pretraining stage and the annealing stage.

2.1 RefineCode

Pretraining data forms the foundation for the capabilities of LLM. While The Stack v2 (Lozhkov et al., 2024a) has been a valuable resource for training code LLMs in the open-source community, its quality is insufficient for top-tier model performance.
To address this, we introduce RefineCode, a high-quality, reproducible dataset of 960 billion tokens across 607 programming languages, comprising raw code and code-related web data. RefineCode is the first pre-training code dataset that performs language-specific refinement, employing customized cleaning thresholds and domain-adapted rules for each programming language. A comparison between RefineCode and all versions of The Stack is provided in Table 2.

<table>
  <tr>
    <th></th>
    <th># Total</th>
    <th># Web</th>
    <th># Programs</th>
    <th># Rules</th>
    <th>LS Rules</th>
  </tr>
  <tr>
    <td>The Stack v1</td>
    <td>200 B</td>
    <td>\</td>
    <td>88</td>
    <td>-15</td>
    <td>x</td>
  </tr>
  <tr>
    <td>The Stack v2</td>
    <td>900 B</td>
    <td>-30 B</td>
    <td>619</td>
    <td>-15</td>
    <td>x</td>
  </tr>
  <tr>
    <td><b>RefineCode</b></td>
    <td>960 B</td>
    <td>-75 B</td>
    <td>607</td>
    <td>-130</td>
    <td>✓</td>
  </tr>
</table>

Table 2: Comparison of training data between RefineCode and The Stack series. “Total” represents the total tokens, “Web” indicates tokens from web-related texts. “Programs” refers to programming languages, “Rules” denotes filtering rules applied, and “LS Rules” represents the language-specific filtering rules.

2.1.1 Raw Code
We collect raw code primarily from GitHub repositories up to November 2023 and non-GitHub data from The Stack v2. To ensure the curation of high-quality raw code data, we have developed the code-specific data processing pipeline including modules of preprocessing, deduplication, transformation, filtering, and data sampling. We briefly outline the pipeline, with additional details provided in Appendix A.

Preprocessing To optimize resources, we exclude files larger than 8 MB, as they are typically non-text and resource-intensive. We then filter for files related to programming languages based on their extensions, as defined by linguist (Linguist, 2024), and discard those with low capacity or quality. This results in a final selection of 607 distinct programming language file types.

Deduplication Deduplication is a crucial module in the data pipeline to enhance both pretraining efficiency and efficacy (Lee et al., 2021). We first perform exact deduplication using SHA256 to eliminate fully duplicate files, then apply fuzzy deduplication. For the latter, we use MinHash (Broder, 1997) and LSH (Leskovec et al., 2014) to remove near-identical files.

Transformation To address minor issues without discarding entire files, we apply two transformation rules before filtering: (1) we remove repetitive and irrelevant copyright notices from the beginning of over 15% of code files; and (2) to mitigate privacy risks, we detect and replace Personally Identifiable Information (PII)—such as passwords and emails—with placeholders like “<names>” and “<password>” using regular expressions.

Filtering The quality of code files on GitHub varies significantly, with lower-quality code potentially hindering LLM pretraining. We propose the first heuristic filtering framework tailored to code pretraining data by considering the unique characteristics of different programming languages. This framework provides over 130 heuristic rules with customized weight assignments across three categories, resulting in more precise and higher-quality data cleansing. Detailed heuristic rules and high-level design principles are provided in Appendix B.

Data Sampling Structured data formats with specific syntax (e.g., JSON/HTML) present a unique challenge in code pretraining. While these formats represent a substantial portion of publicly available code corpora, their disproportionate representation in pre-training data may paradoxically undermine model generalization capacity through pattern memorization. Similar to modern code processing methods (Lozhkov et al., 2024a,b) we perform downsampling on programming languages (e.g., HTML, Java).

2.1.2 Code-Related Web Data
Inspired by DeepSeekMath (Shao et al., 2024), we collect high-quality code-related data corpus from the Common Crawl dataset, Finweb (Penedo et al., 2024), skyPile (Wei et al., 2023a) and web part of
<table>
  <tr>
    <th>Category</th>
    <th>Source</th>
    <th># Tokens</th>
    <th>Per.</th>
  </tr>
  <tr>
    <td rowspan="3">Raw Code</td>
    <td>Github</td>
    <td>755 B</td>
    <td>78.4%</td>
  </tr>
  <tr>
    <td>Jupyter Notes</td>
    <td>11 B</td>
    <td>1.1%</td>
  </tr>
  <tr>
    <td>The Stack v2</td>
    <td>120 B</td>
    <td>12.5%</td>
  </tr>
  <tr>
    <td rowspan="3">Code-related Web</td>
    <td>CC</td>
    <td>13 B</td>
    <td>1.4%</td>
  </tr>
  <tr>
    <td>SkyPile</td>
    <td>3 B</td>
    <td>0.3%</td>
  </tr>
  <tr>
    <td>FineWeb</td>
    <td>55 B</td>
    <td>5.7%</td>
  </tr>
  <tr>
    <td>OpenSource</td>
    <td>AutoMathText</td>
    <td>3 B</td>
    <td>0.3%</td>
  </tr>
</table>

Table 3: The Composition of RefineCode.

AutoMathText Dataset (Zhang et al., 2024b). Due to the lack of open-source fine-gained code corpus, we first annotate 500k high-quality code-like data from CommonCrawl using the Autonomous Data Selection (Zhang et al., 2024b) method as seed.

2.2 Annealing Data

Following the training strategy in MiniCPM (Hu et al., 2024), our model undergoes a rapid learning rate annealing phase after the general pretraining stage, where very high-quality training data is used to further enhance the model’s capabilities. In addition to the RefineCode from the original distribution, we further incorporated the Algorithmic Corpus and synthetic data during the annealing phase.

RefineCode During the annealing stage, we maintain distribution consistency with the pretraining phase to prevent catastrophic forgetting (Hu et al., 2024; Shen et al., 2024). 84% of the annealing data is drawn from the original RefineCode.

Algorithmic Corpus Algorithmic code files exhibit strong code logic and minimal dependency on external files, demonstrating excellent self-containment. They align well with the smaller, independent tasks typical of real-world interactive scenarios. Therefore, we extract a subset of the pretraining data containing keywords like "def solution" or "class solution" to construct this corpus. We also use model-based method, the results in section shows that rule is better.

High Quality Code Snippet Inspired by the synthetic CodeExercises dataset in Gunasekar et al. (2023), we utilized the algorithmic corpus as seeds and employed LLM to synthesize self-contained independent functions along with corresponding test cases. We only retained the data that successfully passed the test cases. We extend this pipeline to support multiple program languages.

Code Textbook Pretraining data with a clear semantic mapping between code and natural language is scarce. (Song et al., 2024). To address this issue, we utilize a powerful LLM to extract and elaborate on abstract code knowledge from high-quality datasets like HQCode (Yuxiang630, 2024). This approach is designed to help the model learn code from diverse perspectives.

<table>
  <tr>
    <th>Category</th>
    <th>Dataset</th>
    <th># Token</th>
  </tr>
  <tr>
    <td rowspan="2">Original Data</td>
    <td>RefineCode</td>
    <td>83.94 B</td>
  </tr>
  <tr>
    <td>Algorithmic Corpus</td>
    <td>12.44 B</td>
  </tr>
  <tr>
    <td rowspan="2">Synthetic Data</td>
    <td>High Quality Code Snippet</td>
    <td>2.71 B</td>
  </tr>
  <tr>
    <td>Code Textbooks</td>
    <td>0.91 B</td>
  </tr>
</table>

Table 4: Detailed data mixture for annealing data.

2.3 Visual Inspection

The pretraining data processing pipeline (e.g., deduplication, filtering) involves numerous hyperparameters, making ablation studies for each economically infeasible. Instead, we use PCA to visualize embeddings extracted from CodeBERT (Feng et al., 2020) and perform spot checks on outliers, providing an effective way to understand the distribution of cleaned pretraining data.

Interestingly, visualization reveals the quality gap between RefineCode and The Stack v2 even before pretraining. As shown in Figure 3, The Stack V2 data shows a greater number of outliers, while the embeddings of RefineCode appear more tightly clustered. Besides, after analyzing the outlier data, we observe the outliers usually show many low-quality patterns, such as pure text comments, hexadecimal-only data, and excessively short code lacking computational logic, which can distort the distribution of the pretraining dataset and ultimately hurt the efficiency of pretraining.

3 Pretraining

3.1 Model Architecture

OpenCoder follows the architecture of LLaMA 3 (Dubey et al., 2024) and is in two sizes: 1.5B and 8B parameters. The 1.5B model features 24 layers, a hidden size of 2240, and 14 attention heads, with a context window size of 4096. The 8B model has 32 layers, a hidden size of 4096, 32 attention heads, and grouped query attention with 8 key-value heads. Both models employ SwiGLU (Shazeer, 2020) and
utilize the tokenizer proposed by INF-Team (2024). Detailed configurations are presented in Table 5.

<table>
  <tr>
    <th></th>
    <th>1.5B</th>
    <th>8B</th>
  </tr>
  <tr>
    <td>Layers</td>
    <td>24</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Model Dimension</td>
    <td>2240</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Attention Heads</td>
    <td>14</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Key / Value Heads</td>
    <td>14</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Activation Function</td>
    <td colspan="2">SwiGLU</td>
  </tr>
  <tr>
    <td>Vocab Size</td>
    <td colspan="2">96640</td>
  </tr>
  <tr>
    <td>Positional Embedding (\( \theta \))</td>
    <td>10,000</td>
    <td>500,000</td>
  </tr>
  <tr>
    <td>Context Window Size</td>
    <td>4096</td>
    <td>8192</td>
  </tr>
</table>

Table 5: Overview of the key hyperparameters of OpenCoder, including 1.5B and 8B. RoPE is selected for Positional Embedding.

3.2 Training Details

Optimizer Both models employ the WSD learning schedule (Hu et al., 2024). The schedule include a warm-up phase of 2,000 steps over 8B tokens, followed by a peak learning rate of 3e-4, which remained constant after the warm-up. During the final 100B token annealing phase, the learning rate decayed exponentially from 3e-4 to 1e-5.

Training Framework The training for both models was conducted using Megatron-LM (Shoeybi et al., 2020) with distributed optimization and DDP gradient overlap. The 1.5B model was trained on 2 trillion tokens from scratch with a sequence length of 4096, a micro-batch size of 4, and a global batch size of 1024. The training process was conducted on a cluster of 256 H800 GPUs over a duration of 109.5 hours, totaling 28,034 GPU hours. For the 8B model, training was performed on 2.5 trillion tokens from scratch with a sequence length of 8192, a micro-batch size of 1, tensor parallelism (TP) of 2, and a global batch size of 1024. This training was executed on 512 H100 GPUs over 187.5 hours, resulting in a total of 96,000 GPU hours.

4 Post Training

We constructed a diverse dataset of over 1 million instructions to fine-tune opencoder. In addition to open-source data, we employ multiple synthetic approaches to construct code instruction datasets. Full details are provided in appendix H.

4.1 Data Collection

Open-source Training Data We curate a collection of high-quality open-source code instruction datasets, including Evol-Instruct (Luo et al., 2024), Infinity-Instruct (BAAI, 2024), and McEval (Chai et al., 2024). Furthermore, we extract real user queries from WildChat (Zhao et al., 2024) and Code-290k-ShareGPT (Computations, 2023), and employ LLM to identify code-related dialogue histories, followed by rigorous data cleaning. The resulting dataset, termed RealUser-Instruct, not only demonstrates high diversity but also closely mirrors real-world problem complexity, aligning to authentic scenarios.

Verified Instruction Synthesis Following oss-instruct (Wei et al., 2023b), we utilizes raw code as initial seed to generate question-answer pairs. To further enhance code quality, we implement a rigorous verification process through test case execution. Our methodology involves: (1) sampling high-quality code from RefineCode; (2) using a teacher model to generate question-answer pairs with chain-of-thought reasoning and multiple test cases for code segments; (3) executing these test cases using code interpreter. (4) retain codes passing over 80% test executions.

Package Instruction Synthesis Package instruction data are generated to further enhance proficiency of code LLM with common packages. Our methodology involves a four-step process: (1) extracting high-quality code snippets using packages from RefineCode; (2) retrieving the corresponding API documentation and usage guidelines from PyDoc; (3) prompting a teacher model to generate QA data based on the snippets and documentation; (4) prompting a strong model to verify if the QA data adhere to the API usage guidelines. While used for Python data, it is important to clarify that this method generalizes easily to other languages via their API references.

![Visualization on the python of RefineCode and The Stack v2.](page_120_120_600_400.png)

Figure 3: Visualization on the python of RefineCode and The Stack v2.
Large-scale Diverse Instruction Synthesis  Pre-training web corpus contains a vast and diverse collection of reasoning data (Yue et al., 2024). We develop a large-scale instruction framework comprising four key components: (1) Context refinement, where an LLM filters irrelevant web content and extracts meaningful sentences as question seeds; (2) Task specification, which defines programming languages, difficulty levels, and task types through a configurable module, with prompt engineering generating diverse, context-rich templates; (3) Content generation, where an advanced LLM produces both questions and corresponding answers, validated through automated code execution and unit testing; (4) Response refinement, where an LLM enhances outputs with code comments and detailed explanations.

4.2  Two-Stage Training Strategy

To develop a language model proficient in both theoretical and practical aspects of computer science, we implement a two-stage instruction fine-tuning process. In the first stage, we enhance the model’s theoretical understanding with a comprehensive and diverse set of domain-specific question-answer pairs. This stage covers a wide range of topics, including algorithms, and data structures, enabling the model to provide accurate responses to complex theoretical queries that span both computer science theory and real-world user scenarios. The second stage is more focused, concentrating on practical downstream tasks by refining the model’s code generation and error correction capabilities to ensure strong performance in real-world applications.

<table>
  <tr>
    <th>Stage</th>
    <th>Data Source</th>
    <th># Examples</th>
  </tr>
  <tr>
    <td rowspan="3">Stage1</td>
    <td>RealUser-Instruct</td>
    <td>0.7 M</td>
  </tr>
  <tr>
    <td>Large-scale Instruct</td>
    <td>2.3 M</td>
  </tr>
  <tr>
    <td>Infinity-Instruct</td>
    <td>1.0 M</td>
  </tr>
  <tr>
    <td rowspan="4">Stage2</td>
    <td>McEval-Instruct</td>
    <td>36 K</td>
  </tr>
  <tr>
    <td>Evol-Instruct</td>
    <td>111 K</td>
  </tr>
  <tr>
    <td>Verified-Instruct</td>
    <td>110 K</td>
  </tr>
  <tr>
    <td>Package-Instruct</td>
    <td>110 K</td>
  </tr>
</table>

Table 6: Detailed data composition of Two-Stage SFT.

5  Experimental Results

This section contains evaluations led to OpenCoder with many more multilingual evaluation details provided in appendix G.

Base Evaluation  In Table 7 we benchmark OpenCoder-base series on common downstream tasks. We find that OpenCoder series achieve the best performance among fully open models (green lines), pushing the frontier of open-source code llm. On the widely-used code benchmarks HumanEval(+) and MBPP(+), OpenCoder-base achieve state-of-the-art performance, surpassing leading industrial code llms.

Chat Evaluation  In Table 8 we benchmark OpenCoder-instruct series on common tasks. OpenCoder-1.5B-instruct maintains leading performance among all 1.5B models. In BigCodeBench, a benchmark reflecting the comprehensive capabilities of code llms, OpenCoder shows strong performance. OpenCoder-8B was trained on a total of 2.5 T tokens, significantly fewer than industrial codellm (e.g., Yi-Coder-9B: 6.0T; Qwen2.5-Coder-7B: 23.5T). At 8B scale, OpenCoder’s performance remains in the top tier and maintains a leading position among fully open-source models.

6  Ablation Study

File-level deduplication outperforms repo-level deduplication for code corpus  Data deduplication improves efficiency and reduces overfitting, with both file-level and repo-level methods being applied for code data (Lee et al., 2021; Lozhkov et al., 2024a; Guo et al., 2024). We conduct a detailed comparison of deduplication levels, with experiments details in Appendix C and key findings listed below: (i) File-level deduplication leads to better training efficiency despite more aggressive
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="2">HumanEval</th>
    <th colspan="3">MBPP</th>
    <th colspan="2">BigCodeBench</th>
  </tr>
  <tr>
    <th>HE</th>
    <th>HE+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
    <th>3-shot</th>
    <th>Full</th>
    <th>Hard</th>
  </tr>
  <tr>
    <th colspan="9">1B+ Models</th>
  </tr>
  <tr>
    <td>DeepSeek-Coder-1.3B-Base</td>
    <td>1.3B</td>
    <td>34.8</td>
    <td>26.8</td>
    <td>55.6</td>
    <td>46.9</td>
    <td>46.2</td>
    <td>26.1</td>
    <td>3.4</td>
  </tr>
  <tr>
    <td>Yi-Coder-1.5B</td>
    <td>1.5B</td>
    <td>41.5</td>
    <td>32.9</td>
    <td>27.0</td>
    <td>22.2</td>
    <td>51.6</td>
    <td>23.5</td>
    <td>3.4</td>
  </tr>
  <tr>
    <td>CodeGemma-2B</td>
    <td>2B</td>
    <td>31.1</td>
    <td>16.5</td>
    <td>51.1</td>
    <td>43.1</td>
    <td>45.4</td>
    <td>23.9</td>
    <td>7.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-1.5B</td>
    <td>1.5B</td>
    <td>43.9</td>
    <td>36.6</td>
    <td>69.2</td>
    <td>58.6</td>
    <td><b>59.2</b></td>
    <td><b>34.6</b></td>
    <td><b>9.5</b></td>
  </tr>
  <tr>
    <td>StarCoder2-3B</td>
    <td>3B</td>
    <td>31.7</td>
    <td>27.4</td>
    <td>60.2</td>
    <td>49.1</td>
    <td>46.4</td>
    <td>21.4</td>
    <td>4.7</td>
  </tr>
  <tr>
    <td>OpenCoder-1.5B-Base</td>
    <td>1.5B</td>
    <td><b>54.3</b></td>
    <td><b>49.4</b></td>
    <td><b>70.6</b></td>
    <td><b>58.7</b></td>
    <td>51.8</td>
    <td>24.5</td>
    <td>5.4</td>
  </tr>
  <tr>
    <th colspan="9">6B+ Models</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>7B</td>
    <td>33.5</td>
    <td>26.2</td>
    <td>55.3</td>
    <td>46.8</td>
    <td>41.4</td>
    <td>28.7</td>
    <td>5.4</td>
  </tr>
  <tr>
    <td>CodeGemma-7B</td>
    <td>7B</td>
    <td>39.0</td>
    <td>32.3</td>
    <td>50.5</td>
    <td>40.7</td>
    <td>55.0</td>
    <td>38.3</td>
    <td>10.1</td>
  </tr>
  <tr>
    <td>DS-Coder-6.7B-Base</td>
    <td>6.7B</td>
    <td>47.6</td>
    <td>39.6</td>
    <td>70.2</td>
    <td>56.6</td>
    <td>60.6</td>
    <td>41.1</td>
    <td>11.5</td>
  </tr>
  <tr>
    <td>DS-Coder-V2-Lite-Base (MoE)</td>
    <td>16B</td>
    <td>40.9</td>
    <td>34.1</td>
    <td>71.9</td>
    <td>59.4</td>
    <td>62.6</td>
    <td>30.6</td>
    <td>8.1</td>
  </tr>
  <tr>
    <td>CodeQwen1.5-7B</td>
    <td>7B</td>
    <td>51.8</td>
    <td>45.7</td>
    <td>72.2</td>
    <td>60.2</td>
    <td>61.8</td>
    <td>45.6</td>
    <td>15.6</td>
  </tr>
  <tr>
    <td>Yi-Coder-9B</td>
    <td>9B</td>
    <td>53.7</td>
    <td>46.3</td>
    <td>48.4</td>
    <td>40.7</td>
    <td><b>69.4</b></td>
    <td>42.9</td>
    <td>14.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-7B</td>
    <td>7B</td>
    <td>61.6</td>
    <td>53.0</td>
    <td>76.9</td>
    <td>62.9</td>
    <td>68.8</td>
    <td><b>45.8</b></td>
    <td><b>16.2</b></td>
  </tr>
  <tr>
    <td>Crystal-7B</td>
    <td>7B</td>
    <td>22.6</td>
    <td>20.7</td>
    <td>38.6</td>
    <td>31.7</td>
    <td>31.0</td>
    <td>10.8</td>
    <td>4.1</td>
  </tr>
  <tr>
    <td>StarCoder2-7B</td>
    <td>7B</td>
    <td>35.4</td>
    <td>29.9</td>
    <td>54.4</td>
    <td>45.6</td>
    <td>55.2</td>
    <td>27.7</td>
    <td>8.8</td>
  </tr>
  <tr>
    <td>StarCoder2-15B</td>
    <td>15B</td>
    <td>46.3</td>
    <td>37.8</td>
    <td>66.2</td>
    <td>53.1</td>
    <td>15.2</td>
    <td>38.4</td>
    <td>12.2</td>
  </tr>
  <tr>
    <td>OpenCoder-8B-Base</td>
    <td>8B</td>
    <td><b>66.5</b></td>
    <td><b>63.4</b></td>
    <td><b>79.9</b></td>
    <td><b>70.4</b></td>
    <td>60.6</td>
    <td>40.5</td>
    <td>9.5</td>
  </tr>
</table>

Table 7: Performance of various base models on HumanEval, MBPP, and the “complete” task of BigCodeBench. Models trained on reproducible datasets are marked with green.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench</th>
    <th rowspan="2">LiveCodeBench Avg</th>
  </tr>
  <tr>
    <th>HE</th>
    <th>HE+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
    <th>Full</th>
    <th>Hard</th>
  </tr>
  <tr>
    <th colspan="9">1B+ Models</th>
  </tr>
  <tr>
    <td>DS-coder-1.3B-Instruct</td>
    <td>1.3B</td>
    <td>65.2</td>
    <td>61.6</td>
    <td>61.6</td>
    <td>52.6</td>
    <td>22.8</td>
    <td>3.4</td>
    <td>9.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-1.5B-Instruct</td>
    <td>1.5B</td>
    <td>70.7</td>
    <td>66.5</td>
    <td>69.2</td>
    <td>59.4</td>
    <td>32.5</td>
    <td>6.8</td>
    <td><b>15.7</b></td>
  </tr>
  <tr>
    <td>Yi-Coder-1.5B-Chat</td>
    <td>1.5B</td>
    <td>67.7</td>
    <td>63.4</td>
    <td>68.0</td>
    <td>59.0</td>
    <td>24.0</td>
    <td>6.8</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>OpenCoder-1.5B-Instruct</td>
    <td>1.5B</td>
    <td><b>72.5</b></td>
    <td><b>67.7</b></td>
    <td><b>72.7</b></td>
    <td><b>61.9</b></td>
    <td><b>34.6</b></td>
    <td><b>11.5</b></td>
    <td>12.8</td>
  </tr>
  <tr>
    <th colspan="9">6B+ Models</th>
  </tr>
  <tr>
    <td>DS-Coder-V2-Lite-Instruct</td>
    <td>16B</td>
    <td>81.1</td>
    <td>75.0</td>
    <td>82.3</td>
    <td>68.8</td>
    <td>36.8</td>
    <td>16.2</td>
    <td>24.3</td>
  </tr>
  <tr>
    <td>CodeLlama-7B-Instruct</td>
    <td>7B</td>
    <td>45.7</td>
    <td>39.6</td>
    <td>39.9</td>
    <td>33.6</td>
    <td>21.9</td>
    <td>3.4</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>CodeGemma-7B-It</td>
    <td>7B</td>
    <td>59.8</td>
    <td>47.0</td>
    <td>69.8</td>
    <td>59.0</td>
    <td>32.3</td>
    <td>7.4</td>
    <td>14.7</td>
  </tr>
  <tr>
    <td>DS-Coder-6.7B-Instruct</td>
    <td>6.7B</td>
    <td>78.6</td>
    <td>70.7</td>
    <td>75.1</td>
    <td>66.1</td>
    <td>35.5</td>
    <td>10.1</td>
    <td>20.5</td>
  </tr>
  <tr>
    <td>Yi-Coder-9B-Chat</td>
    <td>9B</td>
    <td>82.3</td>
    <td>72.6</td>
    <td>81.5</td>
    <td>69.3</td>
    <td>38.1</td>
    <td>11.5</td>
    <td>23.4</td>
  </tr>
  <tr>
    <td>CodeQwen1.5-7B-Chat</td>
    <td>7B</td>
    <td>86.0</td>
    <td>79.3</td>
    <td>83.3</td>
    <td>71.4</td>
    <td>39.6</td>
    <td><b>18.9</b></td>
    <td>20.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-7B-Instruct</td>
    <td>7B</td>
    <td><b>88.4</b></td>
    <td><b>84.1</b></td>
    <td><b>83.5</b></td>
    <td><b>71.7</b></td>
    <td><b>41.0</b></td>
    <td>18.2</td>
    <td><b>37.6</b></td>
  </tr>
  <tr>
    <td>CrystalChat-7B</td>
    <td>7B</td>
    <td>34.1</td>
    <td>31.7</td>
    <td>39.1</td>
    <td>32.7</td>
    <td>26.7</td>
    <td>2.3</td>
    <td>6.1</td>
  </tr>
  <tr>
    <td>StarCoder2-15B-Instruct-v0.1</td>
    <td>15B</td>
    <td>72.6</td>
    <td>63.4</td>
    <td>75.2</td>
    <td>61.2</td>
    <td>37.6</td>
    <td>12.2</td>
    <td>20.4</td>
  </tr>
  <tr>
    <td>OpenCoder-8B-Instruct</td>
    <td>8B</td>
    <td>83.5</td>
    <td>78.7</td>
    <td>79.1</td>
    <td>69.0</td>
    <td>42.9</td>
    <td>16.9</td>
    <td>23.2</td>
  </tr>
</table>

Table 8: Performance of various chat models on HumanEval, MBPP, the “instruct” task of BigCodeBench and LiveCodeBench. Models trained on reproducible datasets are marked with green.

token reduction. File-level dedup retains only a third of the tokens compared to repo-level deduplication but results in higher training efficiency, as shown in Table 9. (ii) Repo-level dedup leaves high redundancy. With further analysis of repo-level dedup results, we find that 52B tokens (52%) exhibits complete character-level equivalence with another file. When conducting file-level dedup as post-processing step, 68B tokens (68%) could be further deduplicated. The performance trending can be found in Figure 5(a).
<table>
  <tr>
    <th>Dedup Level</th>
    <th>Token(ratio)</th>
    <th>HE</th>
    <th>MBPP</th>
  </tr>
  <tr>
    <td>File-Level</td>
    <td>32.7 (2.4%)</td>
    <td>18.9</td>
    <td>19.4</td>
  </tr>
  <tr>
    <td>Repo-Level</td>
    <td>99.5 (7.3%)</td>
    <td>17.0</td>
    <td>13.6</td>
  </tr>
</table>

Table 9: Token counts and benchmark results using different deduplication strategies on RefineCode Python(1364B). HE/MBPP are obtained by training a 1.5B model for one epoch on all deduplicated tokens.

<table>
  <tr>
    <th></th>
    <th>HE(+)</th>
    <th>MBPP(+)</th>
    <th>BCB</th>
    <th>Arena</th>
  </tr>
  <tr>
    <td>S1</td>
    <td>52.4(48.1)</td>
    <td>68.7(57.4)</td>
    <td>22.1</td>
    <td>5.8</td>
  </tr>
  <tr>
    <td>S2</td>
    <td>69.1(64.0)</td>
    <td>69.5(60.3)</td>
    <td>32.6</td>
    <td>5.3</td>
  </tr>
  <tr>
    <td>S1+S2</td>
    <td><b>72.5(67.7)</b></td>
    <td><b>72.7(61.9)</b></td>
    <td><b>34.6</b></td>
    <td><b>6.9</b></td>
  </tr>
  <tr>
    <td>Mix</td>
    <td>55.5(51.2)</td>
    <td>52.0(58.7)</td>
    <td>23.9</td>
    <td>3.8</td>
  </tr>
</table>

Table 10: Performance of different training strategies across benchmarks. Mix Training refers to the process of combining and shuffling the data from Stage 1 and Stage 2 for joint training.

High-quality code data in annealing significantly boosts performance We compared the impact of high-quality data (the Algorithmic Corpus and Synthetic Data) during the annealing phase. From Figure 5(b), we observe that the performance drops a lot when the high-quality training data is removed, which demonstrates the effectiveness of our constructed high-quality data in the annealing phase.

GitHub star filtering limits diversity, reducing effectiveness Github stars deteriorate performance (Allal et al., 2023). We further validate this conclusion and provide analysis from a visualization perspective. Specifically, we train two 1.5B LLMs, where one is trained original data and another is trained by data filtered by stars (stars>=5). As shown in Figure 5(c), star filter leads to decreased performance. We attribute this performance decline to the star filter’s potential reduction of data diversity. As dedicated in Figure 5(d) and Figure 5(e), data applied star filter reflects a lower training loss and a more concentrated distribution, indicating that star filter significantly compromises data diversity. Upon closer examination of the filtered data, we find that it still contains a considerable amount of well-structured, algorithmically rich code. Therefore, we argue that using stars as a filtering criterion is not an optimal choice.

Two-Stage SFT Strategy: First Broaden Knowledge, Then Sharpen Skills Stage1 data exhibits significant diversity, while stage2 data shows higher quality. We believe this two-stage strategy enables the acquisition of broad capabilities in Stage 1, followed by targeted enhancement of code-related tasks in Stage 2. Besides, separating high-quality data can also prevent the gradient dilution problem that occurs when mixing datasets, maximizing the utilization of high-quality data. As shown in table 10, two-stage SFT strategy can bring consistent improvement in both public benchmarks and real-word scenarios.

7 Related Work

Open Large Language Models. Recently, numerous open-sourced LLMs, such as LLaMA (Touvron et al., 2023), Mistral (Jiang et al., 2023), and Qwen (Bai et al., 2023), have empowered the research community, fostering innovation. Open datasets like RedPajama (Computer, 2023) and SlimPajama (Soboleva et al., 2023), Map-Neo (Zhang et al., 2024a) alongside chat datasets such as WildChat (Zhao et al., 2024), further accelerate LLM advancements. Notably, fully open LLMs like OLMo (Groeneveld et al., 2024), OLMoE (Muenninghoff et al., 2024b), and LLM360 (Liu et al., 2023b) provide comprehensive reproduction details, including data pipelines and checkpoints. In the realm of code LLMs, StarCoder (Allal et al., 2023) and StarCoderV2 (Lozhkov et al., 2024a) share their data pipelines and high-quality pretraining corpora.

Code Large Language Models. The rapid advancement of generative language models has led to significant progress in AI-assisted software engineering (Radford et al., 2019; Brown et al., 2020; Touvron et al., 2023; Sun et al., 2024). Proprietary models (Achiam et al., 2023; Chen et al., 2021; Chowdhery et al., 2023) have achieved strong performance across various code-related benchmarks (Chen et al., 2021; Hendrycks et al., 2020), yet the lack of accessible model weights limits reproducibility and downstream innovation. In response, the community has introduced several open-source alternatives, such as CodeGen (Nijkamp et al., 2023b,a), StarCoder (Li et al., 2023b; Lozhkov et al., 2024b), CodeLlama (Roziere et al., 2023), and DeepSeekCoder (Guo et al., 2024; Zhu et al., 2024), which support broader experimentation and development.

Code Benchmarks. Many benchmark datasets have been proposed to comprehensively assess code large language models, covering diverse tasks such as code generation (Chen et al., 2021; Austin
Figure 5: (a-c) shows 1.5B Code LLM performance across applying different ablation settings. (d) shows the comparison of training loss using star-based data filtering or not. (e) shows the distribution of data processed star-based data filtering or not.

et al., 2021; Athiwaratkun et al., 2023; Gu et al., 2024; Lai et al., 2023; Chai et al., 2024; Muenninghoff et al., 2024a), code retrieval (Husain et al., 2019; Lu et al., 2021), translation (Yan et al., 2023), and efficiency (Du et al., 2024). Particularly, repository-level code completion has emerged as a challenging yet critical benchmark (Liu et al., 2023a; Shrivastava et al., 2023; Zhang et al., 2023; Deng et al., 2024; Liu et al., 2024b).

8 Conclusion

In this paper, we present OpenCoder, an open LLM specialized in code intelligence that achieves top-tier performance. To advance research transparency and reproducibility, we release our complete training materials, including: the complete data processing pipeline, the reproducible pretraining dataset, the synthetic dataset, the open code SFT dataset, rigorous experimental ablation results, detailed training protocols and intermediate checkpoints. The performance of OpenCoder is on par with leading proprietary models, and it surpasses most previous open-source models at the 1B+ and 6B+ parameter scale. We hope the release of OpenCoder can democratize access to all aspects of a top-tier code LLM, serving as both a powerful model and an open foundation to accelerate research and enable reproducible advancements in code AI.

9 Limitations

Although OpenCoder has explored the entire workflow for building a code LLM, our project still has several limitations. First, during the pretraining data filtering process, we focused exclusively on developing rule-based filtering rules but did not explore model-based filtering approaches. In addition, due to resource constraints, we only utilized raw code and code-related text data in the training process, without considering the potential promotion of general natural language data on the performance of code LLMs. Finally, in the post-training phase, we performed only supervised fine-tuning (SFT) on the base model, without considering reinforcement learning from human feedback (RLHF) for better alignment with human preferences.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayamal Anadkat, et al. 2023. Gpt-4 technical report.

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muenninghoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santocoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988.

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giacinto, Haifeng Qian, Murali Krishnan Ramathan, Ramesh Nallapati, Baisakhik Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. ArXiv preprint, abs/2108.07732.

BAAI. 2024. Hqcode dataset. Accessed: 2024-02-16.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong
Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Owen technical report. arXiv preprint arXiv:2309.16609.

Andrei Z. Broder. 1997. On the resemblance and containment of documents. In Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings, pages 21–29. IEEE.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Federico Cassano, John Gouwar, Daniel Nguyen, Syndey Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227.

Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Cognitive Computations. 2023. Code-290k sharegpt vicuna dataset. https://huggingface.co/datasets/cognitivecomputations/Code-290k-ShareGPT-Vicuna.

Together Computer. 2023. Redpajama: an open dataset for training large language models.

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, Yuanxing Zhang, Wenbo Su, Bangyu Xiang, Tiezheng Ge, and Bo Zheng. 2024. R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models. ArXiv, abs/2406.01359.

Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models. arXiv preprint arXiv:2402.07844.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiacheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155.

Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khayathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muenninghoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. 2024. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics.

Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayibeng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186.

Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code-searchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436.

INF-Team. 2024. Inf’s open-source large language models.

Albert Qiaocheu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elo Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mstral 7b. ArXiv.

Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 18319–18345. PMLR.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499.

Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. 2014. Mining of Massive Datasets, 2nd Ed. Cambridge University Press.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023a. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023b. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.

GitHub Linguist. 2024. languages.yml. Accessed: 2024-11-07.

Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang, Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, et al. 2024a. E2-llm: Efficient and extreme length extension of large language models. arXiv preprint arXiv:2401.06951.

Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, Ge Zhang, Zekun Moore Wang, Guoan Zhang, Bangyu Xiang, Wenbo Su, and Bo Zheng. 2024b. M2rc-eval: Massively multilingual repository-level code completion evaluation.

Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, et al. 2024c. Ddk: Distilling domain knowledge for efficient large language models. arXiv preprint arXiv:2407.16154.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024d. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. 2024e. MdevaL: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310.

Tianyang Liu, Canwen Xu, and Julian J. McAuley. 2023a. Repobench: Benchmarking repository-level code auto-completion systems. abs/2306.03091.

Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. 2023b. Llm360: Towards fully transparent open-source llms. Preprint, arXiv:2312.06550.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024a. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024b. Starcoder 2 and the stack v2: The next generation.

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2024a. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wetting, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. 2024b. Olmoe: Open mixture-of-experts language models. Preprint, arXiv:2409.02060.

Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023a. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023b. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations.

Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557.

Jim Plotts and Megan Risdal. 2023. Meta kaggle code.

Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174–15186.

Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yi Ma, Feiyu Duan, Zhiqu Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. 2024. D-cpt law: Domain-specific continual pre-training scaling law for large language models. ArXiv, abs/2406.01375.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI preprint.

Baptiste Roziere, Jonas Gehring, Fabian Glocckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. Wu Y.K. Li, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.

Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202.

Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. 2024. Jetmoe: Reaching llama2 performance with 0.1 m dollars. arXiv preprint arXiv:2404.07413.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-lm: Training multi-billion parameter language models using model parallelism. Preprint, arXiv:1909.08053.

Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31693–31715. PMLR.

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, et al. 2024. Code needs comments: Enhancing code llms with comment augmentation. arXiv preprint arXiv:2402.13013.

Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. 2024. Unicoder: Scaling code large language model via universal code. arXiv preprint arXiv:2406.16441.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Tianhua Tao, Junbo Li, Bowen Tan, Hongyi Wang, William Marshall, Bhargav M Kanakia, Joel Hestness, Natalia Vassileva, Zhiqiang Shen, Eric P Xing, et al. Crystal: Illuminating llm abilities on language and code. In First Conference on Language Modeling.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.

Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhui Chen, Jie Fu, and Junran Peng. 2023. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746.

Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weimei Li, Rui Hu, et al. 2023a. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023b. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120.

Yanan Wu, Jie Liu, Xingyuan Bu, Jiaheng Liu, Zhanhui Zhou, Yuanxing Zhang, Chenchen Zhang, Zhiqi Bai, Haibin Chen, Tiezheng Ge, et al. 2024. Conceptmath: A bilingual concept-wise benchmark for measuring mathematical reasoning of large language models. arXiv preprint arXiv:2402.14660.

Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. Codetransoceano: A comprehensive multilingual benchmark for code translation.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhui Chen. 2024. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548.

Yuxiang630. 2024. Hqcode dataset. Accessed: 2024-02-16.

Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570.

Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. 2024a. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.19327.

Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. 2024b. Automathext: Autonomous data selection with language models for mathematical texts. arXiv preprint arXiv:2402.07625.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations.

Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. 2024. Programming every example: Lifting pre-training data quality like experts at scale. arXiv preprint arXiv:2409.17115.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widysari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.
A  Raw Code Processing

In this section, we delve into the details of the data processing pipeline in RefineCode. Specifically, we focus on on the design and considerations of two crucial modules: deduplication and filtering, and talk about their orders.

A.1 Processing Details

Deduplication  Owing to the extremely high repetition of the source code in Github, we adopt an aggressive file-level deduplication strategy (see elaborate analysis in Appendix C). More specifically, we leverage both exact deduplication and fuzzy deduplication methods to eliminate documents containing identical or near-identical code content shown as follows:

Exact Deduplication: Due to the prevalence of forking and copy-pasting within the codebase, nearly 75% of files are completely duplicated. On account of this, differing from general deduplication process, Identity removal is applied towards code data at the first step in this module. We compute the SHA256 hash value for each document, where files with identical hash values are compared, and only the code files with the highest star count as well as the latest commit time are retained, in order to preserve the highest quality and most up-to-date files.

Fuzzy Deduplication: This step aim to deduplicate those near-identical files. We split the raw text into 5-gram pieces, and then calculate the 2048 MinHash functions (Broder, 1997). Additionally, we utilize LSH (Leskovec et al., 2014) by setting bands to 16 and rows to 128, to retain only those distinct files with the highest stars and latest commit time. This process removes 6% file volume.

Filtering  Given the distinct nature of code compared to natural language, the criteria for high-quality code differ significantly from those for natural language. Furthermore, different programming languages also exhibit distinct properties. Based on this, we believe that designing a set of detailed heuristic filtering rules tailored specifically to the characteristics of pretraining data is important to enhance the model’s capabilities. Drawing inspiration from the principles of high-quality code data proposed in Gunasekar et al. (2023), we consider the following guidelines when designing our filters: 1) Filter out files with poor self-containment; 2) Filter out files with poor or minimal logical structure; 3) Remove files that deviate significantly from standard formatting.

Based on these guidelines and the characteristics of our dataset, our work presents the first heuristic filtering framework by considering the unique characteristics of different programming languages. Based on RedPajama (Computer, 2023), this framework extends and refines the existing rules from StarCoder (Li et al., 2023b) to better align with the unique properties of code datasets, resulting in more precise and higher-quality data cleansing. We developed the following three categories of filtering rules:

1. Natural Language Filtering Rules: These rules filter data based on common properties for all text files, such as file size, number of lines, and other general metrics. Both text and code files share these filtering rules.

2. General Code Filtering Rules: These rules apply to all code files by filtering data based on general code characteristics, such as the number of variables, average function length, and other common features.

3. Language-Specific Filtering Rules: These rules are designed according to the unique characteristics of specific programming languages, such as the frequency of “pass” statements in Python or the use of “goto” statements in C. We have developed these rules for the following eight commonly used programming languages: Python, C, C++, C#, Java, JavaScript, Go, and HTML.

Heuristic rules involve extensive threshold setting. When defining these rules and determining thresholds, we consistently follow a guiding principle: to remove harmful data as much as possible, while ensuring the overall distribution of the dataset is not significantly affected. We outline our motivations for rule design in Appendix B.1, along with a detailed explanation of the tuning process for the corresponding thresholds. Besides, we show the details of several representative rules in Appendix B.2.

A.2 Processing Order

Most LLM data processing pipelines adopt a strategy where filtering is applied first, followed by deduplication. In contrast, our approach prioritizes deduplication before filtering, which offers advantages from two perspectives:
• Processing Cost and Efficiency: As mentioned earlier, over 90% of files in raw code are exact duplicates. Performing deduplication upfront helps avoid the computational overhead costing by filtering redundant files in the subsequent filtering phase. Additionally, the filtering rules are subject to frequent revisions, which means that both the filtering phase and the stages following it would need to be repeated. By conducting deduplication before filtering, we can mitigate the extra computational demands that arise from these repeated adjustments.

• Data Intuition: The effectiveness of the filtering stage must be evaluated based on the distribution of the processed data. Therefore, when filtering is applied last, the resulting data distribution directly reflects the distribution used during training, allowing for more intuitive and rapid adjustments to the filtering rules. In contrast, if filtering is applied before deduplication, the final data distribution used for training will be altered by the deduplication process, making it difficult to adjust the filtering rules based solely on the post-filtered data distribution.

Given these considerations, we argue that performing deduplication before filtering is a more rational choice for code pretraining data.

B Filtering Rules

B.1 Design of Filtering Rules

Designing heuristic filtering rules is inherently challenging, often requiring iterative refinement and experimentation to ultimately develop an effective set of rules. Given this complexity, in addition to providing detailed explanations of our designed rules, we will also share the general insights and methodologies we have accumulated throughout the designing process. We believe that this section will offer valuable guidance for designing heuristic filtering rules applicable to any dataset, thereby significantly enhancing the efficiency of constructing an effective data cleaning pipeline.

Heuristic rules filter data based on specific characteristics of a file, which, for each file, are ultimately expressed as a score representing the file’s attribute and a corresponding threshold set by the rule. During the rule design process, we found that understanding the distribution of scores and the impact of different threshold settings on data filtering is critical to creating effective rules. Therefore, based on the approach used in RedPajama (Computer, 2023), we decompose the heuristic filtering process into two steps: quality signal computation and filtering execution. The quality signal computation calculates the scores for all rules for each file, while the filtering execution module decides whether a file is retained based on its quality signal scores and the corresponding thresholds.

Additionally, we recommend placing the heuristic filtering process as late as possible in the overall data pipeline. Unlike other, more fixed stages of the data processing pipeline, this stage requires frequent adjustments based on the final quality of the data. Placing it later in the process allows for more precise control over the data and minimizes the need to repeat subsequent steps after this filtering module.

The specific steps for designing our heuristic filtering rules are as follows:

1. Quality Signals Designing: Based on the definition of low-quality data and the attributes of the dataset, we firstly design a series of quality signals that describe the attributes contributing to file quality.

2. Coarse Threshold Tuning: Referring to the definition of low-quality data and the distribution of quality signal scores, we roughly set filtering thresholds for all rules at once. We then apply the filters to obtain an initial version of the filtered dataset.

3. Fine-grained Threshold Tuning: For each rule, we focus on the data that was exclusively affected by that specific rule, meaning it did not trigger other filters. This part of the data is directly influenced by the current rule, so we can examine whether the retention or removal of this data under different threshold settings aligns with the intended purpose of the rule. If a rule is effective in improving data quality based on its target attribute, we select the optimal threshold; otherwise, the rule is discarded. After evaluating each rule, we apply the filters again to obtain a more refined filtered dataset.

4. Data Quality Inspection: We then assess whether the filtered dataset meets our expectations for the quality of pretraining data. In addition to traditional manual inspection, we
introduce a perplexity (PPL)-based method for data quality evaluation. Specifically, we randomly sample a set of data from the filtered dataset and use a high-performing LLM to compute the PPL on these samples. We then examine the top-N and bottom-N samples based on PPL. Generally, extremely low PPL suggests that the data is overly simplistic, containing limited valuable knowledge, while extremely high PPL indicates that the data may lack learnable patterns. Both of them are advisable to be filtered out. We closely inspect both sets of samples and, based on their characteristics, decide whether to add new rules or adjust existing thresholds. This process can be repeated until the dataset reaches the desired quality.

B.2 Examples of Filtering Rules

We elaborate several representative examples about general code filtering rules in Table 11 and language-specific filtering rules in Table 12 and explain their rationale. It is essential to note that for general code filtering rules, the threshold values may be slightly adjusted depending on the programming language of the file. For specific threshold values, please refer to our implementation details of the data processing pipeline.

C Analysis on Chunk-level Deduplication

During pretraining, data is first randomly concatenated and segmented into chunks of context length, followed by full-attention computation within each chunk. We further explored chunk-level deduplication. Specifically, the pretraining data was randomly concatenated and segmented into chunks of 4096 tokens, followed by MinHash and LSH deduplication on these chunks. Additionally, we applied chunk-level deduplication after file-level and repo-level deduplication.

From the results in Table 13, We observe that chunk-level deduplication alone was even less effective than repo-level deduplication, and applying chunk-level deduplication after file-level removed only an additional 0.04B of data. This indicates that chunk-level deduplication is not an effective approach. We pre-trained three 1.5B models on the data retained under file-level, repo-level, and repo-level + chunk-level deduplication strategies. The benchmark results are shown in Figure 6. It is evident that file-level deduplication achieves the highest training efficiency, while repo-level + chunk-level deduplication outperforms repo-level alone. We attribute the superior performance of file-level deduplication to its higher degree of data removal. Overall, we conclude that file-level deduplication is the most suitable method for GitHub data.

![Impact of Deduplication Strategies](page_820_172_579_377.png)

Figure 6: Comparison of Pass@1 performance on HumanEval & MBPP for different dedup strategies (File-Level, Repo-Level, and Repo-level + Chunk-Level) across RefineCode Python corpus.

D Extra Data Processing

D.1 Code-Related Data from Web Corpus

As mentioned in Section 2.1.2, we recall code-related data from web corpus referring to the method proposed in DeepSeekMath (Shao et al., 2024). The concrete steps are shown in the following:

1. Annotation: We annotate 500k high-quality code-like data from CommonCrawl using the Autonomous Data Selection method (Zhang et al., 2024b) as seed.

2. Fasttext Training: We train a fasttext model using annotated data to identify whether those documents from web data are code-related or not.

3. Common Crawl Recall: We retrieve code-related web pages from Common Crawl using the trained fasttext model.

4. Domain Discovery: Domains (e.g., stackoverflow.com) are identified based on URL statistics, and those domain including over
Table 11: Examples of general code filtering rules.

<table>
  <tr>
    <th>Description</th>
    <th>Explanation</th>
    <th>Filtering Quota</th>
  </tr>
  <tr>
    <td>The proportion of lines in strings with a word count exceeding.</td>
    <td>Files with too many long strings indicate a lack of code logic.</td>
    <td>score &gt; 0.2</td>
  </tr>
  <tr>
    <td>The proportion of characters in words from strings with a character count exceeding 20.</td>
    <td>String variables containing long sequences of characters are often indicative of meaningless content such as base64 data, Hash encoding, url, etc.</td>
    <td>score &gt; 0.4</td>
  </tr>
  <tr>
    <td>The proportion of hexadecimal characters.</td>
    <td>Files with two many hexadecimal characters indicate a lack of code logic.</td>
    <td>score &gt; 0.4</td>
  </tr>
  <tr>
    <td>The proportion of lines like "you code here", "TODO" or "FIXME".</td>
    <td>We found that these elements tend to be excessively repeated in the dataset, which increases the likelihood that the model, during code completion, will output placeholders like the ones mentioned above instead of generating actual code.</td>
    <td>score &gt; 0.01</td>
  </tr>
  <tr>
    <td>The proportion of lines containing an "assert" statement.</td>
    <td>Files containing a large number of 'assert' statements are often test files, which tend to have relatively simple and repetitive code patterns.</td>
    <td>score &gt; 0.4</td>
  </tr>
</table>

Table 12: Examples of python-specific filtering rules.

<table>
  <tr>
    <th>Description</th>
    <th>Explanation</th>
    <th>Filtering Quota</th>
  </tr>
  <tr>
    <td>The proportion of the number of python functions to the total number of lines.</td>
    <td>A higher number of Python functions in a file may indicate that the functions are overly simple, with limited code logic, or have a bad code format.</td>
    <td>score &gt; 0.2</td>
  </tr>
  <tr>
    <td>Whether the file can be parsed into an python abstract syntax tree (AST).</td>
    <td>Files that cannot be parsed into an AST contain syntax errors and should be filtered out.</td>
    <td>score == False</td>
  </tr>
  <tr>
    <td>The proportion of lines that are "import" statements.</td>
    <td>A file with exceeding proportion of "import" statements indicates to have sparse code logic.</td>
    <td>score &gt; 0.3</td>
  </tr>
</table>

10% code-related pages are classified as code-related.

5. URL Annotation: We manually annotate code-related URL patterns (e.g., stackoverflow.com/questions) and iteratively include misclassified but relevant pages into the seed corpus, and repeat the process from step 2.

We iterate this process three times and finally obtain 75 billion code-related tokens.

D.2 Chinese Code-Like Domains Annotation

The manual annotation of the URLs of the website is presented as shown in Table 14. For future new CC datasets, we can sample pages in these domains as initial seed corpus.

D.3 Code-Related Data from Github Text Files

Github Text files primarily consist of content written in natural languages, which includes abundant code-related knowledge. However, we observed that a substantial portion of the dataset is unrelated to code, which is detrimental to the model’s ability to learn code-related knowledge. Therefore, we employed the following strategies to extract and retain the code-relevant portions before our filtering module. Firstly, following the strategy used in star-coder (Li et al., 2023b), we retained the files with "requirement" in the lowercased filename, or if the filename without the extension is one of "readme", "notes", "todo", "description", "cmakelists", in order to ensure that only text files pertinent to coding
Table 13: Comparison of deduplication strategies on Python data. At the file-level, "Lines" refers to the number of lines in individual files; at the repo-level, it indicates the line count of aggregated strings; Note that for all deduplication strategies involving the Chunk level, "Lines" specifically refers to 4096-token chunks.

<table>
  <tr>
    <th>Dedup Level</th>
    <th># Total Samples</th>
    <th># Retained Samples</th>
    <th># Retained Tokens</th>
  </tr>
  <tr>
    <td>Chunk</td>
    <td>333,007,812</td>
    <td>79,272,460</td>
    <td>324.70 B</td>
  </tr>
  <tr>
    <td>File</td>
    <td>485,817,123</td>
    <td>30,488,834</td>
    <td>32.74 B</td>
  </tr>
  <tr>
    <td>File+Chunk</td>
    <td>333,007,812</td>
    <td>7,993,164</td>
    <td>32.70 B</td>
  </tr>
  <tr>
    <td>Repo</td>
    <td>11,037,352</td>
    <td>7,480,488</td>
    <td>99.47 B</td>
  </tr>
  <tr>
    <td>Repo+Chunk</td>
    <td>333,007,812</td>
    <td>17,675,781</td>
    <td>72.40 B</td>
  </tr>
</table>

contexts are preserved. This strategy recalled 3% volume of the whole text part. Additionally, we trained a fasttext model to recall code-related text files and recalled extra 7% file volume from the original text data.

D.4 Jupyter Notebooks

Our Jupyter notebook data is sourced from GitHub and Meta Kaggle code (Plotts and Risdal, 2023). We converted this type of data into the Jupyter-structured format used in StarCoder (Li et al., 2023b), which consists of a triplet of consecutive markdown, code, and code execution results. However, we discarded the Jupyter-script format mentioned in StarCoder. Because the code files generated from Jupyter notebook conversions tend to have poor overall code writing standards, and the content in Jupyter-script and Jupyter-structured formats is highly redundant, making it sufficient to retain only one format.

E Programming Languages Categories

E.1 Included Programming Languages

Included programming languages can be categorized into three classes: code, data and text. Among them, the "code" category represents files rich in code logic, while the "data" category primarily consists of files with structured data, and the "text" category refers to files dominated by natural language content. The threshold settings for the filtering rules vary slightly depending on the data type.

Code(470 types): 1C Enterprise, 4D, ABAP, ABAP CDS, AIDL, AL, AMPL, ANTLR, API Blueprint, APL, ASL, ASP.NET, ATS, ActionScript, Ada, Agda, Alloy, Alpine Build, AngelScript, Apex, Apollo Guidance Computer, AppleScript, Arc, AspectJ, Assembly, Astro, Asymptote, Augeas, AutoHotkey, AutoIt, Awk, BASIC, BQN, Ballerina, Batchfile, Beef, Befunge, Berry, Bikeshed, Bison, BitBake, Blade, BlitzBasic, BlitzMax, Bluespec, Boo, Boogie, Brainfuck, Brightscript, C, C#, C++, C2hs Haskell, CAP CDS, CLIPS, CMake, COBOL, CUE, Cadence, Cairo, CameLIGO, Cap'n Proto, Ceylon, Chapel, Charity, Chuck, Circom, Cirru, Clarion, Clarity, Classic ASP, Clean, Click, Clojure, Closure Templates, CodeQL, CoffeeScript, ColdFusion, ColdFusion CFC, Common Lisp, Common Workflow Language, Component Pascal, Coq, Crystal, Csound, Csound Document, Csound Score, Cuda, Curry, Cycript, Cypher, Cython, D, D2, DIGITAL Command Language, DM, Dafny, Dart, DataWeave, Dhall, Diff, Dockerfile, Dogescript, Dylan, E, ECL, EJS, EQ, Earthly, Edge, EdgeQL, Elixir, Elm, Elvish, Emacs Lisp, EmberScript, Erlang, F#, F*, FIRRTL, FLUX, Factor, Fancy, Fantom, Faust, Fennel, Filebench WML, Fluent, Forth, Fortran, Fortran Free Form, FreeBasic, Futhark, GAML, GAMS, GAP, GDB, GLSL, GSC, Game Maker Language, Genero 4gl, Genero per, Genshi, Gentoo Ebuild, Gentoo Eclass, Gherkin, Gleam, Glimmer JS, Glyph, Go, Golo, Gosu, Grace, Grammatical Framework, Groovy, Groovy Server Pages, HCL, HLSL, HTML, HTML+ECR, HTML+EEX, HTML+ERB, HTML+PHP, HTML+Razor, Hack, Haml, Handlebars, Harbour, Haskell, Haxe, HiveQL, HolyC, Hy, IDL, Igor Pro, Idris, ImageJ Macro, Imba, Inform 7, Ink, Inno Setup, Io, Ioke, Isabelle, Isabelle ROOT, J, JCL, JFlex, JSONiq, Janet, Jasmin, Java, Java Server Pages, JavaScript, JetBrains MPS, Jinja, Jison, Jison Lex, Jolie, Jsonnet, Julia, Just, KRL, Kaitai Struct, KakouneScript, KerboScript, Kit, Kotlin, LFE, LLVM, LOCLCODE, LSL, LabVIEW, Latte, Lean, Less, Lex, LigoLANG, LilyPond, Limbo, Liquid, Literate Agda, Literate CoffeeScript, Literate Haskell, LiveScript, Logos, Logtalk, LookML, Lua, Luau, M, M4, M4Sugar, MATLAB, MAXScript, MLIR, MQL4, MQL5, MTML, MUF, Macaulay2, Makefile, Mako, Marko, Mask, Mathematica, Mercury,
Table 14: We manually annotate code-like and math-like Chinese domains, utilizing the ‘%’ symbol as a wildcard in our pattern matching. For example, the URL ‘https://my.oschina.net/u/4/blog/111’ is matched by the pattern ‘%my.oschina.net%blog%’.

<table>
  <tr>
    <th>Domain</th>
    <th>Prefix</th>
    <th>Tag</th>
  </tr>
  <tr>
    <td>cloud.tencent.com</td>
    <td>%cloud.tencent.com/developer/article%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>cloud.tencent.com</td>
    <td>%cloud.tencent.com/ask%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>cloud.tencent.com</td>
    <td>%cloud.tencent.com/developer/information%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>cloud.tencent.com</td>
    <td>%cloud.tencent.com/document%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>my.oschina.net</td>
    <td>%my.oschina.net%blog%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>ask.csdn.net</td>
    <td>%ask.csdn.net/questions%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>www.cnblogs.com</td>
    <td>%www.cnblogs.com%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>forum.ubuntu.org.cn</td>
    <td>%forum.ubuntu.org.cn%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>q.cnblogs.com</td>
    <td>%q.cnblogs.com/q%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>segmentfault.com</td>
    <td>%segmentfault.com/q%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>segmentfault.com</td>
    <td>%segmentfault.com/a%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>woshipm.com</td>
    <td>%woshipm.com/data-analysis%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>zgserver.com</td>
    <td>%zgserver.com/server%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>zgserver.com</td>
    <td>%zgserver.com/linux%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>zgserver.com</td>
    <td>%zgserver.com/ubuntu%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>juejin.cn</td>
    <td>%juejin.cn/post%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>jijizhixin.com</td>
    <td>%jijizhixin.com/articles%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>help.aliyun.com</td>
    <td>%help.aliyun.com/zh%</td>
    <td>Code</td>
  </tr>
  <tr>
    <td>jyeoo.com</td>
    <td>%jyeoo.com%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>www.haihongyuan.com</td>
    <td>%haihongyuan.com%shuxue%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>www.03964.com</td>
    <td>%www.03964.com%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>www.nbhkdz.com</td>
    <td>%www.nbhkdz.com%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>9512.net</td>
    <td>%9512.net%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>lanxicy.com</td>
    <td>%lanxicy.com%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>bbs.emath.ac.cn</td>
    <td>%bbs.emath.ac.cn%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>math.pro</td>
    <td>%math.pro%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>mathschina.com</td>
    <td>%mathschina.com%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>shuxue.chazidian.com</td>
    <td>%shuxue.chazidian.com%</td>
    <td>Math</td>
  </tr>
  <tr>
    <td>shuxue.ht88.com</td>
    <td>%shuxue.ht88.com%</td>
    <td>Math</td>
  </tr>
</table>

Mermaid, Meson, Metal, MiniD, Mint, Mirah, Modelica, Modula-3, Module Management System, Mojo, Monkey, MoonScript, Motorola 68K Assembly, Move, Mustache, Myghty, NASL, NSIS, NWScript, Nearly, Nemerle, NetLinx, NetLogo, Nextflow, Nim, Nit, Nix, Nu, NumPy, Nunjucks, OCaml, Oberon, Objective-C++, Objective-J, Omgrofl, Opa, Opal, Open Policy Agent, OpenCL, OpenQASM, OpenSCAD, Ox, Oxygene, Oz, P4, PDDL, PEG.js, PHP, PLSQL, PLpgSQL, Pact, Pan, Papyrus, Parrot, Parrot Assembly, Parrot Internal Representation, Pascal, Pawn, Pep8, Perl, PigLatin, Pike, PogoScript, Polar, Pony, Portugol, PowerBuilder, PowerShell, Praat, Processing, Pro-File, Prolog, Promela, Propeller Spin, Pug, Puppet, PureScript, Prover9, Pyret, Python, Q#, QML, QMake, Qt Script, Quake, R, RAML, REALbasic, REXX, RPGLE, RUNOFF, Racket, Ragel, Raku, Rascal, ReScript, Reason, ReasonLIGO, Rebol, Red, Redcode, RenderScript, Ring, Riot, Robot-Framework, Roc, Rouge, Ruby, Rust, SAS, SMT, SQF, SQL, Sage, SaltStack, Sass, Scala, Scaml, Scenic, Scheme, Scilab, Self, Shell, ShellSession, Shen, Sieve, Singularity, Slash, Slim, Slint, SmPL, Smali, Smalltalk, Smarty, Smithy, Snakemake, SourcePawn, Squirrel, Stan, Standard ML, Starlark, Stata, Stylus, SugarSS, Svelte, Sway, Swift, SystemVerilog, TI Program, TL-Verilog, TLA, TSX, TXL, Talon, Tcl, Tcsh, Tea, Terraform Template, Thrift, Toit, Turing, Twig, TypeScript, Typst, Unified Parallel C, Uno, UnrealScript, UrWeb, V, VBA, VBScript, VCL, VHDL, Vala, Velocity Template
Language, Verilog, Vim Script, Vim Snippet, Visual Basic .NET, Visual Basic 6.0, Volt, Vue, Vyper, WDL, WGLS, WebAssembly, WebIDL, Whiley, Witcher Script, Wollok, Wren, X10, XC, XProc, XQuery, XS, XSLT, Xojo, Xonsh, Xtend, YARA, YASnippet, Yacc, Yul, ZAP, ZIL, Zeek, ZenScript, Zephir, Zig, Zimpl, eC, fish, hoon, kvlang, mIRC Script, mcfunetion, mupad, nesC, ooc, templ, wisp, xBase

Data(115 types): ABNF, ASN.1, Adobe Font Metrics, Altium Designer, Ant Build System, ApacheConf, Avro IDL, BibTeX, Browserslist, CIL, CODEOWNERS, CSOM, CSS, Cabal Config, Caddyfile, CartoCSS, Cloud Firestore Security Rules, CoNLL-U, DNS Zone, Darcs Patch, Debian Package Control File, Dotenv, EBNF, Eagle, Easybuild, Ecere Projects, EditorConfig, Edje Data Collection, FIGlet Font, Formatted, GEDCOM, GN, Gemfile.lock, Gerber Image, Git Attributes, Git Config, Glyph Bitmap Distribution Format, Go Checksums, Go Module, Go Workspace, Godot Resource, Gradle, Gradle Kotlin DSL, GraphQL, Graphviz (DOT), HAProxy, HTTP-CON, HTTP, HXML, INI, Ignore List, JAR Manifest, JSON, JSON with Comments, Jest Snapshot, Kusto, Lark, Linker Script, Maven POM, NEON, NL, NPM Config, Nginx, Ninja, ObjDump, Object Data Instance Notation, OpenStep Property List, OpenType Feature File, Option List, PlantUML, PostCSS, Prisma, Protocol Buffer, Protocol Buffer Text Format, Python traceback, RBS, RON, Readline Config, Record Jar, Redirect Rules, Regular Expression, SCSS, SELinux Policy, SPARQL, SSH Config, STAR, STON, ShellCheck Config, Simple File Verification, Soong, Spline Font Database, TOML, TextMate Properties, Turtle, Type Language, Valve Data Format, Wavefront Material, Web Ontology Language, WebAssembly Interface Type, Wget Config, Windows Registry Entries, X BitMap, X Font Directory Index, XCompose, XML, XML Property List, XPages, YAML, YANG, eURL Config, crontab, desktop, dircolors, edn, nanorc

Text(22 types): AsciiDoc, Creole, Gemini, Gettext Catalog, MDX, Markdown, Muse, Org, Pod, Pod 6, RDoc, RMarkdown, Rich Text Format, Roff, SRecode Template, Sweave, TeX, Texinfo, Text, Textile, Wikitext, reStructuredText

E.2 Excluded Programming Languages
2-Dimensional Array, AGS Script, Adblock Filter List, Bicep, COLLADA, CSV, Checksums, DirectX 3D File, E-mail, G-code, Git Revision List, Gnuplot, IRC log, KiCad Layout, KiCad Legacy Layout, KiCad Schematic, Lasso, Linux Kernel Module, Max, Microsoft Developer Studio Project, Microsoft Visual Studio Solution, POV-Ray SDL, Pic, Pickle, PostScript, Public Key, Pure Data, Pure-Basic, Raw token data, Roff Manpage, STL, SVG, SubRip Text, TSV, Unity3D Asset, Wavefront Object, WebVTT, X PixMap, robots.txt

F Raw Code Data Composition
Figure 15 shows the composition of raw code data for top 85 programming languages in the RefineCode dataset, both after deduplication and filtering process, and Figure 7 unveil the training data compisition trending without data sampling. It can be observed that, after filtering, the proportion of data for different programming languages has shifted significantly, with a notable increase in the representation of commonly used programming languages.

G Benchmark
G.1 Code Benchmark
In this section, we will present the code benchmarks used for model evaluation and supplement the complete results.

G.1.1 Benchmark for Base Models
HumanEval & MBPP We selected two widely used code completion benchmarks to evaluate OpenCoder, HumanEval (Chen et al., 2021)and MBPP (Austin et al., 2021). To further enhance the accuracy of the evaluation, EvalPlus (Liu et al., 2024d) extends HumanEval and MBPP to HumanEval Plus and MBPP Plus by adding unique and challenging test cases and correcting for inaccurate ground truth solutions. These results can be used to indicate the model’s ability to understand and apply basic Python data structures and knowledge of algorithms. For HumanEval, we report the 0-shot results. For MBPP, we report 3-shots’ results on 500 questions in the test split from MBPP (Austin et al., 2021), while the base and the plus results following EvalPlus (Liu et al., 2024d) report results on 378 questions in the sanitized part. Therefore, these results are not comparable and evaluated based on different data splits.
BigCodeBench BigCodeBench (Zhuo et al., 2024) is a challenging benchmark for code completion, designed to assess the models’ ability to handle complex instructions and make accurate function calls across diverse external libraries. In the Completion setup, models are provided with a function signature, related documentation to generate appropriate code, and a unit test for the completed function. Covering a range of practical programming tasks, it evaluates the ability of the models to handle real-world scenarios involving complex, task-specific libraries.

G.1.2 Benchmark for Instruct Model

LiveCodeBench LiveCodeBench is a comprehensive, contamination-free benchmark assessing highly complex algorithmic tasks’ reasoning and problem-solving abilities. The benchmark is continuously updated with new problems from platforms such as LeetCode, AtCoder, and CodeForces, ensuring the challenges remain current and diverse. LiveCodeBench provides a robust measure of a model’s ability to handle sophisticated logical processes, which is essential in competitive programming contexts. The instruct models are evaluated on the 2305-2409 data split.

We follow the QwenCoder evaluation code * to systematically measure performance in MultiPL-E (Cassano et al., 2022), providing insight into the adaptability and precision of the generation of LLM codes in Table 16. In addition, we evaluate our model using two more comprehensive benchmarks: McEval (Chai et al., 2024) in Table 8, and MdEval (Liu et al., 2024e) in Table 9.

MultiPL-E MultiPL-E extends the HumanEval benchmark to evaluate the code generation capabilities of large language models across multiple languages. MultiPL-E translates tasks into languages such as C++, Java, PHP, TypeScript, C#, Bash, and JavaScript, providing a consistent basis for assessing how models apply their programming skills across different syntaxes and paradigms. We follow the evaluation code of QwenCoder† to systematically measure performance in each language, providing insights into the adaptability and code generation accuracy of LLMs in a multilingual context.

McEval The comprehensive multilingual code evaluation benchmark McEval (Chai et al., 2024) employed a detailed assessment of OpenCoder’s programming capabilities across 40 languages. In contrast to MultiPL-E, this benchmark is not derived from HumanEval or MBPP. Figure 8 depicts the results of the multilingual generation task for OpenCoder-8B-Instruct, which comprises nearly 2,000 samples. The figure illustrates that the model exhibits superior multilingual performance compared to other open-source models of comparable size.

MdEval OpenCoder is also evaluated on the comprehensive multilingual code debugging benchmark MdEval (Liu et al., 2024e) across 18 languages. In contrast to McEval, this benchmark focuses on the assessment of code debugging, especially for language-specific bugs. Figure 9 shows the results of the multilingual automated program repair task for OpenCoder-8B-Instruct, which comprises nearly 1.2K samples, which demonstrates that OpenCoder can effectively find the bugs and fix them compared to other open-source models of comparable size.

G.2 Comparison of OpenCoder with Reasoning Models

We also compared OpenCoder against several strong open-source reasoning models of similar sizes, including DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, and DeepSeek-R1-Distill-Qwen-1.5B (Guo et al., 2025). As shown in Table 17, OpenCoder performs competitively and often surpasses reasoning models on tasks like HumanEval, MBPP which require less complex reasoning. Additionally, BigCodeBench is designed to evaluate diverse function calls and complex code instructions. OpenCoder built Package-Instruct and Real-Instruct for package usage and real-world coding, outperforming reasoning models in complex tasks like function calls. However, on more logic-intensive benchmarks such as LiveCodeBench, reasoning models exhibit stronger performance due to their enhanced inference capabilities, showing their potential in this area. Overall, considering the scale of OpenCoder model family, it demonstrates strong and competitive performance in the code domain, and remains highly effective even compared to top open-source reasoning models.

G.3 Performance on General Benchmark

To provide a more comprehensive view of the model’s capabilities on general tasks such
as instruction-following and chat, we conducted additional evaluations using the OpenCompass (Contributors, 2023). Specifically, we compared OpenCoder-8B-Instruct against DeepSeek-6.7B-Instruct, StarCoder2-15B-Instruct-v0.1, and Qwen2.5-Coder-7B on a range of general-purpose metrics including BBH (Suzgun et al., 2022), GSM8K (Cobbe et al., 2021), IFEVAL (Zhou et al., 2023), MATH (Hendrycks et al., 2021), MMLU (Hendrycks et al., 2020), CMMLU (Li et al., 2023a).

As shown in Table 18, OpenCoder-8B-Instruct achieves comparable performance to models trained from scratch for code, including DeepSeek-6.7B-Instruct and StarCoder2-15B-Instruct-v0.1, and demonstrates particularly strong performance on instruction-following tasks (e.g., IFEVAL), indicating better alignment with user intent. Compared to Qwen2.5-Coder-7B, built on Qwen2.5-7B with 18T training tokens, OpenCoder performs lower on general metrics. This is expected because OpenCoder focuses more on code-related learning and uses less general-domain data. This design leads to high efficiency and strong performance on code-centric tasks.

H  Prompts For SFT Synthetic Data

Prompts for generating synthetic code SFT data are shown below.
Table 15: Overview of the data composition of in RefineCode. The items in the table are sorted in descending order according to the file volume after filtering.

<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">After deduplication</th>
    <th colspan="3">After filtering</th>
  </tr>
  <tr>
    <th># Files</th>
    <th>Vol(GB)</th>
    <th>Ratio(%)</th>
    <th># Files</th>
    <th>Vol(GB)</th>
    <th>Ratio(%)</th>
  </tr>
  <tr><td>html</td><td>141,081,897</td><td>3,175.4</td><td>8.56</td><td>45,100,466</td><td>582.4</td><td>18.08</td></tr>
  <tr><td>java</td><td>215,177,833</td><td>706.8</td><td>1.90</td><td>124,751,295</td><td>474.3</td><td>14.72</td></tr>
  <tr><td>python</td><td>109,725,362</td><td>493.3</td><td>1.33</td><td>58,640,346</td><td>271.1</td><td>8.41</td></tr>
  <tr><td>csharp</td><td>88,825,202</td><td>364.2</td><td>0.98</td><td>57,910,485</td><td>232.4</td><td>7.21</td></tr>
  <tr><td>javascript</td><td>190,670,421</td><td>1,925.0</td><td>5.19</td><td>69,579,517</td><td>226.9</td><td>7.04</td></tr>
  <tr><td>php</td><td>84,378,361</td><td>374.4</td><td>1.01</td><td>60,089,397</td><td>222.7</td><td>6.91</td></tr>
  <tr><td>cpp</td><td>51,362,503</td><td>375.2</td><td>1.01</td><td>38,037,406</td><td>176.9</td><td>5.49</td></tr>
  <tr><td>go</td><td>35,649,865</td><td>301.1</td><td>0.81</td><td>26,723,829</td><td>153.7</td><td>4.77</td></tr>
  <tr><td>typescript</td><td>40,211,985</td><td>287.4</td><td>0.77</td><td>20,621,755</td><td>140.4</td><td>4.35</td></tr>
  <tr><td>ruby</td><td>15,735,042</td><td>244.5</td><td>0.66</td><td>8,285,561</td><td>122.7</td><td>3.81</td></tr>
  <tr><td>perl</td><td>16,354,543</td><td>121.7</td><td>0.33</td><td>9,532,620</td><td>65.6</td><td>2.04</td></tr>
  <tr><td>rust</td><td>10,605,421</td><td>63.6</td><td>0.17</td><td>6,086,150</td><td>39.9</td><td>1.24</td></tr>
  <tr><td>r</td><td>6,132,978</td><td>92.5</td><td>0.25</td><td>4,803,109</td><td>34.7</td><td>1.08</td></tr>
  <tr><td>swift</td><td>4,238,754</td><td>47.9</td><td>0.13</td><td>2,938,498</td><td>31.8</td><td>0.99</td></tr>
  <tr><td>kotlin</td><td>4,493,548</td><td>56.4</td><td>0.15</td><td>3,123,156</td><td>29.8</td><td>0.94</td></tr>
  <tr><td>dart</td><td>4,087,329</td><td>33.0</td><td>0.09</td><td>2,161,462</td><td>18.5</td><td>0.57</td></tr>
  <tr><td>java-pages</td><td>6,174,654</td><td>31.0</td><td>0.08</td><td>4,145,336</td><td>15.4</td><td>0.48</td></tr>
  <tr><td>css</td><td>39,822,744</td><td>241.5</td><td>0.65</td><td>15,771,061</td><td>15.3</td><td>0.47</td></tr>
  <tr><td>lua</td><td>4,027,221</td><td>116.0</td><td>0.31</td><td>2,538,234</td><td>14.4</td><td>0.45</td></tr>
  <tr><td>xml</td><td>61,171,289</td><td>1,934.2</td><td>5.21</td><td>3,173,128</td><td>12.8</td><td>0.40</td></tr>
  <tr><td>scala</td><td>5,897,567</td><td>19.7</td><td>0.05</td><td>4,204,979</td><td>11.7</td><td>0.36</td></tr>
  <tr><td>shell</td><td>12,054,632</td><td>23.0</td><td>0.06</td><td>6,043,070</td><td>11.2</td><td>0.35</td></tr>
  <tr><td>pascal</td><td>1,306,130</td><td>27.8</td><td>0.07</td><td>960,497</td><td>9.5</td><td>0.29</td></tr>
  <tr><td>fortran</td><td>2,274,663</td><td>39.7</td><td>0.10</td><td>1,218,491</td><td>8.6</td><td>0.27</td></tr>
  <tr><td>perl6</td><td>1,943,430</td><td>16.4</td><td>0.04</td><td>1,034,748</td><td>8.6</td><td>0.27</td></tr>
  <tr><td>rmarkdown</td><td>1,317,760</td><td>14.0</td><td>0.04</td><td>827,951</td><td>7.9</td><td>0.25</td></tr>
  <tr><td>html+erb</td><td>7,618,377</td><td>11.4</td><td>0.03</td><td>4,452,355</td><td>7.8</td><td>0.24</td></tr>
  <tr><td>smali</td><td>3,457,531</td><td>37.9</td><td>0.10</td><td>1,408,274</td><td>7.4</td><td>0.23</td></tr>
  <tr><td>scss</td><td>18,061,278</td><td>35.6</td><td>0.10</td><td>7,705,822</td><td>7.4</td><td>0.23</td></tr>
  <tr><td>gettext catalog</td><td>1,100,044</td><td>51.3</td><td>0.14</td><td>442,385</td><td>6.3</td><td>0.19</td></tr>
  <tr><td>haskell</td><td>1,746,444</td><td>24.0</td><td>0.06</td><td>1,218,491</td><td>6.8</td><td>0.27</td></tr>
  <tr><td>tcl</td><td>253,345</td><td>4.2</td><td>0.01</td><td>136,171</td><td>1.0</td><td>0.03</td></tr>
  <tr><td>gradle</td><td>2,431,985</td><td>2.9</td><td>0.01</td><td>724,609</td><td>1.0</td><td>0.03</td></tr>
  <tr><td>scheme</td><td>357,909</td><td>4.7</td><td>0.01</td><td>201,170</td><td>1.0</td><td>0.03</td></tr>
  <tr><td>qml</td><td>354,756</td><td>1.8</td><td>0.01</td><td>252,621</td><td>1.0</td><td>0.03</td></tr>
  <tr><td>mdx</td><td>795,525</td><td>6.4</td><td>0.17</td><td>222,013</td><td>1.0</td><td>0.03</td></tr>
  <tr><td>classic asp</td><td>220,344</td><td>2.8</td><td>0.08</td><td>141,236</td><td>0.9</td><td>0.03</td></tr>
  <tr><td>xbase</td><td>192,780</td><td>2.5</td><td>0.07</td><td>80,396</td><td>0.9</td><td>0.03</td></tr>
  <tr><td>ini</td><td>7,232,136</td><td>19.1</td><td>0.05</td><td>1,517,099</td><td>1.3</td><td>0.04</td></tr>
  <tr><td>objective-c++</td><td>197,416</td><td>2.4</td><td>0.01</td><td>149,223</td><td>1.3</td><td>0.04</td></tr>
  <tr><td>motorola68k</td><td>1,066,095</td><td>26.5</td><td>0.07</td><td>220,218</td><td>1.2</td><td>0.04</td></tr>
  <tr><td>gap</td><td>752,261</td><td>2.6</td><td>0.01</td><td>510,420</td><td>1.2</td><td>0.04</td></tr>
</table>
Figure 7: The distribution of top program languages in RefineCode (before data sampling).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="8">1B+ Models</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Python</th>
    <th>Java</th>
    <th>C++</th>
    <th>C#</th>
    <th>TS</th>
    <th>JS</th>
    <th>PHP</th>
    <th>Bash</th>
  </tr>
  <tr>
    <td>DS-Coder-1.3B-Instruct</td>
    <td>1.3B</td>
    <td>65.2</td>
    <td>51.9</td>
    <td>45.3</td>
    <td>55.1</td>
    <td>59.7</td>
    <td>52.2</td>
    <td>45.3</td>
    <td>12.7</td>
    <td>48.4</td>
  </tr>
  <tr>
    <td>Yi-Coder-1.5B-Chat</td>
    <td>1.5B</td>
    <td>67.7</td>
    <td>51.9</td>
    <td>49.1</td>
    <td>57.6</td>
    <td>57.9</td>
    <td>59.6</td>
    <td>52.2</td>
    <td>19.0</td>
    <td>51.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-1.5B-Instruct</td>
    <td>1.5B</td>
    <td>71.2</td>
    <td>55.7</td>
    <td>50.9</td>
    <td>64.6</td>
    <td>61.0</td>
    <td><b>62.1</b></td>
    <td><b>59.0</b></td>
    <td>29.1</td>
    <td>56.7</td>
  </tr>
  <tr>
    <td>OpenCoder-1.5B-Instruct</td>
    <td>1.5B</td>
    <td><b>72.5</b></td>
    <td><b>64.6</b></td>
    <td><b>50.9</b></td>
    <td>61.4</td>
    <td><b>63.5</b></td>
    <td><b>62.1</b></td>
    <td>55.3</td>
    <td><b>29.7</b></td>
    <td><b>57.5</b></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="8">6B+ Models</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Python</th>
    <th>Java</th>
    <th>C++</th>
    <th>C#</th>
    <th>TS</th>
    <th>JS</th>
    <th>PHP</th>
    <th>Bash</th>
  </tr>
  <tr>
    <td>DS-Coder-6.7B-Instruct</td>
    <td>6.7B</td>
    <td>78.6</td>
    <td>68.4</td>
    <td>63.4</td>
    <td>72.8</td>
    <td>67.2</td>
    <td>72.7</td>
    <td>68.9</td>
    <td>36.7</td>
    <td>66.1</td>
  </tr>
  <tr>
    <td>DS-Coder-V2-Lite-Instruct</td>
    <td>16B</td>
    <td>81.1</td>
    <td><b>76.6</b></td>
    <td><b>75.8</b></td>
    <td>76.6</td>
    <td>80.5</td>
    <td>77.6</td>
    <td>74.5</td>
    <td>43.0</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td>CodeLlama-7B-Instruct</td>
    <td>7B</td>
    <td>45.7</td>
    <td>32.2</td>
    <td>28.6</td>
    <td>32.9</td>
    <td>39.0</td>
    <td>43.5</td>
    <td>31.7</td>
    <td>10.1</td>
    <td>33.0</td>
  </tr>
  <tr>
    <td>CodeGemma-7B-It</td>
    <td>7B</td>
    <td>59.8</td>
    <td>48.1</td>
    <td>46.6</td>
    <td>51.9</td>
    <td>54.7</td>
    <td>54.0</td>
    <td>46.6</td>
    <td>10.1</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>CodeQwen1.5-7B-Chat</td>
    <td>7B</td>
    <td>83.5</td>
    <td>70.9</td>
    <td>72.0</td>
    <td>75.9</td>
    <td>76.7</td>
    <td>77.6</td>
    <td>73.9</td>
    <td>41.8</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>Yi-Coder-9B-Chat</td>
    <td>9B</td>
    <td>85.4</td>
    <td>76.0</td>
    <td>67.7</td>
    <td>76.6</td>
    <td>72.3</td>
    <td>78.9</td>
    <td>72.1</td>
    <td>45.6</td>
    <td>71.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-7B-Instruct</td>
    <td>7B</td>
    <td><b>87.8</b></td>
    <td>76.5</td>
    <td>75.6</td>
    <td><b>80.3</b></td>
    <td><b>81.8</b></td>
    <td><b>83.2</b></td>
    <td><b>78.3</b></td>
    <td><b>48.7</b></td>
    <td><b>76.5</b></td>
  </tr>
  <tr>
    <td>OpenCoder-8B-Instruct</td>
    <td>8B</td>
    <td>83.5</td>
    <td>72.2</td>
    <td>61.5</td>
    <td>75.9</td>
    <td>78.0</td>
    <td>79.5</td>
    <td>73.3</td>
    <td>44.3</td>
    <td>71.0</td>
  </tr>
</table>

Table 16: Performance of various chat models on the MultiPL-E benchmark across different programming languages.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="2">HumanEval</th>
    <th colspan="2">MBPP</th>
    <th colspan="2">BigCodeBench</th>
    <th rowspan="2">LiveCodeBench Avg</th>
  </tr>
  <tr>
    <th>HE</th>
    <th>HE+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
    <th>Full</th>
    <th>Hard</th>
  </tr>
  <tr>
    <th colspan="9">1B+ Models</th>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Qwen-1.5B</td>
    <td>1.5B</td>
    <td>48.7</td>
    <td>45.1</td>
    <td>51.8</td>
    <td>43.9</td>
    <td>7.0</td>
    <td>0.0</td>
    <td><b>16.9</b></td>
  </tr>
  <tr>
    <td>OpenCoder-1.5B-Instruct</td>
    <td>1.5B</td>
    <td><b>72.5</b></td>
    <td><b>67.7</b></td>
    <td><b>72.7</b></td>
    <td><b>61.9</b></td>
    <td><b>34.6</b></td>
    <td><b>11.5</b></td>
    <td><b>12.8</b></td>
  </tr>
  <tr>
    <th colspan="9">6B+ Models</th>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Qwen-7B</td>
    <td>7B</td>
    <td>73.9</td>
    <td>69.5</td>
    <td>69.3</td>
    <td>61.1</td>
    <td>10.6</td>
    <td>3.4</td>
    <td>37.6</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Llama-8B</td>
    <td>8B</td>
    <td>74.3</td>
    <td>65.8</td>
    <td>67.8</td>
    <td>57.6</td>
    <td>17.5</td>
    <td>2.0</td>
    <td><b>39.6</b></td>
  </tr>
  <tr>
    <td>OpenCoder-8B-Instruct</td>
    <td>8B</td>
    <td><b>83.5</b></td>
    <td><b>78.7</b></td>
    <td><b>79.1</b></td>
    <td><b>69.0</b></td>
    <td><b>42.9</b></td>
    <td><b>16.9</b></td>
    <td><b>23.2</b></td>
  </tr>
</table>

Table 17: Comparison of OpenCoder with reasoning models on HumanEval, MBPP, the “instruct” task of Big-CodeBench and LiveCodeBench.
Figure 8: The McEval performance of OpenCoder-8B-Instruct in comparison to other open-source code models of comparable size.

Figure 9: The MdEval performance of OpenCoder-8B-Instruct in comparison to other open-source code models of comparable size.
<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>BBH</th>
    <th>GSM8K</th>
    <th>IFEVAL</th>
    <th>MATH</th>
    <th>MMLU</th>
    <th>CMMILU</th>
  </tr>
  <tr>
    <td>DeepSeek-6.7B-Instruct</td>
    <td>6.7B</td>
    <td>45.60</td>
    <td>34.80</td>
    <td>21.70</td>
    <td>3.12</td>
    <td>37.91</td>
    <td>37.94</td>
  </tr>
  <tr>
    <td>Starcoder2-15B-Instruct-v0.1</td>
    <td>15B</td>
    <td>21.58</td>
    <td>53.15</td>
    <td>45.92</td>
    <td>14.46</td>
    <td>44.12</td>
    <td>44.76</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-7B</td>
    <td>7B</td>
    <td><b>60.60</b></td>
    <td><b>80.82</b></td>
    <td><b>70.62</b></td>
    <td><b>43.40</b></td>
    <td><b>66.11</b></td>
    <td><b>66.29</b></td>
  </tr>
  <tr>
    <td>OpenCoder-8B-Instruct</td>
    <td>8B</td>
    <td>46.10</td>
    <td>34.95</td>
    <td>54.44</td>
    <td>9.58</td>
    <td>41.96</td>
    <td>38.85</td>
  </tr>
</table>

Table 18: Performance of various chat models on common general benchmarks.

Prompt for Educational Instruction Synthesis

You are a teaching assistant helping to create a Python programming task from a given code snippet. You must provide the best response to the Python programming task, including reasoning thought, reference solutions, explanation of test cases, and test code.
[Code Snippet]
{Code}

Your response must have these parts:

[Task]
{Create an independent and detailed Python programming task}

[Analysis]
{Analyze the task and reason about the given task step by step}

[Solution]
{Write a high-quality reference solution in a self-contained script that solves the task}

[Test]
{Provide ten assert statements to check the correctness of your solution}
Prompt for Package-related Instruction Synthesis

You are exceptionally skilled at crafting high-educational level problems and offering precise solutions. Please gain inspiration from the following code snippet to create a high-quality programming problem, which is beneficial for learning the use of corresponding libraries. Present your output in two distinct sections: [Problem Description] and [Solution].

[Code Snippet]
{Code}

[Library Api Requirements]
{Api Requirements}

[Library Api Doc]
{Api Doc}

Guidelines for each section:
1. [Problem Description]: This should be **completely self-contained**, providing all the contextual information one needs to understand and solve the problem. Assume common programming knowledge, but ensure that any specific context, variables, or code snippets pertinent to this problem are explicitly included. This problem should be **educational for learning the provided Library api, and please explicitly request the use of the relevant package in the question. This question should only concern the writing of **one function***, and you need to be clear about the function name and role of this function.

2. [Solution]: Offer a comprehensive, **correct** solution that addresses the [Problem Description] you provided. This solution should follow the standard of corresponding Library Api doc. Please ensure that the Solution only involves answering the Problem, **without addressing the requirements I provided!** Please provide essential explanation abouth this solution, especially the use of required Library Api.

Prompt for Large-scale Diverse Instruction Synthesis

You are an expert in designing high-quality programming questions based on the given text.

[Guidelines]
- You can draw inspiration from the given text to create the programming questions.
- The created question should be a self-contained question, which does not depend on any external context.
- The created response must contain the complete code snippet.

[Given Text]
{Given Text}

[Created Question]
{Created Question}

Reviewer paper summaries: This paper demonstrates the full process of building a code LLM. From data collection to fine-tuning, the authors detailly explain almost everything the research community needs. Experiments shows that OpenCoder surpasses most fully open models.
Summary of strengths: 1. The paper is well written.
2. Open-source efforts to the community, including the whole training pipeline of code LLMs and the model weights.
3. Comprehensive evaluations and strong results.
Summary of weaknesses: 1. Most of the contributions are in engineering, but I argue that the valuable open-sourcing contributions can make it stand out.
Comments, suggestions and typos: 1. What is Code-Related Web Data ? And how it is annnotated ? Could you provide some examples for illustration ? 
2. Have you ever tried on some RL strategies during the post-training ?
3. Is OpenCoder trained from stratch or from base LLM ?
4. How does the model perform on some general tasks, such as instruction-following and chat ?
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper represents a recept for building a good code LLM from scratch focusing on not only pre-training but also post-training. For training such a model for code domain, this paper discusses both the data curation details and the training details.
Summary of reasons to publish: The reviewers agree that this work has the following advantanges:
- This paper is easy-to-follow and well-strcuted.
- A comprehensive roadmap for building a good code LLM including data curation and training details.