Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining

## Abstract

Abstract Recent studies have been increasingly demonstrating that high-quality data is crucial for effective pretraining of language models. However, the precise definition of “high-quality” remains underexplored. Focusing on the code domain, we introduce Arctic-SnowCoder -1.3B, a data-efficient base code model pretrained on 555B tokens through three phases of progressively refined data:
(1) general pretraining with 500B standard-quality code tokens, preprocessed through basic filtering, deduplication, and decontamination,
(2) continued pretraining with 50B high-quality tokens, selected from phase one by a BERT-style quality annotator trained to distinguish good code from random data, using positive examples drawn from high-quality code files, along with instruction data from Magicoder and StarCoder2-Instruct,
and (3) enhanced pretraining with 5B synthetic data created by Llama-3.1-70B using phase two data as seeds, adapting the Magicoder approach for pretraining.
Despite being trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art performance on BigCodeBench, a coding benchmark focusing on practical and challenging programming tasks, compared to similarly sized models trained on no more than 1T tokens, outperforming Phi-1.5-1.3B by 36%.
Across all evaluated benchmarks, Arctic-SnowCoder -1.3B beats StarCoderBase-3B pretrained on 1T tokens.
Additionally, it matches the performance of leading small base code models trained on trillions of tokens.
For example, Arctic-SnowCoder -1.3B surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a benchmark that evaluates function-level code generation, and remains competitive on BigCodeBench. Our evaluation presents a comprehensive analysis justifying various design choices for Arctic-SnowCoder . Most importantly, we find that the key to high-quality data is its alignment with the distribution of downstream applications.

## 1 Introduction

Pretraining large language models (LLMs) has generally relied on vast quantities of data. This emphasis on data volume is especially true in specialized domains like code, where researchers obtain massive code pretraining datasets by crawling platforms like GitHub [, , , , , ].
Recent studies, however, have increasingly showed that high-quality data is crucial for effective pretraining [, , , ], including the code domain [, , ].

In the general domain, researchers have explored various techniques to curate high-quality pretraining data for language models. FineWeb-Edu [] uses a linear regressor built on Snowflake-arctic-embed-m [] embeddings to assess the educational value of web pages and select high-quality content, while the DCLM [] approach employs a fastText-based [] filter trained on positive examples from high-quality online sources [] and instruction data [], and random negative web pages to identify high-quality text. These model-based quality filters have been shown to significantly enhance language model performance on downstream tasks, compared to using unfiltered, large-scale datasets. Similarly, researchers have recognized the importance of high-quality code data for pretraining, with Phi-1 [] using a random forest classifier on CodeGen [] embeddings to select educational code samples, and DeepSeek-Coder-V2 [] employing a multi-stage fastText-based [] pipeline to recall web-related code data and high-quality code from GitHub, achieving state-of-the-art coding performance.

In this paper, we introduce Arctic-SnowCoder-1.3B, a high-performing small code model created by a novel three-step training methodology focused on progressive improvements in data quality. As a result of this methodology, Arctic-SnowCoder-1.3B outperforms StarCoderBase-3B [] across all evaluated benchmarks and exceeds Phi-1.5-1.3B [] by 36% on the complex and practical BigCodeBench benchmark [], a benchmark that truly matters for real-world programming.
As shown in Figure 1, Arctic-SnowCoder is developed through a three-stage, data-efficient pretraining process that progressively refines the quality of the data used. The first stage involves general pretraining for a 500B token horizon using 400B unique raw code data, which have been preprocessed through basic filtering, deduplication, and decontamination. The 400B raw corpus is primarily derived from the coding data used to train Snowflake Arctic [],
combining cleaned The Stack v1 [] and GitHub crawls.
This is followed by continued pretraining on 50B tokens, utilizing a smaller, high-quality subset of 12.5B code files, repeated four times. The high-quality tokens are selected from phase one by a BERT-based [] quality annotator trained to distinguish good code from random data, using positive examples drawn from publicly available high-quality code files [], along with instruction data from Magicoder [] and StarCoder2-Instruct [].
Finally, the model undergoes an enhanced pretraining phase for 5B tokens, leveraging roughly 2B synthetic data generated by Llama-3.1-70B []. This process uses the phase two data as seeds and adapts the OSS-Instruct methodology from Magicoder [] by transforming lower-quality seed code into high-quality code documents.
Notably, all training phases of Arctic-SnowCoder derive data from the same raw pretraining corpus, ensuring that minimal new knowledge is introduced.

Arctic-SnowCoder-1.3B achieves state-of-the-art results on BigCodeBench [], a coding benchmark focusing on practical and challenging programming tasks, among models of similar size trained with $\leq$ 1T tokens. Particularly, it outperforming Phi-1.5-1.3B [] by 36%. Despite being trained on 555B tokens, compared to other state-of-the-art small code models trained on trillions of tokens, Arctic-SnowCoder matches or surpasses the performance of these models on several benchmarks. For instance, Arctic-SnowCoder-1.3B beats StarCoderBase-3B [], trained on over 1T tokens, across all evaluated benchmarks. Arctic-SnowCoder-1.3B outperforms StarCoder2-3B [], trained on over 3T tokens, on HumanEval+ [, ] (28.0 vs. 27.4), a benchmark evaluating function-level code generation, while remaining competitive on BigCodeBench (19.4 vs. 21.4).
We conduct comprehensive ablation studies to validate the design decisions behind training Arctic-SnowCoder:

- •
First, our findings indicate that, in general pretraining, organizing file-level data into repositories after partitioning by programming language significantly outperforms the approach of grouping data solely by repository names.
- •
Additionally, we determine the optimal learning rate schedule, which involves a re-warmup phase followed by linear decay, as well as the ideal repetition of high-quality data during continued pretraining, which we find to be four times.
- •
More importantly, our comparisons of model-based quality annotators, trained on various data combinations, highlight that the alignment of pretraining data with downstream tasks is crucial for achieving superior performance.

In summary, we make the following contributions:

- •
We introduce Arctic-SnowCoder-1.3B, a high-performing small code model trained on 555B tokens that benefits from progressive improvements in data quality.
- •
We demonstrate that high-quality data and synthetic data can significantly improve the model performance despite being seeded from the same raw corpus.
- •
For the first time, we demystify the notion of data quality in code pretraining by systematically comparing model-based quality annotators trained on different data combinations.
- •
We provide practical insights into optimal design choices for repo-level grouping in general pretraining, and optimal learning rate schedules and repetitions of high-quality data during continued pretraining, providing practical guidelines for future model development.

## 2 Arctic-SnowCoder

In this section, we provide a detailed explanation of the training methodology used for Arctic-SnowCoder-1.3B, as illustrated in Figure 1. We begin by discussing the composition of the raw training data in Section 2.1, followed by an overview of the general pretraining phase in Section 2.2. Next, we describe the continued pretraining process using high-quality data in Section 2.3, and finally, we elaborate on the enhanced pretraining with synthetic data in Section 2.4. The model architecture is based on Llama-2 [], with specific details provided in Section 2.

Table: Table 1: Model architecture details of Arctic-SnowCoder.

### 2.1 Raw data

The raw pretraining data used to train Arctic-SnowCoder-1.3B consists exclusively of code, primarily derived from the coding data used to train Snowflake Arctic []. This data combines cleaned versions of The Stack v1 [] and GitHub crawls. From this data, we select 18 popular programming languages for training, similar to StarCoder2-3B []. These languages include Python, Java, C++, C, JavaScript, PHP, C#, Go, TypeScript, SQL, Ruby, Rust, Jupyter Notebook, Scala, Kotlin, Shell, Dart, Swift, amounting to a total of 400B unique tokens.

### 2.2 General pretraining

In general pretraining, the model is trained for 500B tokens with a sequence length of 8,192 and a batch size of 512 using Adam []. The learning rate follows a cosine decay after a linear warmup of 600 iterations. We set the maximum learning rate to $5.3\times 10^{-4}$ and the minimum to $5.3\times 10^{-5}$, following DeepSeek-Coder [].
In this phase, we use the entire 400B raw data without applying additional quality filtering. We start by partitioning code files by programming language, grouping them by repository, and then concatenating them in random order, similar to the StarCoder2 [] approach.
In Section 3.3, we show the advantage of first partitioning code files by programming language.
We name the model produced by this phase as Arctic-SnowCoder-alpha.

### 2.3 Continued pretraining with high-quality data

After general pretraining, we continue pretraining Arctic-SnowCoder-alpha with 50B high-quality tokens sourced from the same raw pretraining corpus. The 50B high-quality tokens are formed by repeating 12.5B top-percentile code file tokens for 4 times scored by our code quality annotator.
Inspired by FineWeb-Edu [] and DCLM [],
we train a linear classification head on top of Snowflake-arctic-embed-m [], a state-of-the-art embedding model based on BERT [].
The training data comprises 300k positive examples, sampled from a blend of 220k high-quality open-source code files [], 80k high-quality instruction data from Magicoder [] and StarCoder2-Instruct [], and 300 randomly selected code documents from the pretraining corpus.
Prior research on code quality, such as Phi-1 [], often overemphasizes the “educational value” of code, skewing models towards simpler benchmarks like HumanEval+ [, ]. In Section 3.2, we show that our annotation leads to a more balanced enhancement of model capabilities.
Furthermore, given that these code documents typically exceed 1000 tokens, surpassing the BERT context window size of 512, we improve over FineWeb-Edu’s pipeline to calculate the score for each file by averaging the scores from the top, middle, and bottom sections as produced by the quality annotator.
In this phase, we rewarmup the learning rate for 1000 iterations from 0 to $5.3\times 10^{-4}$, the maximum pretraining learning rate, followed by a linear decay to 0. The model produced in this phase is referred to as Arctic-SnowCoder-beta. In Section 3.4, we validate all of our design choices.

### 2.4 Enhanced pretraining with synthetic data

In the enhanced pretraining stage, we generate even higher-quality data than in continued pretraining leveraging Llama-3.1-70B-Instruct [] and increase the Python mix ratio to approximately 50% while keeping the proportions of the other languages unchanged.
Phi-1 [] demonstrates that synthetic, textbook-like pretraining data can significantly enhance model performance. However, overemphasis on such data risks skewing the model’s distribution, potentially impairing its effectiveness in real-world coding tasks.
For example, we show in Section 3.2 that Phi-1.5 excels in HumanEval+ [, ] and MBPP+ [, ], which resemble textbook exercises, but performs less effectively on the more complex and practical coding tasks in BigCodeBench [].
To address this, we adapt the OSS-Instruct method from Magicoder [] for pretraining purposes. Originally, OSS-Instruct was originally designed to generate realistic instruction-tuning data by prompting a model to create question-answer pairs inspired by open-source code snippets. In contrast, we produce high-quality synthetic pretraining data by using Llama-3.1-70B-Instruct to generate high-quality and problem-solving oriented code files, seeded with code documents scored in the top percentile during the continued pretraining phase.
In Section 3.2, we demonstrate that each pretraining phase significantly outperforms the previous one, highlighting the effectiveness of progressively enhancing data quality.

## 3 Experiments

In this section, we compare Arctic-SnowCoder with state-of-the-art small language models and show performance boost over each pretraining stage (Section 3.2), evaluate two strategies of forming repo-level data in general pretraining (Section 3.3), and perform detailed ablation to justify our design choices in continued pretraining (Section 3.4).

### 3.1 Experimental setup

We consider the following four diverse programming benchmarks to comprehensively evaluate the code generation capability of different code models:

HumanEval [] and MBPP [] are the two most widely-used benchmarks for function-level code generation. We adopt their augmented version powered by EvalPlus [], with 80×/35× more test cases for rigorous evaluation. HumanEval+ and MBPP+ include 164 and 378 coding problems, respectively.

is a program synthesis benchmark suite created by evolving existing benchmarks into different targeted domains. We employ its five default transformation categories, namely difficult, creative, subtle, combine and tool_use, totaling 500 tasks.

evaluates LLMs with practical and challenging programming tasks. It has 1140 programming tasks, where each task in BigCodeBench is created through human-LLM collaboration, where the task quality is ensured by human experts.

We incorporate HumanEval+, MBPP+, EvoEval, and BigCodeBench for baseline comparison in Section 3.2. For the subsequent ablation studies in Sections 3.3 and 3.4,
we include the base versions of HumanEval and MBPP while omitting BigCodeBench for faster evaluation.
Throughout the experiments, we report the pass@1 metric [] using greedy decoding.

### 3.2 Baseline comparison and effectiveness of three-stage pretraining

Table: Table 2: Comparing Arctic-SnowCoder with state-of-the-art small language models (< 3B), divided by whether training compute > 1T tokens. Arctic-SnowCoder-alpha and Arctic-SnowCoder-beta are checkpoints after general pretraining and continued pretraining with high-quality data, respectively. Arctic-SnowCoder is the final checkpoint after enhanced pretraining with synthetic data.

### 3.3 Repo-level data in general pretraining

In the general pretraining phase, we adopt StarCoder2’s approach to group file-level data randomly into repositories through a random concatenation of file contents [].
In Section 3.3, we study two methods: (1) grouping files just by repository names, meaning that each training document can be a mix of multi-lingual code files if the repository is written in different languages, and (2) partitioning files into different programming languages before grouping them into repositories, meaning that each training document only focuses on one single language.

Table: Table 3: Comparison of two methods for grouping repo-level data for pretraining. (1) “Group by repo” treats each repository as a single training unit with possibly mixed languages, and (2) “Group by language and repo” partitions data by programming language before grouping by repository.

### 3.4 Design choices in continued pretraining

In continued pretraining, we source high-quality tokens from our pretraining corpus and train an improved base model. To obtain high-quality tokens, a model-based quality annotator is employed. In this section, we experiment with various design choices, including the training data for the annotator, the learning rate used in continued pretraining, and the optimal repetitions of high-quality tokens.

#### Model-based quality annotator

Similar to FineWeb-Edu [], we train a linear head on top of the Snowflake-arctic-embed-m [] embedding model to score each code file.
In Section 3.4, we experiment with 4 variants:

- •
Ann-Edu: We prompt Mixtral-8x7B-Instruct [] to annotate the educational value of each code file (1 to 5). 400k annotations are used to train a linear regression head.
For the following variants, similar to DCLM [], we sample negative documents randomly and change the positive parts only. A linear classification head is used instead.
- •
Ann-Ins: Positives are a mix of 100k educational data (3.5+) bootstrapped from Ann-Edu and 100k high-quality instruction data from Magicoder [] and StarCoder2-Instruct [].
- •
Ann-HQ: Positives are 220k open-source, synthetic, high-quality code files [].
- •
Ann-HQIns: Positives are a mix of 220k Ann-HQ training data and 80k instruction data from Magicoder [] and StarCoder2-Instruct [].

Table: Table 4: Comparison of downstream performance by applying model-based quality annotators trained with different recipes to 10B continued pretraining.

#### Learning rate schedule

We also study different learning rate schedules for continued pretraining in Section 3.4, including (1) a linear annealing starting from the minimum pretraining learning rate to zero, (2) a constant schedule using the minimum pretraining learning rate, and (3) a re-warmup to the maximum pretraining learning rate followed by a linear decay to zero.

Table: Table 5: Comparison of different learning rate schedules in 10B continued pretraining using Ann-HQIns. Here $\texttt{MIN_LR}=5.3\times 10^{-5}$ and $\texttt{MAX_LR}=5.3\times 10^{-4}$.

#### Repetitions of high-quality data

Finally, we scale up the token horizon from 10 billion to 50 billion in continued pretraining. One remaining question to address is determining the optimal repetitions for high-quality tokens. We experiment with repetitions ranging from 1 to 5, as shown in Section 3.4, by selecting the top percentile tokens ranked by Ann-HQIns.

Table: Table 6: Downstream performance with varying repetitions of high-quality data in 50B continued pretraining using Ann-HQIns.

## 4 Related Work

#### Code pretraining corpus for language models

Code data is essential to improving the reasoning capabilities of large language models (LLMs) [, , , , ]. Typically, researchers obtain massive code pretraining data by crawling from public platforms hosting code repositories such as GitHub [, , , , , ].
For example The Stack v1 [] is a 3.1 TB dataset consisting of permissively licensed source code mined from GitHub in 30 programming languages.
Its successor The Stack v2 [], built on the Software Heritage archive [], is an order of magnitude larger, with a raw dataset of 67.5 TB spanning 619 programming languages.
However, directly using these massive unfiltered code for pretraining is suboptimal, because the code documents may contain undesired contents or duplicates. Therefore, further preprocessing steps are needed to downscale the raw corpus, which can include deduplication [, , , , , , ], PII (Personally Identifiable Information) redaction [, , ], benchmark decontamination [, , , ], and model-based filtering [].
As an example, StarCoder2 [] selects only 3 TB of data for pretraining from the 67.5 TB total data available in The Stack v2.
The code pretraining corpus of Arctic-SnowCoder follows a similar preprocessing pipeline, comprising approximately 400B unique tokens from a mix of filtered The Stack v1 and GitHub crawls.

#### Model-based quality filtering

In addition to common preprocessing steps like deduplication and heuristic filtering, a recent trend is using model-based quality filters to select high-quality pretraining data.
Phi-1 [] employs a random forest classifier trained on top of the CodeGen [] embedding layer on GPT-4 annotations, to assess the educational value of files. This filter selects high-quality The Stack v1 and StackOverflow content, significantly enhancing coding performance.
FineWeb-Edu [] employs a linear regressor built on Snowflake-arctic-embed-m [], an advanced embedding model based on BERT []. This regressor, trained on 400k Llama-3 [] annotations rating the educational value (0-5) of FineWeb dataset documents, significantly enhances STEM performance.
DCLM-Baseline [] uses a fastText [] filter trained on positives from OpenHermes 2.5 [], high-scoring posts from r/ExplainLikeImFive, and random negatives. It outperforms FineWeb-Edu in top-10% selection.
DeepSeek-Coder-V2 [] follows DeepSeek-Math [] by leveraging a multi-stage fastText-based pipeline to recall high-quality code and math contents.
Llama-3 [] uses fastText for recognizing text referenced by Wikipedia [] and Roberta-based [] classifiers trained on Llama-2 [] predictions.
While prior work focuses on initial pretraining, Arctic-SnowCoder demonstrates that high-quality data from the pretraining corpus can significantly enhance model performance during continued pretraining. We are also the first to uncover the secret of data quality, revealing the importance of matching data distribution with downstream tasks.

#### High-quality code data for pretraining

Phi-1 [] is one of the first to study the impact of high-quality code data. It first uses a random forest classifier to filter out high-quality code data from The Stack v1 and StackOverflow, and then creates synthetic textbook-like data and exercises using GPT-3.5 [], showing significant coding performance with only 50B+ training tokens.
DeepSeek-Coder-V2 [], pretrained for around 14T tokens in total, achieves state-of-the-art coding performance, with a multi-stage fastText-based [] pipeline to recall web-related code data as well as high-quality GitHub code.
Arctic-SnowCoder utilizes a high-quality code annotator to extract high-quality code from pretraining datasets and generates synthetic files seeded from this high-quality data, adapting Magicoder OSS-Instruct [] into pretraining.

## 5 Conclusion

We introduce Arctic-SnowCoder-1.3B, a high-performing code model that underscores the critical importance of data quality in the pretraining process. Trained on 555B tokens, Arctic-SnowCoder-1.3B achieves competitive results with state-of-the-art small code models while using significantly fewer tokens. Our three-stage pretraining process begins with 500B tokens of general pretraining on a raw code corpus, followed by 50B high-quality tokens scored by a quality annotator, and concludes with 5B tokens of synthetic data for further enhancement. This work demystifies the notion of high-quality data in code pretraining by demonstrating the key to high-quality data is its alignment with the distribution of downstream applications. Additionally, the paper offers practical guidelines for repo-level data grouping, learning rate scheduling, and the repetition of high-quality data, paving the way for more efficient and effective code model development.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) Could you provide more detailed analysis or ablation studies on how the quality annotator specifically improves the model’s learning? What are the key features or patterns it identifies that contribute most to the performance boost?

2) How does the synthetic data generated by Llama-3.1-70B differ in quality or characteristics from the high-quality tokens selected by the annotator? Could you provide examples or metrics that highlight these differences?

3) Your results suggest that grouping by language before repository improves performance. Could you elaborate on why this approach works better? Have you considered any hybrid methods that combine cross-language learning with language-specific training?

4) Given the success of Arctic-SnowCoder-1.3B with 555B tokens, how do you envision scaling this approach for larger models or different domains? Are there diminishing returns or unique challenges you anticipate?

5) The paper focuses on benchmarks like BigCodeBench and HumanEval+. How do you ensure these benchmarks reflect real-world programming challenges? Have you considered any additional metrics or benchmarks that might better capture practical coding scenarios?
soundness: 3
strengths: Arctic-SnowCoder demonstrates remarkable strengths among small size model, particularly in achieving state-of-the-art results on BigCodeBench with a 36% performance improvement over Phi-1.5-1.3B, despite using only 555B tokens compared to models trained on trillions of tokens. Arctic-SnowCoder-1.3B outperforms StarCoderBase-3B across all benchmarks and surpasses StarCoder2-3B, trained on over 3.3T tokens, on HumanEval+ with a score of 28.0 compared to 27.4. The model also achieves competitive results on MBPP+ (42.9) and EvoEval (18.0), showing that it can match or exceed the performance of larger models like StableCode-3B and Granite-Code-Base-3B, which are trained on 1.3T and 4.5T tokens, respectively. These results, combined with thorough ablation studies, highlight the effectiveness of its three-phase pretraining strategy, focusing on high-quality and synthetic data, while providing concrete evidence of its efficiency and superior performance in practical and complex coding tasks.
summary: This paper introduces Arctic-SnowCoder-1.3B, a small code model trained on 555B tokens through a meticulously designed three-phase pretraining process. The first phase involves general pretraining with 500B tokens of standard-quality code, filtered and deduplicated. The second phase refines this with 50B tokens of high-quality code, identified using a BERT-based quality annotator trained on positive examples from curated open-source repositories and instruction datasets. In the final phase, 5B tokens of synthetic data are generated using Llama-3.1-70B, seeded from the high-quality data to further enhance model performance. The model achieves state-of-the-art results on BigCodeBench, significantly outperforming larger models on practical and challenging programming benchmarks such as HumanEval+ and MBPP+. The paper highlights the importance of progressively improving data quality and aligning it with downstream tasks, offering comprehensive evaluations, ablation studies, and practical insights into optimal pretraining strategies, such as learning rate schedules and data repetition, to maximize the efficiency of smaller language models in code generation tasks.
weaknesses: While the synthetic data significantly boosts performance, as seen in the 36% improvement over Phi-1.5-1.3B on BigCodeBench, an overreliance on synthetic data risks skewing the model’s understanding of practical coding tasks. Additionally, the performance on HumanEval+ (28.0) and MBPP+ (42.9), although impressive, shows only incremental improvements over models like StarCoder2-3B (27.4 on HumanEval+ and 49.2 on MBPP+), indicating room for optimization in handling more complex or diverse programming tasks. The quality annotator, trained on specific curated datasets, could introduce biases that may not adequately represent broader coding practices, potentially limiting its effectiveness across all programming domains.

The paper’s approach to handling repo-level data in the general pretraining phase is insightful but has room for further exploration. The authors compare two methods: grouping files by repository names and by language before repository. They conclude that partitioning by language yields better results, as evidenced by improved scores on HumanEval+ (17.1 vs. 12.8), MBPP+ (33.9 vs. 30.7), and EvoEval (7.4 vs. 7.0). This method ensures that training documents are more focused and homogenous, which likely aids the model in learning language-specific patterns effectively. However, this method might overlook the potential benefits of cross-language learning, especially in multi-language projects where inter-language interactions are critical. Future work could explore hybrid approaches that maintain language-specific grouping but occasionally incorporate multi-language contexts to enhance the model’s ability to handle real-world, polyglot codebases. Additionally, more granular investigations into the impact of repository size and the diversity of code snippets within a repository could provide deeper insights into optimizing repo-level data grouping for enhanced model performance.

In addition, my concern is that the paper presents compelling results among small language models, particularly with its strong performance on benchmarks like BigCodeBench and HumanEval+. However, the underlying reasons for achieving such high performance despite the relatively small training dataset (555B tokens) are not fully unpacked. While the authors attribute the success to the progressive refinement of data quality and the use of synthetic data, the detailed mechanisms by which these factors translate into superior model performance remain somewhat opaque. 

Finally, I suggest evaluating the model on the CodeMMLU benchmark, which could provide a broader assessment of the model’s capabilities across a diverse set of coding tasks, thereby offering more comprehensive insights into its strengths and potential areas for improvement.

[1] CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs, https://arxiv.org/abs/2410.01999

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: The authors address a critical aspect of language model development—high-quality data in the code domain—which is essential for improving model performance and applicability. For the experiments, Arctic-SnowCoder-1.3B demonstrates good results, achieving state-of-the-art performance on BigCodeBench and surpassing similarly sized models trained on up to 1 trillion tokens, including a notable 36% improvement over Phi-1.5-1.3B.

However, I have three concerns:

1. Limited Novelty: While the use of a data annotator to extract high-quality data for pretraining is a valuable approach, it is not entirely novel. Similar methodologies have been employed, such as using GPT-4 as a data annotator. This raises questions about the uniqueness of the authors' contributions.

2. Missing Baselines: The evaluation would benefit from the inclusion of additional baselines, such as OpenAI's GPT models. Comparing or discussing these established models would provide a more comprehensive context for assessing Arctic-SnowCoder's performance and highlight its relative strengths and weaknesses.

3. Lower than Phi-1.5-1.3B on HumanEval+ MBPP+ and EvoEval: Despite achieving strong performance on BigCodeBench, Arctic-SnowCoder-1.3B underperforms compared to Phi-1.5-1.3B on more general code generation tasks, such as HumanEval+, MBPP+, and EvoEval. Interestingly, Phi-1.5-1.3B achieved better results with less training data, which suggests that Arctic-SnowCoder's specialized pretraining on high-quality code tokens may not necessarily translate into better generalization across a broader range of benchmarks.
soundness: 3
strengths: + Important Area.

The authors address a critical aspect of language model development—high-quality data in the code domain—which is essential for improving model performance and applicability. 



+ Good Performance on BigCodeBench

Arctic-SnowCoder-1.3B demonstrates good results, achieving state-of-the-art performance on BigCodeBench and surpassing similarly sized models trained on up to 1 trillion tokens, including a notable 36% improvement over Phi-1.5-1.3B.
summary: The authors emphasize the critical role of high-quality data in the effective pretraining of language models, particularly within the code domain, while noting that the precise definition of "high-quality" remains inadequately explored. To address this, they introduce Arctic-SnowCoder-1.3B, a data-efficient base code model pretrained on 555 billion tokens through three phases of progressively refined data. The first phase involves general pretraining with 500 billion standard-quality code tokens, processed through basic filtering, deduplication, and decontamination. The second phase continues with 50 billion high-quality tokens, selected from the first phase by a BERT-style quality annotator trained to distinguish good code from random data, using high-quality code files and instruction data from Magicoder and StarCoder2-Instruct. The final phase employs 5 billion synthetic tokens generated by Llama-3.1-70B, using phase two data as seeds and adapting the Magicoder approach for pretraining. Despite the limited dataset, Arctic-SnowCoder achieves state-of-the-art performance on BigCodeBench, outperforming similarly sized models trained on up to 1 trillion tokens, including a 36% improvement over Phi-1.5-1.3B. Across various benchmarks, Arctic-SnowCoder-1.3B performs better than StarCoderBase-3B pretrained on 1 trillion tokens and matches the performance of leading small base code models trained on trillions of tokens.
weaknesses: 1. Limited Novelty: While the use of a data annotator to extract high-quality data for pretraining is a valuable approach, it is not entirely novel. Similar methodologies have been employed, such as using GPT-4 as a data annotator. This raises questions about the uniqueness of the authors' contributions.

2. Missing Baselines: The evaluation would benefit from the inclusion of additional baselines, such as OpenAI's GPT models. Comparing or discussing these established models would provide a more comprehensive context for assessing Arctic-SnowCoder's performance and highlight its relative strengths and weaknesses.

3. Lower than Phi-1.5-1.3B on HumanEval+ MBPP+ and EvoEval: Despite achieving strong performance on BigCodeBench, Arctic-SnowCoder-1.3B underperforms compared to Phi-1.5-1.3B on more general code generation tasks, such as HumanEval+, MBPP+, and EvoEval. Interestingly, Phi-1.5-1.3B achieved better results with less training data, which suggests that Arctic-SnowCoder's specialized pretraining on high-quality code tokens may not necessarily translate into better generalization across a broader range of benchmarks.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: There are some questions after reviewing the paper:
1. In line 190,  "increase the Python mix ratio to approximately 50% while keeping the proportions of the other languages unchanged.", why is Python set as the primary language data, and how can it be adjusted for other languages?
2. In line 295,  "We can observe that the second approach, which we finally adopt in general pretraining, performs significantly better than the first one.", could you further explain the reason behind this conclusion?
3. In line 351, "the key to high-quality data is essentially the alignment with downstream application distributions.", what is the difference between alignment and fine-tuning of pre-trained models? And what are the advantages of the method proposed compared to fine-tuning techniques?
soundness: 2
strengths: 1. This paper proposes a method for improving the performance of pre-training models by focusing on multi-stage data quality enhancement. It introduces a high-performing code model with low token usage. 
2. Additionally, the paper analyzestraining strategies, including emphasizing the preparation of training data files and the characteristics of learning rate scheduling.
summary: This paper introduces a  method that includes three different pre-training phases, combined with iterative improvements to the quality of training data. It presents a code model—Arctic-SnowCoder-1.3B—which demonstrates competitive performance compared to current small code models, while significantly reducing the number of tokens used. The paper also provides guidelines for repo-level data grouping, learning rate scheduling, and emphasizes the importance of high-quality data.
weaknesses: This paper primarily focuses on techniques for enhancing and filtering the quality of code training data, with a key emphasis on how high-quality, filtered data improves model performance. However, an important question arises: could this improvement come at a cost, such as reduced generalization ability on non-target domain tasks? 

Additionally, the paper should review some existing techniques for improving training data quality and, where appropriate, include comparative analyses to demonstrate the advantages of the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Regarding the openness of this work, will the authors consider making the data pipeline and models publicly available? This will greatly help future studies on Code LM pretraining.
2. Regarding the evaluation, can the authors provide some explanations as to why BigCodeBench results are omitted in most of the tables?
3. Can the authors share more insights on how current setups documented in the paper can be generalized to larger Code LMs (e.g., 10B+)?
soundness: 3
strengths: The authors documented various valuable practices of pretraining Code LMs from scratch, which can inspire future work in this direction:
- Training a BERT-based classifier to annotate code quality, which is very efficient compared to any LLMs-as-Judges approaches.
- Using re-warmup as the learning rate schedule is quite novel.
-  While previous studies [1] suggest that deduplicating the data will result in better model performance, training on the repeated high-quality code data can further improve the coding capabilities.

The authors also shared a few interesting findings:
- "Textbook" is not all you need, so improving "educational value" in the training data may not be optimal.
- Re-warmup performs much better than other conventional schedules, such as linear and constant.
- When the number of high-quality tokens is limited to 50B, the setup of 12.5B with four repetitions could be more optimal.

[1] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022, May). Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8424-8445).
summary: The authors provide practical findings through pretraining a small code language model (Code LM), which achieves SOTA performance on representative coding benchmarks like HumanEval, MBPP, and BigCodeBench, among the Code LMs with similar sizes.
weaknesses: There are a few weaknesses in this paper:
- The current evaluations are Python-only, and the evaluation on multilingual (programming languages) code generation may share more interesting findings.
- The authors only study the 1.3B models, which is considered a bit small. While I understand that pretraining LMs is very costly, is it possible for the authors to provide more motivation for studying the ~1B models?
- It is most likely that the data cannot be opened due to legal constraints.

Meta-review: This work presents a 1.3B coding-specialized model that demonstrates strong performance on benchmarks such as HumanEval, MBPP, and others. A key contribution lies in the innovative data curation process, leveraging a lightweight BERT classifier to filter high-quality data and later generating synthetic data using LLama-3.1-70B.

While reviewers recognize the value of the provided artifacts, including the model and data pipeline for the open-source community, the main concerns center on whether all components will be fully released due to proprietary constraints. Despite the authors expressing their intention to make a best-effort attempt to share as much as possible, the lack of a guarantee on the release has led to the leaning towards the rejection.

Decision: Reject