Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale

## Abstract

Abstract Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date.
However, these rules lack the flexibility to address the unique characteristics of individual example effectively.
Meanwhile, applying tailored rules to every example is impractical for human experts.
In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial
data refining capabilities comparable to those of human experts.
We introduce Programming Every Example ( ProX ), a novel framework that treats data refinement as a programming task , enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale.
Experimental results show that models pre-trained on ProX -curated data outperform either original data or data filtered by other selection methods by more than 2 % percent 2 2\% 2 % across various downstream benchmarks.
Its effectiveness spans various model sizes and pre-training corpora, including C4, RedPajama-V2 , and FineWeb .
Furthermore, ProX exhibits significant potential in domain-specific continual pre-training: without
domain specific design, models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving average accuracy by 7.6 % percent 7.6 \mathbf{7.6\%} bold_7.6 % over Mistral -7B, with 14.6 % percent 14.6 \mathbf{14.6\%} bold_14.6 % for Llama-2 -7B and 20.3 % percent 20.3 \mathbf{20.3\%} bold_20.3 % for CodeLlama -7B, all within 𝟏𝟎 10 \mathbf{10} bold_10 B tokens to be comparable to models like Llemma -7B trained on 𝟐𝟎𝟎 200 \mathbf{200} bold_200 B tokens.
Further analysis highlights that ProX significantly saves training FLOPs, offering a promising path for efficient LLM pre-training.
We are open-sourcing ProX with ≥ 𝟏𝟎𝟎 absent 100 \mathbf{\geq 100} ≥ bold_100 B corpus, models, and sharing all training and implementation details for reproducible research and future innovation. • HF Repo : https://huggingface.co/gair-prox • Code : https://github.com/GAIR-NLP/ProX

### 1 Introduction

Large Language Models (LLMs) have made significant strides in capabilities (Meta, ; Achiam et al., ; Anthropic, ; Reid et al., ), excelling in tasks such as creative writing (Yuan et al., ), complex reasoning (Wei et al., ; Kojima et al., ), and agentic task planning and execution (Fan et al., ; Park et al., ). Behind these, massive, high-quality pre-training corpora form the backbone of these models, equipping them with the essential knowledge and reasoning abilities crucial for a wide range of downstream tasks (Together, ; Penedo et al., ).

The Internet offers vast amounts of data, but much of it is noisy and unrefined, requiring extensive cleaning and quality enhancement before being applied for pre-training.
Previous works focus primarily on designing heuristic-based pipelines to lift data quality, such as
document filtering (Rae et al., ; Penedo et al., ; Soldaini et al., ) and
perplexity-based scoring methods (Together, ), relying heavily on human expertise and manual adjustments (Zhang et al., ). While widely adopted, these labor-intensive solutions are inherently limited by rule coverage and their inability to address every specific case. Recently, some efforts have explored leveraging LLMs for high-quality data acquisition. On the one hand, language models have been applied for data filtering or selection (Xie et al., ; Wettig et al., ; Yu et al., ; Dubey et al., ), but their role is largely limited to identifying low-quality documents without enabling fine-grained refinements (e.g., string-level). On the other hand, LLMs are also being used directly generating high-quality data, *i.e.*, data synthesis (Gunasekar et al., ; Li et al., ; Ben Allal et al., ). Unlike filtering, synthesis methods actively create or refine data to produce new documents, but they require substantial computational resources, limiting scalability. Despite their success, these methods can also inherit issues like hallucination (Maini et al., ), and assessing their correctness and completeness in an interpretable manner remains a challenge (Liu et al., ).

Standing at the intersection of data processing efficiency and data quality improvement, in this work, we propose ProX, a model-based framework for pre-training level data refinement.
ProX focuses on refining large-scale data with relatively smaller models, offering a more efficient alternative.
As shown in Figure 2, in practice, ProX first adapts
a small base language model (less than $1$B)
to data refining tasks via fine-tuning on seed data.
This ProX’s refining model then determines the appropriate operations for each example in the pre-training corpora
through versatile programs, including operations such as filtering, string normalization and noisy line removal.
Finally, the generated program is executed by a pre-defined
executor, producing refined corpus ready for pre-training.
In this way, ProX is empowered with language models to autonomously refine pre-training corpora, leveraging flexible function calls to enhance data quality.

Experimental results demonstrate that the proposed ProX framework consistently lifts data quality for pre-training.
Specifically, ProX achieves an average improvement of $2.1\%$ over $10$ downstream benchmarks and outperforms state-of-the-art data selection methods by over $2.0\%$.
Furthermore, ProX shows broad applicability across model sizes from $0.3$B to $1.7$B and shows consistent performance gains across diverse pre-training corpora of varying quality, including RedPajama-V2 (Together, ), C4 (Raffel et al., ), and FineWeb (Penedo et al., ).
In domain-specific continual pre-training, ProX yields an $11\%$ gain over OpenWebMath (Paster et al., ) for TinyLlama-1.1B and $7.6\%$ for Mistral-7B across $9$ mathematical tasks, with similar improvements seen on Llama-2-7B and CodeLlama-7B.
Beyond performance gains, results also suggest that pre-training on the refined corpus significantly boosts pre-training efficiency, achieving similar downstream performance with up to $\mathbf{20}\times$ less computing.
We believe it is worthwhile to scale up computing FLOPs for data refinement, which enables similar performance with much less training cost and offers a promising path for efficient LLM pre-training.

### 2 Approach: Programming Every Example

#### 2.1 Data Refinement Task Formulation

Given any document in the corpus $d\in\mathcal{D}$, such as an HTML extract or a textbook, we define data refinement as the process of transforming $d$ into $\hat{d}$, where $\hat{d}$ exhibits higher quality.
While it is challenging to formally define “higher quality” for pre-training data, we assume it can be described through qualitative improvements, such as the removal of advertisements, meaningless URL links, random code gibberish, and content lacking educational value, just as shown on the left side of Figure 2.
Specifically, we formulate this refining process as the generation of a data processing program $\mathcal{Z}$, conditioned on $d$.
The refined document $\hat{d}$ is then produced by executing program $\mathcal{Z}$ on the original document $d$.
For instance, the “string normalization” can be a very fine-grained process transforming noisy strings into clean ones with executor $\mathcal{E}$ and program $\mathcal{Z}_{\text{normalize}}$ :

$$ $\mathcal{E}(\mathcal{Z}_{\text{normalize}},d)=(s^{\prime}_{i})_{i=1}^{|d|}, \text{ where }s^{\prime}_{i}=\text{normalize}(s_{i})\text{ if }s_{i}\text{ needs normalization else }s_{i}$ (1) $$

Here,
$d=(s_{1},s_{2},...,s_{|d|})$ is the original document represented as a sequence of strings, and
normalize()
is our normalization function that maps certain strings to their normalized versions.
Moreover, the document filtering process can be regarded as a special case of such refining transformation where executing on $\mathcal{Z}_{\text{filter}}$ will lead to removing the whole document, *i.e.*, $\mathcal{E}(\mathcal{Z}_{\text{filter}},d)=\varnothing$.

Figure: Figure 2: An overview of ProX framework: (1) we adapt a base language model to perform data refinement; (2) ProX refining models are able to generate complex programs for each document, including document level filtering and more fine-grained chunk level refining; (3) A Python executor will execute the programs with the docs, producing the refined high-quality corpora.
Refer to caption: x5.png

In this manner, data quality improvement operations, such as data cleaning or normalizing, can be unified into the standardized function that applies a specific transformation or cleaning process to the document.
These operations can be represented as various instantiations of the general executor $\mathcal{E}(\mathcal{Z},d)$, where $\mathcal{Z}$ encodes the function calling snippets or heuristics for the specific task.

#### 2.2 ProX Framework

##### Overview

As shown in Figure 2, given any document $d$ as input, the ProX framework utilizes the language model itself with parameter $\theta$ to generate the data refinement program $\mathcal{Z}=f(\theta,d)$.
The snippet is executed within the executor
$\mathcal{E}$, producing the refined document $\hat{d}=\mathcal{E}(f(\theta,d),d)$.
We include two stages in the ProX framework, aiming to refine the data progressively, from rough to fine-grained.
These two stages are referred to as
document-level programming and chunk-level programming, as illustrated in Figure 2.
In each stage, the ProX refining model will generate programs $\mathcal{Z}_{\text{doc}}$ and $\mathcal{Z}_{\text{chunk}}$ that refine the corpora at varying levels of granularities.

##### ProX Program Design

The detailed program space design is also crucial for maximizing the capabilities of language models.
We believed designing such model-based operations should consider several realistic factors when scaling to large pre-training corpora:
(1) the model does not need to be very powerful or very large to handle these tasks, it only needs to recognize several patterns;
(2) the solution, though requiring more computing budget compared to heuristic-rule-based pipelines, still needs to be simple and efficient.
Under such consideration, we simply let the language models generate function calls without detailed implementations.
These design choices aim to balance functionality with the limitations of small language models, enabling effective document manipulation while maintaining simplicity and coherence.

**Table 1: ProX program design of document-level and chunk-level refining stage. For input, doc and chunk will be sent into the corresponding function as string-type inputs for execution.**
| Stage | Function Interface | Description |
| --- | --- | --- |
| Document<br>Level | drop_doc()$\rightarrow$ <None> | Delete the whole doc. |
| keep_doc()$\rightarrow$ <str> | Return the orignal doc. |  |
| Chunk<br>Level | remove_lines(line_start, line_end)$\rightarrow$ <str><br>$~{}~{}\triangleright$ line_start<int>, index of the first line to be removed<br>$~{}~{}\triangleright$ line_end<int>, index of the last line to be removed | Delete noisy lines from chunk;<br>Return chunk after removal. |
| normalize(source_str, target_str)$\rightarrow$ <str><br>$~{}~{}\triangleright$ source_str<str>, the noisy string pattern<br>$~{}~{}\triangleright$ target_str<str>, the string for replacement | Replace strings with normalized ones;<br>Return chunk after replacement. |  |
| keep_chunk()$\rightarrow$ <str> | Return the orignal chunk. |  |

The most fundamental operations we aim to perform on a document, are deletion and replacement.
We incorporate these types of operations across different programming stages aiming to refine the corpus with different granularities in ProX:
(1) In the document-level programming stage, we simply define the function drop_doc()to delete a document and keep_doc()to retain it.
(2) In chunk-level programming, we split the lengthy documents into smaller chunks
and apply fine-grained operations to these chunks.
These operations include deleting specific lines remove_lines() and replacing strings normalize(), providing flexibility in modifying content rather than simply dropping the whole document.
Also for high-quality chunks that do not require any modifications, we use the keep_chunk() function for flagging.
We present the detailed function definition in Table 1, which is also the generation space of ProX’s refining models.
While the individual functions may seem straightforward, their design space is flexible and capable of expressing complex rules previously developed by human experts as shown in Table 1.
In fact, these rules can be projected into the program space of ProX, showcasing that our approach not only simplifies but also enhances the rule-creation process, offering more systematic and scalable refinement capabilities.

##### ProX Execution

During the execution stage, the generated program snippets $\mathcal{Z}$ will be executed by the executor $\mathcal{E}$ to refine the document.
For simplicity and flexibility, ProX integrates Pythonic grammars, wrapping all operations into different function calling with parameters and implements these function in Python for later execution.
For example, in Figure 2, the document contains some noisy patterns including navigation bars, meaningless HTML links and page indexes.
The refining model will then generate programs to remove the corresponding lines and patterns.
In the document-level and chunk-level cleaning stage, ProX utilizes an independent refining model to generate programs with various function calls described in Table 1.
We believe this sequential approach ensures a structured and effective refinement, addressing the larger document noise first, and then focusing on finer-grained cleaning.

#### 2.3 Model Adaptation for ProX

Figure: Figure 3: The illustration of the model adaptation in ProX. We employ powerful LLMs (Llama-3) to annotate random seed documents with valid programs, and use this doc-program pairs to fine-tune a small base model, obtaining the refining model suitable for fine-grained data refining tasks.
Refer to caption: x6.png

It is generally difficult for base models to directly generate ProX programs.
In fact, even for the most powerful post-trained LLMs, generating custom API calls is relatively challenging at the current stage (Zhuo et al., ).
Thus, it will be necessary that we curate some seed data to adapt the model for these scenarios.
Under such consideration, we employ strong LLMs to annotate these operations via zero-shot and few-shot prompting, and then adapt our base model to these tasks by supervised fine-tuning (SFT).
We first use two additive scale scoring prompts (Yuan et al., ; Penedo et al., ) to split the corpus into kept documents and dropped documents.
And then we use large models to annotate fine-grained programs based on kept documents.
Specifically, we leverage the Llama-3 series of models (Dubey et al., ) for data collection and annotation.
In ProX, this data collection is performed only once, and all base models are adapted with the same curated data.
To ensure the reliability of the collected data, we also conduct necessary checks for grammar correctness and control the removal ratio threshold.
The detailed procedure for program synthesis and post-processing can be found in § A.1.

For simplicity, we directly use a small language model (*e.g.*, $0.3$B parameters) that we have trained on approximately $26$B tokens of original unrefined data as the base model, which also serves as the comparison baseline in subsequent experiments.
The adapted model’s performance is then evaluated using the F1 score on the split validation dataset, ensuring a robust assessment.
We select the highest-performing model checkpoint and employ the model to generate programs $\mathcal{Z}$, for each document or chunk of the dataset. These programs together with the documents are then executed using the corresponding function implementation, resulting in the final processed corpus.
Please see appendix for more training details (§ A.2), implementation for calculating the F1 score (§ A.3), and large scale inference (§ A.4).

### 3 Experiments

In this section, we first describe our experimental setup, then verify the effectiveness of each ProX stage and compare it with existing data selection methods tailored for pretraining corpus (§ 3.2).
We then apply ProX to various model sizes and corpora to demonstrate its broad applicability (§ 3.3).
Finally, we apply ProX to the mathematical domain, demonstrating its superiority and universality in domain-specific training (§ 3.4).

#### 3.1 Experiment Setup

##### Training Corpora

We utilize various corpora for both general and specific domain data in our experiments. For general domain data, we begin with RedPajama-V2 (Together, ), a preprocessed large-scale dataset of $30$ trillion tokens from diverse Internet sources, ready for pre-training. We further apply ProX on the C4 corpus (Raffel et al., ) with $198$ billion tokens and the FineWeb dataset (Penedo et al., ) containing $15$ trillion tokens, noted for high data quality. For specific domain experiments, we use OpenWebMath (Paster et al., ), a math-focused dataset with $15$ billion tokens.
Given the limitations in computational resources, we conduct experiments on a randomly sampled subset of the entire pre-training dataset.
See Table 7 (§ B.2) for sampling details.

##### Base Model Selection

Our pre-training experiments are conducted using various sizes of decoder-only language models. Detailed specifications of these models and all training recipes are provided in § B.3, especially in Table 8 and Table 9.

- 1.
To verify different stages’ effectiveness of ProX, we employ a $750$M sized model sharing Llama-2 architecture (Touvron et al., ), denoted as TLM-s, used for both pre-training from scratch and refining.
We also compare ProX with data selection methods using Pythia-410M/1B’s architecture (Biderman et al., ), as those employed in MATES (Yu et al., ).
- 2.
For further evaluation of ProX using different refining and base model sizes, we scale the model sizes from $350$M ($0.5\times$ smaller, denoted as TLM-xs) and $1.7$B ($2\times$ larger, denoted as TLM-m), all based on the Llama-2 architecture.
- 3.
For domain-specific continual pre-training, we select TinyLlama-1.1B (Zhang et al., ), Llama-2 (Touvron et al., ), CodeLlama (Rozière et al., ) and Mistral-7B (Jiang et al., ) as representative base models for their adequate training and solid performance.

##### Baselines

To ensure a fair comparison w.r.t. training cost, we keep all training hyperparameters, such as training steps and batch size, consistent across baselines, with only the data refining and selection pipelines differing.
We compare ProX to a series of baselines:

- 1.
In § 3.2, to verify ProX’s effectiveness, we first compare with ProX with regular pre-training over the raw RedPajama-V2 data.
We also introduce heuristic baselines used to curate the FineWeb corpora, which is the combination of three filtering strategies from C4 (Raffel et al., ), Gopher (Rae et al., ), and newly crafted rules (as FineWeb rules).
Apart from rule-based baselines,
we also introduce existing data selection techniques proposed in previous works, including
(1) importance resampling: DSIR (Xie et al., );
(2) model-based selection: DsDm (Engstrom et al., ), MATES (Yu et al., ), and QuRating (Wettig et al., ).
- 2.
In § 3.3, to test ProX on different model sizes and training corpora, we finally scale the TLM-m’s training tokens to $50$B over RedPajama-V2, C4, and FineWeb.
To show ProX efficiency, we then directly compare with models covering a variety of pre-training approaches including (1) large-scale pre-training: TinyLlama-1.1B (Zhang et al., ) trained on $3$T tokens;
(2) model pruning from existing models: (SheadLlama (Xia et al., ) pruned from Llama-2 and trained on extra 50B tokens);
(3) LLM synthesis (InstructionLM-1.3B (Cheng et al., ) trained on Mistral-7B generated data and cosmo-1.8B (Ben Allal et al., ) trained on Mistral-8x7B generated data).
- 3.
In § 3.4’s specific domain continual pre-training, apart from standard continual pre-training on TinyLlama-1.1B, Llama-2-7B, CodeLlama-7B, and Mistral-7B, we additionally introduce with well-known and strong baselines trained on public (or partially public) data, including Rho-1 (Lin et al., ), InternLM2-Math (Ying et al., ), Llemma (Azerbayev et al., ), and an internal checkpoint reported in DeepSeek-Math (Shao et al., ).

##### Evaluation Setup

We compare the base models’ performance over a vast of datasets for comprehensive evaluation:
(1) For general pre-training, we evaluate the performance across ten selected tasks using lighteval’s implementation (Fourrier et al., ), and report the zero-shot
accuracy; we have also included LM-eval-harness (Biderman et al., ) for fair comparison with data selection methods.
(2) For domain-specific continual pre-training evaluation, *i.e.*, mathematical related benchmarks, we use the same nine implementation and benchmarks used in Rho-1 (Lin et al., ) and evaluate all the base models with few-shot chain-of-thought (CoT) examples (Wei et al., ).
The selected evaluation benchmarks, number of evaluation examples, and full details can be found in § C.

#### 3.2 Verifying ProX ’s effectiveness

##### Verifying Effectiveness for Each ProX Operation

We first conduct a series of experiments to verify the effectiveness of each ProX operation. We begin by training TLM-s on the RedPajama-V2 raw data for approximately $26$B tokens (or $12.5$K steps) as the initial baseline.
Following Wettig et al. () and for convenience, we then sequentially apply the doc-level and chunk-level refining pipelines by fine-tuning the $0.7$B model itself.
We then perform large-scale program synthesis and execution using the refining models, resulting in $\mathcal{D}_{\text{Doc}}$ and $\mathcal{D}_{\text{Doc+Chunk}}$.
Such $2$-stage synthesis requires approximately $192$ A100-80G GPU hours for processing
$60$B tokens of data.
The resulting zero-shot downstream performance is presented in Table 2, including base models trained on the data produced by ProX refinement methods and different rule-based filtering methods.
Moreover, we visualize the dynamic benchmark performance in Figure 4, implying the consistent improvement of ProX over all baselines.
See § D.1 for full detailed results of all intermediate checkpoints.

These results show that ProX is highly effective, outperforming the raw corpus with an average boost of $2.5\%$, including significant improvements such as $7.6\%$ on ARC-E, $3.3\%$ on HellaSwag, and $2.1\%$ on MMLU.
We believe such consistent performance is significant given that these improvements were achieved even on benchmarks that are typically prone to performance instability, such as SIQA, WinoGrande, and CSQA.
By contrast, rule-based methods demonstrate relatively marginal overall improvement.
For instance, Gopher rules achieve only a $0.2\%$ boost, while C4 shows a modest $0.5\%$ improvement.
Furthermore, combining all three rules (as is done in constructing the official FineWeb corpus), does not lead to any larger enhancement in overall performance.

**Table 2: Zero-shot performance on $10$ selected tasks. All models use the same TLM-s architecture and are trained on RedPajama-V2. The doc-level (ProX-D) and chunk-level (ProX-C) refining are done by fine-tuning the raw data pre-trained model as a refining model. Bolded entries represent the best results. #Win represents the number of tasks where the method achieved the best performance.**
| Method | ARC-C | ARC-E | CSQA | HellaS | MMLU | OBQA | PIQA | SIQA | WinoG | SciQ | AVG | #Win |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Raw | 26.1 | 44.3 | 29.7 | 39.1 | 27.3 | 29.2 | 66.9 | 39.0 | 52.0 | 67.4 | 42.1 | 0 / 10 |
| Rule-based filtering: Go = Gopher rules, C4 = C4 rules, Fw = FineWeb rules. |  |  |  |  |  |  |  |  |  |  |  |  |
| Go | 25.7 | 44.0 | 31.3 | 40.2 | 27.3 | 29.0 | 66.3 | 39.0 | 51.2 | 68.9 | 42.3 | 0 / 10 |
| C4 | 25.0 | 46.0 | 31.0 | 40.5 | 27.1 | 29.2 | 68.5 | 40.5 | 51.7 | 66.6 | 42.6 | 2 / 10 |
| Fw | 25.2 | 46.8 | 32.6 | 39.6 | 27.2 | 29.0 | 66.5 | 39.4 | 52.4 | 69.2 | 42.8 | 2 / 10 |
| Go+C4+Fw | 25.2 | 43.9 | 30.0 | 41.9 | 27.5 | 31.0 | 67.0 | 39.9 | 51.9 | 65.3 | 42.3 | 0 / 10 |
| ProX (ours): D = Doc-level Programming, C = Chunk-level Programming. |  |  |  |  |  |  |  |  |  |  |  |  |
| ProX-D | 26.6 | 49.7 | 30.1 | 40.5 | 29.4 | 30.4 | 66.3 | 39.0 | 51.2 | 71.6 | 43.5 | 2 / 10 |
| ProX-D+C | 26.4 | 51.9 | 30.9 | 42.4 | 29.4 | 31.6 | 67.9 | 40.0 | 52.2 | 73.5 | 44.6 | 5 / 10 |

Figure: Figure 4: Downstream zero-shot performance w.r.t. different training steps: first $0.5$K, then evenly from $2.5$K to $12.5$K. Rule: the best performing FineWeb rule in Table 2.
Refer to caption: x7.png

##### Comparing with Data Selection Methods

Apart from comparing with heuristic methods, we also include
existing representative model-based data selection methods tailored for pertaining corpus to verify ProX’s effectiveness in Table 4, where we report both 0 0-shot and $2$-shot performance under the same settings used in MATES (Yu et al., ).
While we merely apply document-level stage (*i.e.*, ProX-D) which is indeed similar to data selection methods, we can see that ProX outperforms the strongest data selection method MATES, by $2.2\%$ and $2.5\%$ in 0 0-shot and $2$-shot average performance for $410$M model, and by $1.0\%$ and $2.0\%$ for $1$B model.
Additionally, ProX achieves the best performance on $7$ out of $8$ benchmarks tested, demonstrating its superiority over existing data selection methods. Full evaluation results are provided in Table 11 (§ D.2).

#### 3.3 Applying ProX across model sizes and pretraining corpora

In this section, we demonstrate that ProX can effectively benefit models beyond scale and across different corpora, showing potential for iterative pre-training improvements.

##### ProX works well across different scales.

We train a family of models from $350$M to $1.7$B (*i.e.*, TLM-xs, TLM-s, and TLM-m) on the same $26$B tokens used in § 3.2, and then fine-tune these models on doc-level and chunk-level tasks, obtaining refining models with different sizes.
We then apply these models in doc-level refining and chunk-level refining stages, and use the curated data for from-scratch pre-training.
We report in Table 3.3 the adaptation performance on refining tasks of different refining model sizes.
According to the validation performance, adapting ProX works well across all model sizes, all achieving $80\%$ F1 on doc-level
refinement, and $75\%$ F1
on chunk-level refinement.
We further train these models of different sizes from scratch using data produced by refining models of varying sizes.
In Figure 5, the results indicate that refining models of all sizes help improve performance over raw data, with a consistent absolute gap of $2\%$ over all base model sizes.
While in Figure 5, TLM-xs curated data shows slightly better downstream performance, it has a significantly lower token-level retention ratio ($\mathbf{23.2\%}$ vs. $\mathbf{28.8\%}$) compared to larger models as reflected in Table 3.3.
This implies that moderately larger models suggest a favorable balance between data quality and quantity.
These additional tokens likely provide more knowledge during pre-training without compromising downstream benchmark performance, showcasing an effective trade-off between data refinement and information preservation.

Figure: Table 4: Refining model’s performance on valid set and token retention ratio of original corpus.
Refer to caption: x8.png

##### ProX works well across pre-training corpora.

To assess the applicability of ProX across various pre-training corpora, we extend our experiments beyond RedPajama-V2 to include C4 (Raffel et al., ), and the recently released $15$-trillion-token pre-training corpus, FineWeb (Penedo et al., ). For consistency, we apply exactly the same ProX-xs refining models detailed in Table 3.3 to these corpora without constructing new SFT data for each corpus.
We conducted larger-scale experiments by training our model on approximately $50$ billion tokens, again achieving notable improvements. On ten downstream benchmarks, models trained on our method’s curated data showed improvements of $+2.0\%$ on RedPajama-V2, $+3.1\%$ on C4, and $+2.4\%$ on FineWeb.

##### ProX trains language models with much greater efficiency.

To demonstrate the non-trivial nature of these results, we compared models trained on ProX curated data against various models trained by different approaches.
These include models like
TinyLlama-1.1B-3T (trained directly on $3$ trillion tokens, about $\mathbf{60}\times$ of our training tokens and $\mathbf{40}\times$ training FLOPs),
SheadLlama-1.3B (denoted as S-Llama, a pruned version of Llama-2-7B, with extra training on $50$ billion tokens),
and models using LLM data synthesis, such as InstructionLM-1.3B (denoted as Inst-LM) and Cosmo-1.8B.
Our results, including TLM-m (ProX) and TLM-m (Raw), are presented alongside all these baselines in Figure 6.
On FineWeb, which is recognized for its high-quality data, TLM-m using ProX-refined data performs comparably to pruned models like SheadLlama-1.3B and TinyLlama-1.1B, despite their reliance on additional pruning techniques or much larger datasets.
Moreover, using much less inference-time computing overhead, our model surprisingly outperforms models that rely heavily on LLM data synthesis, underscoring ProX’s efficiency.
Notably, models like Instruct-LM-1.3B, trained on $100$ billion tokens leveraging a fine-tuned Mistral-7B synthesizer, and Cosmo-1.8B, trained on $180$ billion tokens (including $25$ billion tokens synthesized by Mistral-8x7B), require significantly more computational resources than ProX.

#### 3.4 Applying ProX to Domain-Specific Contiual Preraining

We also demonstrate the potential of ProX in the continual pre-training scenario, specifically, in the mathematical domain.
We apply the very same pipeline as in general domains to the already cleaned OpenWebMath corpus (Paster et al., ), aiming to further refine and mine the high quality and clean data from the vast web pages crawled in it.
We then adapt and apply ProX-xs series, which was initially trained on general text as described in § 3.3, and further adapted on math text for the doc-level and chunk-level refining tasks.
Finally, we obtain about $5.5$B tokens left after the document-level cleaning stage and about $4.7$B tokens left after the chunk-level refining stage.
We present the final mathematical evaluation results of models trained on the refined OpenWebMath in Table 5, with full evaluation results presented in § D.4.

**Table 5: OpenWebMath Continual Pre-training (CPT) Results. All models are tested using few-shot CoT prompts. Llemma and InternLM2-Math are continual pre-trained models from CodeLlama and InternLM2 (Team, ) with public available data, respectively. DeepSeek-LLM denotes an internal DeepSeek model, and the model trained on OpenWebMath introduced by Shao et al. (). Note that the unique tokens and training tokens in the column refer exclusively to the token numbers from math-specific corpora (calculated by corresponding tokenizers). ^†: MQA evaluation of InternLM2-Base is based on an alternative prompt due to non-prediction issues with the original prompt. The bolded entries represent the best results within the same base model.**
| Model | Size | Method | Uniq<br>Toks | Train<br>Toks | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Existing Continual Pre-training for Reference |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| DeepSeek-LLM | 1.3B | - | - | - | 2.9 | 3.0 | - | - | - | - | - | 19.5 | 15.6 | - |
| 1.3B | - | 14B | 150B | 11.5 | 8.9 | - | - | - | - | - | 29.6 | 31.3 | - |  |
| CodeLlama (Base) | 7B | - | - | - | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| 34B | - | - | - | 31.8 | 10.8 | 61.9 | 66.0 | 83.4 | 51.6 | 23.7 | 43.0 | 53.1 | 47.3 |  |
| Llemma | 7B | - | 55B | 200B | 38.8 | 17.2 | 56.1 | 69.1 | 82.4 | 48.7 | 41.0 | 45.4 | 59.4 | 50.9 (+21.8) |
| 34B | - | 55B | 50B | 54.2 | 23.0 | 67.9 | 75.7 | 90.1 | 57.9 | 49.8 | 54.7 | 68.8 | 60.1 (+12.8) |  |
| InternLM2-Base | 7B | - | - | - | 27.0 | 6.6 | 49.0 | 59.3 | 74.8 | 40.1 | 20.9^† | 19.0 | 28.1 | 36.1 |
| 20B | - | - | - | 50.6 | 18.8 | 72.5 | 75.9 | 93.9 | 45.4 | 33.1 | 53.7 | 59.4 | 55.9 |  |
| InternLM2-Math | 7B | - | 31B | 125B | 41.8 | 14.4 | 61.6 | 66.8 | 83.7 | 50.0 | 57.3 | 24.8 | 37.5 | 48.7 (+12.6) |
| 20B | - | 120B | 500B | 65.4 | 30.0 | 75.7 | 79.3 | 94.0 | 50.9 | 38.5 | 53.1 | 71.9 | 62.1 (+6.2) |  |
| Applying Data Refinement Approaches |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| TinyLlama (Base) | 1.1B | - | - | - | 2.8 | 3.2 | 10.9 | 18.0 | 20.2 | 12.5 | 14.6 | 16.4 | 21.9 | 14.7 |
| TinyLlama (CPT) | 1.1B | - | 15B | 15B | 6.2 | 4.8 | 22.3 | 36.2 | 47.6 | 19.3 | 11.6 | 20.7 | 25.0 | 21.5 (+8.1) |
| 1.1B | Rho | 15B | 9B^∗(^1^11Rho-1 only counts the selected tokens that are used for training (loss calculation).) | 7.1 | 5.0 | 23.5 | 41.2 | 53.8 | - | 18.0 | - | - | - |  |
| 1.1B | Rule | 6.5B | 15B | 4.5 | 2.8 | 17.5 | 29.4 | 39.3 | 15.1 | 12.4 | 19.4 | 25.0 | 18.4 (+3.7) |  |
| 1.1B | ProX | 5B | 15B | 9.0 | 5.6 | 23.8 | 41.9 | 56.9 | 22.2 | 15.6 | 26.8 | 31.2 | 25.7 (+11.0) |  |
| Llama-2 (Base) | 7B | - | - | - | 14.1 | 3.8 | 39.5 | 51.6 | 63.6 | 30.9 | 12.5 | 32.9 | 34.4 | 31.5 |
| Llama-2 (CPT) | 7B | - | 15B | 10B | 29.6 | 13.6 | 49.2 | 61.9 | 78.4 | 36.3 | 31.9 | 40.5 | 43.8 | 42.8 (+11.3) |
| 7B | ProX | 5B | 10B | 30.6 | 16.8 | 50.2 | 63.7 | 79.3 | 37.3 | 40.1 | 43.8 | 53.1 | 46.1 (+14.6) |  |
| CodeLlama (Base) | 7B | - | - | - | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| CodeLlama (CPT) | 7B | - | 15B | 10B | 31.1 | 14.8 | 51.4 | 62.1 | 81.2 | 33.6 | 30.4 | 40.5 | 43.8 | 43.2 (+14.1) |
| 7B | ProX | 5B | 10B | 35.6 | 17.6 | 55.8 | 67.9 | 82.7 | 41.3 | 38.9 | 42.6 | 62.5 | 49.4 (+20.3) |  |
| Mistral (Base) | 7B | - | - | - | 40.6 | 11.4 | 65.4 | 68.5 | 87.0 | 52.9 | 32.3 | 50.0 | 56.2 | 51.6 |
| Mistral (CPT) | 7B | - | 15B | 10B | 44.4 | 19.2 | 65.2 | 69.6 | 88.4 | 46.6 | 43.1 | 50.8 | 65.6 | 54.8 (+3.2) |
| 7B | ProX | 4.7B | 10B | 51.0 | 22.4 | 64.9 | 72.9 | 89.2 | 49.8 | 53.0 | 54.2 | 75.0 | 59.2 (+7.6) |  |

##### ProX boosts math continual pre-training efficiency vastly.

Without any domain-specific design, Table 5 shows that pre-training on OpenWebMath refined by ProX brings $11.0\%$ average performance improvements for base TinyLlama-1.1B, $14.6\%$ for base Llama-2, $20.3\%$ for base CodeLlama, $7.6\%$ for base Mistral, which clearly exceed the improvements of all baselines, including their counterparts pre-trained on the original corpus, under the same settings. It is also worth noticing that, applying the rule-based filtering method does not bring improvements; instead, it leads to a $3.1\%$ performance degradation compared to continual pre-training on the original corpus. This finding implies that there are no universal workable heuristics for all domains, highlighting the demands for automated pipelines just like ProX.
Moreover, compared with some existing state-of-the-art math continual pre-training models like Llemma and InternLM2-Math typically requiring hundreds of billions of tokens continual pre-training, our ProX demonstrates remarkable efficiency gains.
A more controlled comparison further highlights this efficiency: Llemma-7B, based on CodeLlama-7B, was trained on $200$B tokens, whereas our ProX, also starting from CodeLlama-7B, reaches similar performance levels with just $10$B tokens of training, indicating a $\mathbf{20}$ times reduction in training computes.
These results suggest that our approach may contribute to more efficient and accessible development of LLMs and could offer a new perspective in domain-specific model adaptation, potentially enhancing how to address specialized LLM in resource-constrained settings.

### 4 Analysis

#### 4.1 Impact on the original data

What changes occur in the corpora after applying ProX? We compare the document length distribution of the original corpus with that of the ProX-refined corpus in Figure 7.
In the general domain corpora (RedPajama-V2, C4, and FineWeb), the data refined by ProX exhibits a noticeable shift in the average number of tokens per document.
For instance, in RedPajama-V2, we observe that documents with fewer than $100$ tokens make up a significant portion of the corpus. After applying the ProX, the majority of documents contain more than $200$ tokens, with an average number of tokens per document increasing from $1217$ to over $2000$.
This suggests that very short documents may be noisy and lack sufficient meaningful information to be suitable for pre-training. This shift, however, is not observed in OpenWebMath, where the average number of tokens per document is already larger.
One possible reason for this outlier is that the OpenWebMath corpus is collected mostly from sources different from the general domain, *e.g.,* online forums like Stack Exchange, and academic publisher websites such as arXiv.
The noises of these sources can be quite different from general domains.
Further case studies on these documents are provided in § E.1.

Figure: RedPajama-V2
Refer to caption: x10.png

#### 4.2 Computing Overhead Analysis

Although ProX demonstrates promising results in downstream tasks, it is important to acknowledge that large-scale model inference still requires a substantial computing budget.
For example, as mentioned in § 3.2, and in Table 7, the RedPajama-V2 corpus used for training TLM-s was refined from about $60$B raw tokens.
As calculated in § E.2,
if we utilize ProX-XS for both two refining stages, the additional computational overhead will amount to approximately $C=5\times 10^{19}$ FLOPs,
which is equivalent to training an additional $12$B tokens on TLM-s and $5$B tokens on TLM-m. It is noteworthy that this overhead ratio keeps decreasing as model size increases, meaning that the relative computational cost diminishes for larger models.

Figure: Figure 8: FLOPs comparison for comparable downstream performance with/without ProX refining: 0.3B(Avg.Perf = 40.5), 0.7B (41.6), and 1.7B (42.9).(^2^22The train FLOPs for the base model (approximately $5.3\times 10^{19}$) used to create the refining model are excluded. This is because any pre-trained LLM can theoretically serve as the base for refinement. This also reflects ProX’s flexibility.)
Refer to caption: x14.png

In Figure 2, we compare the FLOPs consumed by checkpoints with similar performance, both with and without applying ProX, across three different model sizes.
As the model size increases, the proportion of inference FLOPs required for applying ProX decreases.
For the $0.7$B model, the total FLOPs when using ProX are already lower than without it ($6.3\times 1e19$ vs. $6.7\times 1e19$).
Notably, for the largest $1.7$B model, we achieve performance comparable to a model pre-trained on the original data, but with only $58\%$ of the total FLOPs. This demonstrates that refining methods like ProX not only enhances data quality but also becomes more computationally efficient as model sizes grow, reinforcing the value of allocating additional resources to refining pre-training data.

### 5 Related Works

##### Pre-training Data Processing

Raw data collected from public sources (*e.g.*, CommonCrawl) are noisy, and directly using these data can greatly hurt model performance; thus, it has been a common practice to execute extensive pre-processing before pre-training (Touvron et al., ; Together, ; Penedo et al., ).
The pipeline usually starts with document preparation, which includes URL filtering, text extraction, language-based filtering (Smith et al., ).
The remaining document will then undergo several quality checks with heuristic rules like overall length, symbol-to-word ratio, and other
criteria to determine whether it is kept, partially or fully aborted (Zhang et al., ; Dou et al., ; Qiu et al., ).
Finally, these documents are deduplicated using different matching methods, *e.g.*, fuzzy match like MinHash (Broder, ), or exact sequences matches (Penedo et al., ).
In ProX, we uses the language model for further data refining, outperforming heuristic rules with acceptable computational overhead.

##### Data Selection Methods

Data selection, slightly distinct from data processing, is more commonly applied in the later stages of large-scale data pre-processing.
In supervised fine-tuning (SFT), it typically involves selecting a much smaller subset of samples to minimize tuning overhead while maintaining performance (Liu et al., ).
Recent efforts have extended these selection strategies to the pre-training stage (Engstrom et al., ; Xie et al., ; Ankner et al., ; Sachdeva et al., ; Liu et al., ).
For instance, Wettig et al. () train a rater model to score documents on four quality criteria in SlimPajama (Soboleva et al., ) and conduct pre-training on a resampled subset based on scores.
MATES (Yu et al., ) apply a smaller model for estimating data influence during pre-training, enabling dynamic data selection schema.
Moreover, as mentioned in Llama-3 (Meta, ), Llama-2 models (Touvron et al., ) was used as text-quality classifiers that underpin Llama-3’s training data.
Instead of merely selecting documents, ProX enables more fine-grained operations within documents, contributing to further performance improvements.

##### Model-based Data Synthesizing

Another branch of research focuses on editing or rephrasing existing data with models to improve the data quality.
Fan et al. () use ChatGPT to rephrase several instruction-tuning datasets for a clear format based on massive scenario-based criteria.
Yue et al. () use LLMs to extract and refine $5$M QA pairs from web documents,
obtaining $10$M instruction-response pairs.
Synthesis techniques have also been applied in the pre-training phase such as the Phi series (Gunasekar et al., ; Li et al., ).
Recently, Maini et al. () and Cheng et al. () utilize off-the-shelf instruction-tuned models to paraphrase web documents in specific styles such as QA, and mix these synthetic rephrases with real data in pre-training.
Ben Allal et al. () further synthesize from mere seed topics, by prompting LLMs to generate pre-training samples in a cleaner format like textbooks. However, despite its success, it typically requires substantial computation to synthesize a pre-training-scale corpus, and more critically, it inevitably inherits flaws from the advanced model, also suffering from hallucination issues (Liu et al., ).
In this work, we focus on leveraging language models to lift data quality through the synthesis of executable and interpretable programs, rather than directly generating data. We demonstrate that ProX could clearly improve data quality at scale only with acceptable extra computing.

##### Inference Time Scaling

Recent trends in language models have begun to explore the potential of allocating additional computing at inference time, complementing the extensive computations already deviated to the pre-training and post-training phases. Several studies have demonstrated the potential of this approach, showing that smaller language models equipped with additional inference-time computing can perform comparably to, or even outperform, significantly larger models, evidenced across various domains, including code generation (Hassid et al., ; Brown et al., ), and math problem-solving (Snell et al., ; Wu et al., ). The significance of this approach has been further corroborated by OpenAI’s latest o1 model release (OpenAI, ). While these studies focus on scaling computing on test time, our work demonstrates an alternative perspective on inference computing scaling. We advocate for allocating computing to refine pre-training corpora, particularly given that Internet-based corpora have been extensively utilized in language model pre-training. Our proposed ProX demonstrates remarkable gains in pre-training efficiency by investing moderately additional compute in the corpus refinement, facilitating more efficient and accessible development of LLMs.

### 6 Conclusion

We introduced ProX, a framework that uses language models to refine pre-training data at scale through program generation.
Our extensive experiments show that ProX curated data improves model performance by over $2\%$ on various downstream benchmarks and is effective across different model sizes and pre-training datasets.
For domain-specific continual pre-training, models trained on ProX curated tokens also yield significant improvements in $20\times$ fewer tokens, and comparable to state-of-the-art models trained on $200$B tokens.
Further analysis also implies applying ProX can achieve similar results with less computing power for large-scale LLM pre-training.
In summary, these results demonstrate ProX’s potential for greatly improving data quality and reducing costs in language model training.

### 7 Implications and Future Directions

The strong results from ProX highlight the potential of automated data refinement to significantly improve model performance while reducing computational costs. By refining data more effectively, ProX opens new possibilities for improving training efficiency and achieving better results across a range of benchmarks.
Looking ahead, these results suggest several future directions.
First, incorporating additional refining operations like reformatting and rephrasing could further enhance data quality.
Second, improving efficiency by reducing model size and applying inference acceleration techniques is a key goal. Expanding ProX to domains like code and multilingual data is also promising.
Scaling up with more computational resources will allow for a thorough evaluation of its potential. Finally, we believe that prioritizing data refinement before pre-training can greatly improve training efficiency, and we encourage continued exploration in this area.

### Acknowledgement

We extend our profound gratitude to Shanghai AI Lab and Sea AI Lab for generously providing valuable computational resources, which were instrumental in the realization of this project.
Our sincere thanks also go to Mingxuan Wang and Jiaze Chen from ByteDance for their crucial support. We are deeply thankful to Ethan Chern from Shanghai Jiao Tong University and Yuqing Yang from University of Southern California for their early discussions and insightful contributions, and equally grateful
to Zhoujun Cheng from UC San Diego, Yiheng Xu and Tianbao Xie from University of Hong Kong, and Terry Yue Zhuo from Monash University for their valuable feedback,
to Guilherme Penedo and Loubna Ben Allal from Hugging Face for their guidance on hyper-parameter tuning,
to Zhibin Gou from Tsinghua University for providing advise on continual pre-training,
to Lyumanshan Ye for helping with illustrations and color scheme design.
Finally, special thanks go to Peiyuan Zhang from UC San Diego, representing the TinyLlama team, for providing a great open pre-training framework and supporting series of acceleration operators.
These collective wisdom and unwavering support have been pivotal to our project.
This project is supported by SJTU SEIEE - ByteDance Large Language Model Joint Laboratory, Shanghai Artificial Intelligence Laboratory.

## 

### Appendix

parttoc
\parttoc

### Appendix A ProX Implementation Details

#### A.1 Supervised Fine-tuning Data Collection

In this section, we elaborate the detailed prompts used to generated the SFT data for model adaptation.
In principle, We apply the same prompts for general domain corpora (including C4 [Raffel et al., ], RedPajama-V2 [Together, ], FineWeb [Penedo et al., ]) and mathematical corpus (OpenWebMath [Paster et al., ]).
And all seed data is randomly sampled from the raw corpora.

##### Document-Level Programming

We apply two zero-shot scoring prompts to evaluate and assign a combined score to each web document before synthesizing the (doc, program) pair.
One of the prompts is the same as the one used in FineWeb-Edu, which is a prompt to let the model decide the educational score.
Additionally in ProX, we add a new format scoring prompt, focusing on the format and structure of the document.
Both prompts follow the additive style proposed by Yuan et al. [].
Given these prompts, the language models generate short critiques and assign a score between 0 0 and $5$.

In FineWeb-Edu, documents are retained only if the educational score (Edu Score) is greater than $2$. However, this approach is too aggressive when attempting to preserve a larger portion of the tokens.
For instance, FineWeb-Edu retains only 1.3 trillion tokens out of the original 15 trillion in the FineWeb corpus.
To recall more documents, we relax the filtering criteria by incorporating the format score as follows:

$$ $\text{Filtering Criteria $=$}\begin{cases}\text{Edu Score}\geq 3,&\text{keep document;}\\ \text{Edu Score}=2\text{ and }\text{Format Score}\geq 4,&\text{keep document;} \\ \text{Edu Score}<2,&\text{drop document.}\end{cases}$ (2) $$

Finally, we use Llama-3-70B-Instruct to annotate $51$K data, splitting $5$K for validation (^3^33In the earlier stage of experiments, we found that a dataset of thousands of data points (i.e., 5K) is also sufficient to equip the model with the “programming” abilities. This generally holds true for both document-level and chunk-level programming tasks. Scaling the dataset size could enhance the model’s robustness across various documents.).

The FineWeb-Edu prompt and our format scoring prompts are presented in Figure 9.

##### Chunk-level Programming

We apply chunk-level programming for more fine-grained operations.
We find three very popular patterns that keep occurring in all corpus: (1) menu, navigation bars at the top of the document; (2) button, html elements, links; (3) footers.

In general, LLMs work well given within $5$ few-shot examples.
But to generate these program snippets more accurately, we apply few-shot prompting with Llama-3-70B-Instruct for each type of noise.
We merge these programs aiming to clean different types of noises, perform some grammar checking, and make them the final data for training and validation during the chunk-level refining stage.
The annotated source comes from the same seed document used in the previous document filtering stage, accumulating to about $57$K data, of which $5$K is split as validation.

After the release of Llama-3.1-405B-Instruct, We also try to use only one prompt aiming to remove all the noises.
However, we find such practices lead to aggressive removal of the original document, often making the document less coherent.
Finally, we decide to only keep the head part and tail part of the program generated by Llama-3.1-405B-Instruct, which is previously mentioned in FinGPT [Luukkonen et al., ], and merge with the previous programs generated by Llama-3-70B-Instruct.

The few-shot prompts used to generate program snippets are presented in Figure 10, Figure 11 and Figure 12.

##### Comparison with FineWeb-Edu’s Approach

Compared with the recently released FineWeb-Edu, which also uses model-based scoring by applying a BERT model to evaluate documents, we find that our relaxed design retains more tokens without compromising overall data quality. Specifically, FineWeb-Edu retains about $1.3$ trillion tokens out of a $15$ trillion token corpus (less than $9\%$), while ProX curation typically keeps $23\%$ to $28\%$, providing up to $\mathbf{3\times}$ more unique tokens for training.

Moreover, we conducted a preliminary study by training $0.7$ billion parameter models on these data. We found that models trained on our curated data achieved similar downstream performance, as shown in Table 6. Therefore, we believe our current strategy is more suitable for large scale pre-training, as it is capable of retaining more tokens while maintaining very high data quality.

**Table 6: Comparing FineWeb-Edu with our strategy on TLM-s.**
| Methods | Kept Ratio | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG | #Win |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FineWeb-Edu | 8.6% | 30.3 | 58.7 | 29.0 | 42.0 | 30.4 | 31.8 | 67.7 | 38.1 | 50.4 | 73.3 | 45.2 | 5/10 |
| FineWeb-ProX | 28.0% | 27.7 | 55.7 | 30.4 | 44.2 | 29.5 | 31.0 | 68.8 | 39.3 | 52.2 | 72.8 | 45.2 | 5/10 |

Figure: Figure 9: Edu scoring prompts used in FineWeb [Penedo et al., ] and newly proposed “format scoring” prompts for ProX.

Figure: Figure 10: Few-shot navigation bar removal prompts.

Figure: Figure 11: Few-shot URL removal prompts.

Figure: Figure 12: Few-shot footer removal prompts.

#### A.2 Supervised Fine-tuning Details

##### Training Parameters

We use llama-factory [Zheng et al., ] as our main code base for Adaptation Stage. We apply full paraemter supervised fine-tuning on our base models: we train on the whole seed dataset for $3$ to $5$ epochs, with batch size as 64, and cosine learning rate schedular (lr from 1e-5 $\rightarrow$ 1e-6).
Also, we find that base model convergent quite fast on these tasks, thus we do not apply a further tuning over hyper-parameters, and keep the same training configurations for all the adaptation tasks.

#### A.3 Evaluation Metrics for ProX Refining Tasks

##### Document-level refining Task

The document filtering task is indeed equal to a binary classification problem, where documents are classified as either to be kept ($1$) or dropped (0 0).
We evaluate the performance using the F1 score, calculated as follows:

$$ $\text{F1}=2\cdot\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+ \text{Recall}}$ (3) $$

where:

$$ $\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}},\quad\text{Recall}= \frac{\text{TP}}{\text{TP}+\text{FN}}$ (4) $$

The F1 score ranges from 0 to 1 and we assume higher F1 score indicates better classification performance.

##### Chunk-level Refining Task

This task actually contains two parts: line removal and string normalization.
However, we find it is rather hard to evaluate the normalization task, so we use the line removal accuracy to reflect the refining performance.
We propose a line-wise F1 score metric:

The F1 score is computed by comparing the predicted noisy lines with the labeled noisy lines.
First, we extract the noisy line indexes from both the prediction and the label.
Then, we calculate the overlap between these two sets. The true positives (TP) are the number of lines in this overlap.
False positives (FP) are the predicted noisy lines that are not in the labeled set, and false negatives (FN) are the labeled noisy lines that are not in the predicted set.
The calculation is actually simple:

$$ TP (True Positives) $\displaystyle=$ $\displaystyle|\text{Predicted Noisy Lines}\cap\text{Actual Noisy Lines}|$ (5) FP (False Positives) $\displaystyle=$ $\displaystyle|\text{Predicted Noisy Lines}\setminus\text{Actual Noisy Lines}|$ (6) FN (False Negatives) $\displaystyle=$ $\displaystyle|\text{Actual Noisy Lines}\setminus\text{Predicted Noisy Lines}|$ (7) $$

Then we use same calculation of F1 score mentioned before, i.e., $\text{F1}=\frac{2\cdot\text{TP}}{2\cdot\text{TP}+\text{FP}+\text{FN}}$.

#### A.4 ProX Inference at Scale

Thanks to the Datatrove project [Penedo et al., ], we are able to efficiently split, and load the whole corpus to each worker (which normally equals to the number of the GPUs since small models do not require tensor parallelism).
We use the vllm [Kwon et al., ] to perform large scale inference.

For chunk-wise programming, we will split the original document into several chunks, controlling the tokens of each chunk less than the context window.
In practice, we normally replace token count process as a word count process for saving time, and control the window size as $1,500$.
The general algorithm is implemented as below:

Figure: Algorithm 1 Document Chunk Splitting Algorithm

### Appendix B Pre-training Details

#### B.1 Training Infrastructure

##### Code Base

Thanks to litgpt [AI, ], and TinyLlaMA [Zhang et al., ], we are able to flexibly train all our base models.
We inherit several fused kernels from the TinyLlaMA, which is installed from the FlashAttention [Dao, ] including fused rotary positional embedding (RoPE), layer normalization, and cross entropy loss to help saving memory.
We mainly apply FSDP strategy [Zhao et al., ] to enable training larger scale models on multiple nodes.

#### B.2 Pre-training Corpora

Due to computing constraints and fair comparison purpose, we cannot exhaustively train over the whole corpora.
Thus, we apply random sampling for some of the pre-training corpora and make them as our pre-training data pools.

- •
For RedPajama-V2, We randomly download $70$ file shards, obtaining a total data pool consisting about $500$B tokens, we evenly separate it into $8$ dumps, with each containing about $62.5$B tokens; due to computing constraints, we use only $1$ dump for verifying effectiveness (Section 3.2) and use $2$ dumps for scaling the training to $50$B tokens (Section 3.3);
- •
For C4, we download the whole dataset, which contains about $198$B tokens;
- •
For FineWeb, we download the official $350$B sample(^4^44[https://huggingface.co/datasets/HuggingFaceFW/fineweb/tree/main/sample/350BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb/tree/main/sample/350BT));
- •
For OpenWebMath, we download the whole dataset.

We report the corpora details applied in each experiment in Table 7.

**Table 7: The detailed breakdown for pre-training corpora in all experiments.**
| Section | Experiments | Source | Data Description | Corpora Size (B) | Effective Train Tokens (B) | Epoch |
| --- | --- | --- | --- | --- | --- | --- |
| Section 3.2 | Table 2, Figure 4 | RedPajama-V2 | raw data size | 62.5 | 26.2 | 0.42 |
| after rule-based filtering | 31.5 | 0.83 |  |  |  |  |
| after ProX-D | 19.0 | 1.38 |  |  |  |  |
| after ProX-D+C | 16.0 | 1.64 |  |  |  |  |
| Section 3.2 | Table 4 | C4 | random | - | 26.2 | - |
| after ProX-D | 41.5 (GPT-NeoX) | 0.63 |  |  |  |  |
| other baselines | - | - |  |  |  |  |
| Section 3.3 | Figure 5 | RedPajama-V2 | raw data size | 62.5 | 26.2 | 0.42 |
| after ProX-D+C (using ProX-xs) | 14.5 | 1.80 |  |  |  |  |
| after ProX-D+C (using ProX-s) | 16.0 | 1.64 |  |  |  |  |
| after ProX-D+C (using ProX-m) | 18.0 | 1.46 |  |  |  |  |
| Section 3.3 | Figure 6 | C4 | raw data size | 198.0 | 52.4 | 0.53 |
| after ProX-D+C (using ProX-xs) | 44.5 | 1.18 |  |  |  |  |
| RedPajama-V2 | raw data size | 123.5 | 0.42 |  |  |  |
| after ProX-D+C (using ProX-xs) | 29 | 1.81 |  |  |  |  |
| FineWeb | raw data size | 79.0 | 0.66 |  |  |  |
| after ProX-D+C (using ProX-xs) | 18.0 | 2.91 |  |  |  |  |
| Section 3.4 | Table 5, 1.1B model | OpenWebMath | raw data size | 15.0 | 15.7 | 1.05 |
| after rule-based filtering | 6.5 | 2.40 |  |  |  |  |
| after ProX-D | 5.5 | 2.85 |  |  |  |  |
| after ProX-D+C | 4.7 | 3.49 |  |  |  |  |
| Section 3.4 | Table 5, 7B model | OpenWebMath | raw data size | 15.0 | 10.5 | 0.70 |
| after ProX-D | 5.5 | 1.91 |  |  |  |  |
| after ProX-D+C | 4.7 | 2.23 |  |  |  |  |

#### B.3 Model Configuration and Training Parameters

##### Model Architecture

The models we used in general and continual pre-training are presented at Table 8 with detailed architecture configuration.

##### Training Hyperparameter Choice

We primarily use a cosine learning rate scheduler and follow established settings used in Zhang et al. [] and Lin et al. [].
The default configurations for each experiment can be found below and we elaborate full details in Table 9.

- 1.
For general pre-training experiments, we set the learning rate to 5e-4 for TLM-xs and TLM-s, 3e-4 for TLM-m; the maximum sequence lengths are uniformly set to 2048, and the global batch size is set to 2M tokens.
- 2.
Additionally, we align all our hyper-parameters with those used in MATES [Yu et al., ] to facilitate a direct comparison with their existing data selection methods, as previously shown in Table 4.
In this case, we switch to the warmup-stable-decay (WSD) learning rate scheduler [Hu et al., ], as implemented in MATES.
For fair comparison with baselines implemented in MATES, we apply the exact same WSD Schedular [Hu et al., ]:
$lr(t)=\begin{cases}\frac{t}{W}\cdot\eta,&\text{if }t<W\\
\eta,&\text{if }W\leq t<S\\
0.5^{4\cdot(t-S)/D}\cdot\eta,&\text{if }S\leq t<S+D\end{cases}$
(8)
where $W$ equals to 2000, $S$ equals to 50000, $D$ equals to 200.
- 3.
For continual pre-training experiments, we set different hyperparameters for different base models, as shown in Table 9. We apply an early-stop mechanism mentioned in InternLM2-Math [Ying et al., ] for 7B model experiments.
We mainly refer these settings to the setup reported in Rho-1 [Lin et al., ] and Llemma [Azerbayev et al., ]. We do not use warmup in continual pre-training experiments.

**Table 8: The details of the pre-training experiments’ model architecture.**
| Model | Hidden Size | Intermediate Size | Context Len | Heads | Layers | Vocab Size | # Params (w/o embed) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Training From Scratch |  |  |  |  |  |  |  |
| TLM-xs | 1,280 | 2,048 | 2,048 | 16 | 24 | 32,000 | 354,284,800 (313,324,800) |
| TLM-s | 1,536 | 4,864 | 2,048 | 24 | 24 | 32,000 | 758,982,144 (709,830,144) |
| TLM-m | 2,048 | 8,192 | 2,048 | 32 | 24 | 32,000 | 1,741,785,088 (1,676,249,088) |
| Pythia-410M | 1,024 | 4,096 | 1,024 | 16 | 24 | 50,304 | 405,334,016 (353,822,720) |
| Pythia-1B | 2,048 | 8,192 | 1,024 | 8 | 16 | 50,304 | 1,011,781,632 (908,759,040) |
| Continual Pre-training |  |  |  |  |  |  |  |
| TinyLlama-1.1B | 2,048 | 5,632 | 2,048 | 32 | 22 | 32,000 | 1,100,048,384 (1,034,512,384) |
| Llama-2-7B | 4,096 | 11,008 | 4,096 | 32 | 32 | 32,000 | 6,738,415,616 (6,607,343,616) |
| CodeLlama-7B | 4,096 | 11,008 | 4,096 | 32 | 32 | 32,016 | 6,738,546,688 (6,607,409,152) |
| Mistral-7B | 4,096 | 14,336 | 4,096 | 32/8 (GQA) | 32 | 32,000 | 7,241,732,096 (7,110,660,096) |

**Table 9: Training hyper-parameters of all base models.**
| Model | Context<br>Length | Batch Size | Max Steps | Warmup<br>Steps | Weight<br>Decay | Optimizer | LR<br>Scheular | LR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training from Scratch |  |  |  |  |  |  |  |  |
| TLM-xs | 1,024 | 2,048 | 12,500 | 500 | 0.1 | AdamW | cosine | 5e-4 $\rightarrow$ 5e-5 |
| TLM-s | 1,024 | 2,048 | 12,500 | 500 | 0.1 | AdamW | cosine | 5e-4 $\rightarrow$ 5e-6 |
| TLM-m | 1,024 | 2,048 | 12,500/2,5000 | 500 | 0.1 | AdamW | cosine | 3e-4 $\rightarrow$ 3e-5 |
| Pythia-410M | 512 | 1,024 | 50,200 | 2,000 | 0.1 | AdamW | WSD | 1e-3 $\rightarrow$ 6.25e-5 |
| Pythia-1B | 512 | 1,024 | 50,200 | 2,000 | 0.1 | AdamW | WSD | 1e-3 $\rightarrow$ 6.25e-5 |
| Continual Pre-training |  |  |  |  |  |  |  |  |
| TinyLlama-1.1B | 2,048 | 1,024 | 7,500 | 0 | 0.1 | AdamW | cosine | 8e-5 $\rightarrow$ 8e-6 |
| Llama-2-7B | 4096 | 256 | 15,000 (early stop at 10,000) | 0 | 0.1 | AdamW | cosine | 8e-5 $\rightarrow$ 8e-6 |
| CodeLlama-7B | 4096 | 1024 | 3,750 (early stop at 2,500) | 0 | 0.1 | AdamW | cosine | 3e-4 $\rightarrow$ 3e-5 |
| Mistral-7B | 4,096 | 256 | 15,000 (early stop at 10,000) | 0 | 0.1 | AdamW | cosine | 2e-5 $\rightarrow$ 2e-6 |

### Appendix C Downstream Tasks Evaluation

#### C.1 General Pre-training Evaluation

##### Lighteval Configurations

We mainly borrow the evaluation benchmarks from the FineWeb’s nine selected “early signal” tasks [Penedo et al., ], and use the implementation of lighteval [Fourrier et al., ] to test all our base models.
We also introduce SciQ [Welbl et al., ] which is widely used in previous works and proved a good testbed [Mehta et al., , Wettig et al., ].
By default, we report the normalized zero-shot accuracy.
All the nine benchmarks at listed below:

- •
ARC [Clark et al., ]: including ARC-Easy (ARC-E) and ARC-Challenge (ARC-C)
- •
CommonSense QA [Talmor et al., ] (CSQA)
- •
HellaSwag [Zellers et al., ]
- •
MMLU [Hendrycks et al., ]
- •
OpenBook QA [Mihaylov et al., ] (OBQA)
- •
PIQA [Bisk et al., ]
- •
SocialIQA [Sap et al., ] (SIQA)
- •
WinoGrande [Sakaguchi et al., ] (WinoG)
- •
SciQ [Welbl et al., ]

We follow the lighteval’s configuration, which randomly picks $1,000$ samples for each dataset (for MMLU, it selects $1,000$ samples for each of the $57$ subsets), and report the normalized accuracy.
These average performance is calculated over the nine benchmarks, where ARC-C and ARC-E are considered as two separate benchmarks, and MMLU is treated as a single benchmark.
This approach differs slightly from the aggregation score calculation in FineWeb, as we believe MMLU’s performance is relatively unstable, and we aim to give equal weight to all benchmarks, preventing MMLU from becoming a dominant factor.
For the original lighteval scores, please refer to the §D.1, where we include a dynamic result curve that clearly illustrates the fluctuations in each benchmark.

We present zero shot evaluation results in Table 2, Figure 4.

##### LM-Eval Harness Configurations

We also include the lm-evel-harness [Biderman et al., ] for zero-shot and few-shot performance, for fair comparison with different data selection methods including DSIR [Xie et al., ],DsDm [Engstrom et al., ], Qurating [Wettig et al., ] MATES [Yu et al., ].
Similar to lighteval configuration, we include:

- •
ARC: including ARC-E and ARC-C
- •
HellaSwag
- •
LogiQA [Liu et al., ]
- •
OpenBook QA (OBQA)
- •
PIQA
- •
WinoGrande (WinoG)
- •
SciQ

We exclude the BoolQ [Clark et al., ] tasks from MATES [Yu et al., ], leaving eight tasks in total. This decision was made because we observed that the BoolQ benchmark performance exhibited severe fluctuations and showed a notable declining trend in the early stages. Therefore, we decided to exclude it from our evaluation set.
Such trend is also observed earlier in the OpenELM work [Mehta et al., ].
We report both zero-shot and two-shot performance. If the metrics include normalized accuracy, we use that measure; otherwise, we use accuracy.

#### C.2 Continual Pre-training Evaluation

We evaluate all benchmarks implemented in the math-eval-harness repository,(^5^55[https://github.com/ZubinGou/math-evaluation-harness](https://github.com/ZubinGou/math-evaluation-harness)) including:

- •
Math (MATH) [Hendrycks et al., ]
- •
GSM8K [Cobbe et al., ]
- •
SVAMP [Patel et al., ]
- •
ASDiv [Miao et al., ]
- •
MAWPS [Koncel-Kedziorski et al., ]
- •
MathQA (MQA) [Amini et al., ]
- •
TableMWP (TAB) [Lu et al., ]
- •
SAT MATH [Azerbayev et al., ]

We use few-shot CoT prompting [Wei et al., ] when evaluating these tasks, and report the accuracy of each task.

### Appendix D Full Evaluation Results

#### D.1 Detailed Performance on 10 Benchmarks in Sec 3.2

We report full evaluation results of checkpoints saved at different training steps in Section 3.2.
We present the results for 0.7B models trained on data curated by different methods in Table 10, including models trained on raw data, rule-based filtered data, and data curated by ProX.

**Table 10: Full evaluation results on TLM-s.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Raw Data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.1 | 39.0 | 27.6 | 31.6 | 25.9 | 26.6 | 61.2 | 37.3 | 48.9 | 59.1 | 37.9 |
| 5000 | 24.4 | 41.2 | 28.8 | 34.8 | 26.7 | 27.0 | 64.9 | 39.3 | 50.4 | 61.9 | 39.9 |
| 7500 | 26.5 | 43.9 | 29.5 | 37.2 | 27.2 | 29.0 | 64.8 | 38.7 | 50.8 | 68.2 | 41.6 |
| 10000 | 25.8 | 43.5 | 29.1 | 38.8 | 27.4 | 29.8 | 66.9 | 39.0 | 51.2 | 66.2 | 41.8 |
| 12500 | 26.1 | 44.3 | 29.7 | 39.1 | 27.3 | 29.2 | 66.9 | 39.0 | 52.0 | 67.4 | 42.1 |
| Gopher |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.3 | 39.4 | 26.6 | 31.3 | 25.6 | 27.0 | 61.1 | 38.9 | 51.3 | 58.6 | 38.2 |
| 5000 | 25.1 | 41.4 | 29.8 | 34.3 | 26.4 | 27.2 | 64.5 | 39.6 | 52.1 | 62.9 | 40.3 |
| 7500 | 26.5 | 43.0 | 30.5 | 38.5 | 27.2 | 28.8 | 65.7 | 38.2 | 53.7 | 66.4 | 41.8 |
| 10000 | 26.2 | 44.2 | 31.8 | 39.2 | 27.5 | 29.4 | 66.6 | 38.9 | 51.3 | 68.2 | 42.3 |
| 12500 | 25.7 | 44.0 | 31.3 | 40.2 | 27.3 | 29.0 | 66.3 | 39.0 | 51.2 | 68.9 | 42.3 |
| C4 |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.6 | 40.6 | 28.8 | 31.3 | 26.2 | 27.4 | 61.7 | 39.3 | 51.2 | 57.1 | 38.6 |
| 5000 | 22.9 | 41.6 | 29.3 | 36.0 | 26.8 | 27.6 | 64.7 | 40.2 | 50.9 | 63.6 | 40.4 |
| 7500 | 24.2 | 44.2 | 29.5 | 39.2 | 27.2 | 28.4 | 66.2 | 40.9 | 51.6 | 63.8 | 41.5 |
| 10000 | 24.6 | 44.8 | 30.4 | 39.5 | 27.0 | 29.4 | 68.7 | 40.9 | 51.7 | 63.9 | 42.1 |
| 12500 | 25.0 | 46.0 | 31.0 | 40.5 | 27.1 | 29.2 | 68.5 | 40.5 | 51.7 | 66.6 | 42.6 |
| FineWeb |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 23.2 | 39.4 | 27.2 | 31.8 | 25.6 | 26.2 | 62.6 | 39.0 | 51.4 | 57.1 | 38.3 |
| 5000 | 24.2 | 42.3 | 29.8 | 36.2 | 27.0 | 28.4 | 64.3 | 38.9 | 51.4 | 61.4 | 40.4 |
| 7500 | 24.4 | 44.1 | 30.4 | 37.8 | 27.2 | 28.2 | 66.1 | 39.5 | 50.8 | 66.2 | 41.5 |
| 10000 | 23.6 | 46.6 | 32.0 | 39.6 | 27.0 | 27.8 | 66.3 | 39.2 | 53.1 | 70.5 | 42.6 |
| 12500 | 25.2 | 46.8 | 32.6 | 39.6 | 27.2 | 29.0 | 66.5 | 39.4 | 52.4 | 69.2 | 42.8 |
| Gopher + C4 + FineWeb |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 23.6 | 39.3 | 27.6 | 32.1 | 25.8 | 26.0 | 61.7 | 39.8 | 50.9 | 55.4 | 38.2 |
| 5000 | 23.9 | 40.9 | 29.0 | 36.2 | 26.9 | 26.8 | 65.3 | 39.3 | 52.7 | 62.4 | 40.3 |
| 7500 | 25.6 | 42.2 | 30.7 | 39.7 | 27.0 | 28.4 | 66.0 | 40.2 | 51.8 | 60.9 | 41.2 |
| 10000 | 25.8 | 43.3 | 30.8 | 41.4 | 27.5 | 29.8 | 66.9 | 39.5 | 51.8 | 63.1 | 42.0 |
| 12500 | 25.0 | 43.9 | 30.0 | 41.9 | 27.5 | 31.0 | 67.0 | 39.9 | 51.9 | 65.3 | 42.3 |
| ProX-D |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 25.6 | 43.2 | 27.7 | 32.9 | 27.2 | 27.0 | 61.3 | 39.4 | 50.6 | 63.0 | 39.8 |
| 5000 | 25.4 | 46.2 | 28.4 | 35.7 | 28.1 | 28.8 | 64.7 | 39.3 | 53.3 | 64.2 | 41.4 |
| 7500 | 26.9 | 49.2 | 29.1 | 39.2 | 28.6 | 30.8 | 65.4 | 38.8 | 51.2 | 71.7 | 43.1 |
| 10000 | 26.7 | 48.2 | 30.5 | 39.9 | 28.6 | 28.6 | 66.2 | 39.7 | 51.9 | 71.2 | 43.2 |
| 12500 | 26.6 | 49.7 | 30.1 | 40.5 | 29.4 | 30.4 | 66.3 | 39.0 | 51.2 | 71.6 | 43.5 |
| ProX-D+C |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.9 | 43.4 | 27.3 | 32.1 | 26.9 | 28.2 | 60.9 | 38.8 | 51.2 | 60.8 | 39.5 |
| 5000 | 24.9 | 49.6 | 28.8 | 36.8 | 27.9 | 30.6 | 64.7 | 38.8 | 51.1 | 66.9 | 42.0 |
| 7500 | 25.5 | 51.2 | 30.8 | 38.8 | 28.4 | 31.2 | 67.3 | 40.2 | 50.3 | 71.7 | 43.5 |
| 10000 | 26.2 | 51.7 | 30.8 | 39.9 | 29.0 | 32.6 | 68.6 | 39.7 | 51.7 | 73.7 | 44.4 |
| 12500 | 26.4 | 51.9 | 30.9 | 42.4 | 29.4 | 31.6 | 67.9 | 40.0 | 52.2 | 73.5 | 44.6 |

#### D.2 Detailed Performance on 8 Benchmarks Used in Data Selection Experiments

The full benchmark performance used in data-selection method comparison experiments is presented in Table 11.

**Table 11: Detailed evaluation results for different data selection methods.**
| Method | ARC-C | ARC-E | HellaSwag | LogiQA | OBQA | PIQA | WinoGrande | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pythia-410M 0 0-shot |  |  |  |  |  |  |  |  |  |
| Random | 25.6 | 40.2 | 39.7 | 24.7 | 29.4 | 67.1 | 50.6 | 64.1 | 42.7 |
| DSIR | 23.8 | 39.9 | 39.6 | 27.0 | 28.4 | 66.8 | 51.5 | 63.1 | 42.5 |
| DsDm | 24.7 | 41.7 | 40.3 | 27.5 | 29 | 68.1 | 50.1 | 65.4 | 43.4 |
| QuRating | 25.4 | 42.0 | 40.7 | 25.3 | 30.2 | 67.5 | 52.1 | 64.8 | 43.5 |
| MATES | 25.0 | 41.8 | 41.0 | 25.7 | 30.8 | 68.7 | 52.7 | 66.0 | 44.0 |
| ProX | 27.2 | 48.9 | 43.1 | 26.9 | 31.8 | 68.4 | 54.1 | 69.5 | 46.2 |
| Pythia-410M $2$-shot |  |  |  |  |  |  |  |  |  |
| Random | 25.3 | 42.6 | 39.9 | 24.1 | 28.6 | 66.9 | 52.2 | 70.6 | 43.8 |
| DSIR | 23.6 | 42.0 | 39.8 | 26.1 | 28.6 | 66.1 | 51.6 | 71.4 | 43.7 |
| DsDm | 23.6 | 44.2 | 40.1 | 23.5 | 29.2 | 66.5 | 51.5 | 74 | 44.1 |
| QuRating | 23.6 | 43.9 | 40.4 | 26.1 | 30.2 | 67.4 | 51.4 | 74.1 | 44.6 |
| MATES | 25.3 | 43.8 | 40.6 | 24.9 | 30.6 | 67.1 | 53.4 | 74.1 | 45.0 |
| ProX | 27.0 | 52.7 | 42.6 | 23.7 | 32.8 | 68.2 | 53.9 | 78.9 | 47.5 |
| Pythia-1B 0 0-shot |  |  |  |  |  |  |  |  |  |
| Random | 25.6 | 43.7 | 43.8 | 27.5 | 31.8 | 68.9 | 50.7 | 65.8 | 44.7 |
| MATES | 25.9 | 44.9 | 45.3 | 28.7 | 32.2 | 69.5 | 52.4 | 67.3 | 45.8 |
| ProX | 26.2 | 49.1 | 46.6 | 24.8 | 32.2 | 70.3 | 54.2 | 70.9 | 46.8 |
| Pythia-1B $2$-shot |  |  |  |  |  |  |  |  |  |
| Random | 25.5 | 45.1 | 42.9 | 24.6 | 30.0 | 68.3 | 52.1 | 74.6 | 45.4 |
| MATES | 26.8 | 46.1 | 44.8 | 25.2 | 30.6 | 68.7 | 51.6 | 75.7 | 46.2 |
| ProX | 27.3 | 54.5 | 46.2 | 26.6 | 32.2 | 69.0 | 53.9 | 77.4 | 48.4 |

Figure: Figure 13: Visualization of dynamic performance on ten benchmarks.
Refer to caption: x15.png

#### D.3 Detailed Performance in Sec 3.3

In § 3.3, we test ProX’s effectiveness using different sizes of refining models, and also train a series of models by using these curated data.
We report these detailed results in Table 12, Table 13 and Table 14.

**Table 12: Full evaluation results of TLM-xs trained on different ProX model curated data.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TLM-xs trained on Raw data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.5 | 38.5 | 27.0 | 29.1 | 25.8 | 25.0 | 60.2 | 38.8 | 50.4 | 58.6 | 37.6 |
| 5000 | 23.6 | 39.2 | 28.7 | 33.1 | 26.1 | 26.6 | 62.2 | 39.5 | 49.9 | 66.2 | 39.5 |
| 7500 | 23.8 | 42.7 | 28.0 | 33.4 | 26.0 | 26.2 | 64.0 | 39.3 | 51.5 | 67.0 | 40.2 |
| 10000 | 23.8 | 41.2 | 27.8 | 35.0 | 26.6 | 28.0 | 65.3 | 40.9 | 50.1 | 65.9 | 40.5 |
| 12500 | 22.6 | 41.9 | 29.7 | 32.8 | 26.2 | 26.4 | 62.2 | 39.3 | 51.3 | 63.3 | 39.6 |
| TLM-xs trained on ProX-xs data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.8 | 43.5 | 26.5 | 30.3 | 26.8 | 26.6 | 59.3 | 38.6 | 50.8 | 60.7 | 38.8 |
| 5000 | 23.7 | 44.3 | 28.1 | 33.8 | 27.3 | 28.8 | 61.3 | 38.9 | 50.9 | 70.2 | 40.7 |
| 7500 | 24.1 | 46.0 | 29.2 | 35.0 | 27.7 | 30.6 | 63.4 | 38.7 | 52.0 | 70.4 | 41.7 |
| 10000 | 25.3 | 46.1 | 28.3 | 35.7 | 28.1 | 29.2 | 64.4 | 38.5 | 51.2 | 70.6 | 41.7 |
| 12500 | 25.9 | 47.5 | 29.2 | 36.7 | 28.1 | 30.2 | 64.6 | 38.0 | 51.7 | 71.4 | 42.3 |
| TLM-xs trained on ProX-s data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 23.5 | 41.9 | 24.9 | 30.4 | 26.6 | 27.6 | 62.0 | 37.8 | 49.3 | 61.4 | 38.5 |
| 5000 | 24.7 | 44.5 | 27.0 | 33.8 | 27.5 | 28.0 | 62.4 | 38.0 | 50.6 | 67.0 | 40.3 |
| 7500 | 25.3 | 45.3 | 27.3 | 34.0 | 27.9 | 29.2 | 63.4 | 37.7 | 52.9 | 68.7 | 41.2 |
| 10000 | 25.6 | 45.7 | 27.6 | 35.6 | 28.6 | 30.2 | 63.6 | 37.4 | 52.0 | 71.1 | 41.7 |
| 12500 | 26.4 | 46.7 | 27.5 | 37.2 | 28.1 | 29.8 | 62.8 | 37.8 | 52.2 | 70.1 | 41.9 |
| TLM-xs trained on ProX-m curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.9 | 41.3 | 26.5 | 31.1 | 26.9 | 27.0 | 62.2 | 37.6 | 50.6 | 62.4 | 38.9 |
| 5000 | 25.8 | 44.0 | 27.3 | 34.0 | 27.1 | 29.6 | 63.1 | 38.5 | 51.8 | 64.9 | 40.6 |
| 7500 | 26.0 | 45.3 | 28.5 | 36.6 | 27.7 | 29.8 | 63.6 | 39.4 | 51.3 | 68.5 | 41.7 |
| 10000 | 26.0 | 46.6 | 28.8 | 37.3 | 27.6 | 30.6 | 63.3 | 38.7 | 51.6 | 70.3 | 42.1 |
| 12500 | 26.5 | 46.4 | 29.1 | 37.6 | 28.1 | 29.4 | 64.1 | 38.7 | 51.5 | 68.0 | 41.9 |

**Table 13: Full evaluation results of TLM-s trained on different ProX model curated data.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TLM-s trained on Raw data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.1 | 39.0 | 27.6 | 31.6 | 25.9 | 26.6 | 61.2 | 37.3 | 48.9 | 59.1 | 37.9 |
| 5000 | 24.4 | 41.2 | 28.8 | 34.8 | 26.7 | 27.0 | 64.9 | 39.3 | 50.4 | 61.9 | 39.9 |
| 7500 | 26.5 | 43.9 | 29.5 | 37.2 | 27.2 | 29.0 | 64.8 | 38.7 | 50.8 | 68.2 | 41.6 |
| 10000 | 25.8 | 43.5 | 29.1 | 38.8 | 27.4 | 29.8 | 66.9 | 39.0 | 51.2 | 66.2 | 41.8 |
| 12500 | 26.1 | 44.3 | 29.7 | 39.1 | 27.3 | 29.2 | 66.9 | 39.0 | 52.0 | 67.4 | 42.1 |
| TLM-s trained on ProX-xs curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 23.8 | 44.1 | 26.5 | 33.5 | 26.9 | 29.4 | 60.7 | 38.9 | 50.6 | 62.1 | 39.6 |
| 5000 | 26.8 | 48.1 | 28.4 | 36.7 | 28.0 | 30.6 | 64.0 | 38.6 | 50.3 | 65.6 | 41.7 |
| 7500 | 26.9 | 49.0 | 30.6 | 39.5 | 28.2 | 29.6 | 65.3 | 39.6 | 52.2 | 69.6 | 43.0 |
| 10000 | 26.7 | 51.3 | 29.4 | 40.1 | 28.3 | 31.8 | 64.1 | 39.3 | 51.4 | 69.9 | 43.2 |
| 12500 | 26.8 | 52.1 | 30.2 | 41.8 | 28.5 | 31.6 | 65.5 | 39.5 | 51.9 | 70.8 | 43.9 |
| TLM-s trained on ProX-s curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.9 | 43.4 | 27.3 | 32.1 | 26.9 | 28.2 | 60.9 | 38.8 | 51.2 | 60.8 | 39.5 |
| 5000 | 24.9 | 49.6 | 28.8 | 36.8 | 27.9 | 30.6 | 64.7 | 38.8 | 51.1 | 66.9 | 42.0 |
| 7500 | 25.5 | 51.2 | 30.8 | 38.8 | 28.4 | 31.2 | 67.3 | 40.2 | 50.3 | 71.7 | 43.5 |
| 10000 | 26.2 | 51.7 | 30.8 | 39.9 | 29.0 | 32.6 | 68.6 | 39.7 | 51.7 | 73.7 | 44.4 |
| 12500 | 26.4 | 51.9 | 30.9 | 42.4 | 29.4 | 31.6 | 67.9 | 40.0 | 52.2 | 73.5 | 44.6 |
| TLM-s trained on ProX-m curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 25.3 | 45.3 | 27.5 | 32.2 | 26.7 | 27.0 | 62.4 | 38.7 | 50.6 | 60.8 | 39.6 |
| 5000 | 26.1 | 45.4 | 28.6 | 37.2 | 27.4 | 27.8 | 65.7 | 38.9 | 50.9 | 65.6 | 41.4 |
| 7500 | 27.1 | 47.5 | 30.6 | 41.0 | 28.6 | 29.2 | 66.8 | 39.3 | 51.1 | 69.9 | 43.1 |
| 10000 | 26.7 | 50.5 | 30.7 | 41.5 | 28.4 | 30.2 | 67.0 | 40.1 | 49.9 | 70.9 | 43.6 |
| 12500 | 27.4 | 50.7 | 30.6 | 42.0 | 28.8 | 30.2 | 67.4 | 39.4 | 48.8 | 70.1 | 43.5 |

**Table 14: Full evaluation results of TLM-m trained on different ProX model curated data.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TLM-s trained on Raw data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 23.5 | 41.5 | 27.5 | 32.9 | 26.4 | 25.2 | 62.1 | 39.4 | 51.5 | 65.1 | 39.5 |
| 5000 | 24.0 | 42.1 | 29.6 | 37.6 | 27.6 | 27.2 | 65.0 | 39.7 | 53.2 | 68.5 | 41.4 |
| 7500 | 24.3 | 44.9 | 28.9 | 39.3 | 27.8 | 27.6 | 66.4 | 40.4 | 51.3 | 69.2 | 42.0 |
| 10000 | 24.8 | 46.1 | 29.6 | 41.4 | 27.9 | 28.4 | 67.5 | 39.8 | 51.9 | 70.9 | 42.8 |
| 12500 | 26.3 | 46.8 | 29.0 | 43.2 | 28.3 | 27.8 | 68.2 | 40.5 | 50.7 | 72.5 | 43.3 |
| TLM-m trained on ProX-xs curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.9 | 49.6 | 26.5 | 34.0 | 27.3 | 30.4 | 61.8 | 37.9 | 51.3 | 65.1 | 40.9 |
| 5000 | 26.7 | 47.6 | 28.6 | 39.7 | 28.5 | 31.8 | 65.4 | 39.5 | 50.2 | 70.7 | 42.9 |
| 7500 | 27.5 | 52.1 | 30.4 | 41.8 | 29.6 | 31.8 | 67.6 | 39.6 | 51.7 | 75.2 | 44.7 |
| 10000 | 28.4 | 54.7 | 29.8 | 45.2 | 30.8 | 31.8 | 67.9 | 39.7 | 52.0 | 77.7 | 45.8 |
| 12500 | 28.8 | 54.2 | 29.7 | 46.5 | 30.9 | 31.8 | 68.2 | 39.9 | 51.3 | 78.3 | 46.0 |
| TLM-m trained on ProX-s curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 25.3 | 45.7 | 27.8 | 34.2 | 27.8 | 29.0 | 64.4 | 37.5 | 49.3 | 66.3 | 40.7 |
| 5000 | 26.1 | 49.0 | 28.8 | 40.2 | 29.2 | 30.8 | 65.6 | 39.0 | 50.5 | 71.2 | 43.0 |
| 7500 | 27.7 | 53.6 | 31.1 | 44.1 | 29.6 | 34.8 | 67.6 | 39.4 | 52.5 | 72.2 | 45.3 |
| 10000 | 27.2 | 54.0 | 31.5 | 45.1 | 30.3 | 33.8 | 67.7 | 39.7 | 52.9 | 74.2 | 45.6 |
| 12500 | 28.6 | 56.1 | 31.8 | 45.5 | 30.5 | 34.4 | 68.5 | 39.4 | 51.3 | 76.1 | 46.2 |
| TLM-m trained on ProX-m curated data |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.7 | 44.1 | 25.9 | 34.8 | 27.4 | 27.8 | 62.9 | 38.9 | 49.2 | 67.0 | 40.3 |
| 5000 | 27.7 | 48.0 | 26.8 | 40.5 | 28.5 | 30.6 | 67.4 | 39.4 | 50.3 | 69.1 | 42.8 |
| 7500 | 26.7 | 51.9 | 26.7 | 42.9 | 29.3 | 31.4 | 69.1 | 40.3 | 50.4 | 73.3 | 44.2 |
| 10000 | 28.4 | 52.4 | 27.9 | 45.0 | 29.7 | 32.0 | 70.2 | 40.0 | 51.9 | 75.4 | 45.3 |
| 12500 | 28.3 | 53.7 | 28.4 | 45.9 | 30.1 | 33.8 | 70.6 | 41.1 | 52.3 | 72.5 | 45.7 |

We also further scale ProX to other two pre-training corpora, C4 and FineWeb. We also scale our training to about $50$B tokens, and directly compare with existing well-trained models developed by different research groups.
We report our detailed results in Table 15, Table 16 and Table 17. We also present other models’ results in Table 18.

**Table 15: Full evaluation results on scaling pre-training to about $50$B tokens on RedPajama-V2.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TLM-m trained on RedPajama-V2 raw data. |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.0 | 42.9 | 26.6 | 33.7 | 25.9 | 26.0 | 62.4 | 39.4 | 52.3 | 64.0 | 39.7 |
| 5000 | 24.3 | 45.9 | 26.4 | 37.4 | 27.0 | 27.6 | 64.1 | 39.7 | 49.5 | 66.2 | 40.8 |
| 7500 | 25.1 | 45.3 | 28.8 | 40.3 | 27.1 | 29.2 | 66.3 | 39.1 | 51.7 | 66.9 | 42.0 |
| 10000 | 25.8 | 49.3 | 31.5 | 42.5 | 28.0 | 28.8 | 66.7 | 39.6 | 51.5 | 74.0 | 43.8 |
| 12500 | 25.3 | 50.1 | 30.2 | 43.0 | 28.2 | 30.0 | 66.6 | 39.2 | 51.1 | 74.2 | 43.8 |
| 15000 | 26.2 | 50.3 | 31.2 | 44.3 | 28.8 | 28.4 | 68.2 | 39.8 | 51.7 | 76.2 | 44.5 |
| 17500 | 25.8 | 51.1 | 30.8 | 44.7 | 29.0 | 29.6 | 67.7 | 39.2 | 52.6 | 75.2 | 44.6 |
| 20000 | 26.7 | 52.5 | 31.7 | 47.2 | 28.6 | 30.4 | 69.0 | 39.6 | 53.0 | 78.2 | 45.7 |
| 22500 | 27.4 | 51.7 | 32.1 | 47.2 | 29.3 | 30.4 | 69.5 | 39.5 | 51.9 | 78.5 | 45.7 |
| 25000 | 26.9 | 51.4 | 32.4 | 47.3 | 29.3 | 32.2 | 69.7 | 39.6 | 52.1 | 79.1 | 46.0 |
| TLM-m trained on ProX refined RedPajama-V2 data. |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.8 | 46.8 | 27.2 | 33.8 | 27.3 | 28.2 | 61.3 | 38.6 | 50.3 | 65.1 | 40.3 |
| 5000 | 26.9 | 49.3 | 28.5 | 40.1 | 28.0 | 30.6 | 66.2 | 39.7 | 50.2 | 70.1 | 43.0 |
| 7500 | 28.5 | 53.1 | 29.2 | 41.7 | 29.4 | 33.2 | 66.9 | 39.3 | 53.0 | 73.0 | 44.7 |
| 10000 | 28.2 | 53.5 | 30.1 | 43.6 | 29.8 | 31.6 | 68.4 | 39.6 | 52.0 | 75.3 | 45.2 |
| 12500 | 29.5 | 55.3 | 30.2 | 46.4 | 30.5 | 32.2 | 68.6 | 40.2 | 52.6 | 76.9 | 46.2 |
| 15000 | 30.0 | 57.1 | 30.2 | 47.6 | 30.9 | 33.0 | 69.5 | 39.8 | 52.2 | 77.8 | 46.8 |
| 17500 | 31.5 | 59.6 | 29.4 | 49.5 | 31.6 | 33.6 | 69.4 | 39.8 | 53.0 | 78.9 | 47.6 |
| 20000 | 31.2 | 61.2 | 29.4 | 50.4 | 31.4 | 35.2 | 70.6 | 40.1 | 53.7 | 79.6 | 48.3 |
| 22500 | 32.0 | 61.7 | 30.2 | 51.4 | 31.4 | 34.0 | 70.0 | 39.9 | 53.2 | 79.5 | 48.3 |
| 25000 | 31.1 | 60.7 | 29.8 | 51.0 | 31.7 | 33.2 | 70.9 | 39.2 | 53.3 | 79.1 | 48.0 |

**Table 16: Full evaluation results on scaling pre-training to about $50$B tokens on C4.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TLM-m trained on C4 raw data. |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.4 | 39.7 | 26.8 | 36.5 | 26.5 | 27.6 | 64.8 | 40.2 | 50.1 | 60.0 | 39.5 |
| 5000 | 23.9 | 42.9 | 27.5 | 42.3 | 27.1 | 29.6 | 68.2 | 39.6 | 50.3 | 66.6 | 41.8 |
| 7500 | 25.1 | 44.8 | 28.2 | 45.4 | 27.1 | 29.2 | 70.7 | 40.7 | 51.6 | 66.3 | 42.9 |
| 10000 | 25.5 | 46.0 | 32.3 | 48.2 | 27.9 | 31.6 | 71.1 | 39.7 | 52.3 | 67.6 | 44.2 |
| 12500 | 25.8 | 48.8 | 30.3 | 49.7 | 27.9 | 31.6 | 71.2 | 40.9 | 52.0 | 69.4 | 44.8 |
| 15000 | 26.9 | 48.0 | 28.2 | 50.5 | 28.5 | 31.4 | 71.9 | 41.1 | 51.4 | 69.7 | 44.8 |
| 17500 | 26.6 | 48.8 | 30.3 | 52.1 | 28.6 | 31.2 | 73.2 | 41.6 | 52.0 | 70.0 | 45.4 |
| 20000 | 26.3 | 50.1 | 29.7 | 52.5 | 28.5 | 32.6 | 72.3 | 41.7 | 52.3 | 71.0 | 45.7 |
| 22500 | 25.8 | 50.7 | 31.0 | 52.9 | 28.8 | 33.8 | 73.0 | 41.6 | 53.0 | 71.5 | 46.2 |
| 25000 | 25.3 | 48.8 | 30.1 | 52.4 | 28.8 | 32.2 | 72.0 | 40.6 | 53.6 | 71.7 | 45.5 |
| TLM-m trained on ProX refined C4 data. |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 24.1 | 45.9 | 26.0 | 37.3 | 27.2 | 29.0 | 66.3 | 39.8 | 50.8 | 65.9 | 41.2 |
| 5000 | 27.3 | 50.0 | 26.6 | 42.4 | 28.6 | 33.8 | 68.1 | 40.5 | 53.0 | 71.9 | 44.2 |
| 7500 | 28.3 | 53.7 | 27.7 | 47.7 | 29.3 | 35.4 | 71.1 | 39.3 | 54.0 | 73.1 | 46.0 |
| 10000 | 30.0 | 54.3 | 28.1 | 50.9 | 30.0 | 33.6 | 71.2 | 40.6 | 52.0 | 74.2 | 46.5 |
| 12500 | 29.3 | 56.7 | 27.5 | 52.3 | 30.9 | 33.8 | 72.8 | 39.9 | 52.5 | 77.5 | 47.3 |
| 15000 | 29.6 | 55.9 | 28.3 | 53.9 | 30.6 | 35.0 | 72.9 | 41.0 | 53.8 | 75.8 | 47.7 |
| 17500 | 30.6 | 55.5 | 28.7 | 53.3 | 31.2 | 34.2 | 73.6 | 40.4 | 53.4 | 76.7 | 47.8 |
| 20000 | 30.0 | 57.6 | 28.3 | 54.9 | 31.1 | 37.2 | 74.6 | 40.7 | 53.6 | 79.4 | 48.7 |
| 22500 | 30.1 | 56.7 | 28.6 | 55.2 | 31.4 | 37.2 | 73.8 | 41.6 | 53.3 | 77.7 | 48.6 |
| 25000 | 31.1 | 56.0 | 28.4 | 55.2 | 31.1 | 36.2 | 74.0 | 41.0 | 54.1 | 76.8 | 48.4 |

**Table 17: Full evaluation results on scaling pre-training to about $50$B tokens on FineWeb.**
| Train<br>Steps | ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TLM-m trained on FineWeb raw data. |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 22.9 | 41.2 | 28.9 | 34.3 | 26.1 | 27.6 | 64.8 | 39.3 | 52.1 | 62.8 | 40.0 |
| 5000 | 25.5 | 44.5 | 30.4 | 39.8 | 26.9 | 32.0 | 68.4 | 39.2 | 52.1 | 67.2 | 42.6 |
| 7500 | 26.8 | 45.6 | 31.4 | 44.1 | 27.6 | 30.2 | 70.9 | 38.8 | 52.2 | 70.3 | 43.8 |
| 10000 | 27.2 | 46.2 | 31.3 | 47.2 | 28.3 | 31.6 | 72.1 | 38.8 | 53.4 | 69.0 | 44.5 |
| 12500 | 26.4 | 49.2 | 32.1 | 48.7 | 28.7 | 31.6 | 71.5 | 40.1 | 52.6 | 74.7 | 45.6 |
| 15000 | 27.1 | 49.6 | 32.8 | 49.5 | 28.9 | 31.0 | 72.7 | 39.0 | 52.3 | 77.1 | 46.0 |
| 17500 | 26.4 | 50.9 | 33.8 | 51.3 | 29.3 | 31.0 | 71.9 | 39.3 | 53.0 | 78.0 | 46.5 |
| 20000 | 27.1 | 53.1 | 33.2 | 51.2 | 29.6 | 32.2 | 73.4 | 39.7 | 52.3 | 76.3 | 46.8 |
| 22500 | 27.1 | 51.2 | 34.9 | 51.7 | 29.5 | 33.4 | 73.7 | 40.1 | 52.4 | 78.0 | 47.2 |
| 25000 | 28.5 | 52.6 | 33.9 | 53.2 | 29.8 | 32.6 | 72.9 | 40.2 | 53.0 | 77.1 | 47.4 |
| TLM-m trained on ProX refined FineWeb data. |  |  |  |  |  |  |  |  |  |  |  |
| 2500 | 25.8 | 46.8 | 27.4 | 36.1 | 27.7 | 28.8 | 63.9 | 39.3 | 51.9 | 69.1 | 41.7 |
| 5000 | 28.5 | 52.1 | 28.8 | 43.5 | 29.3 | 32.6 | 66.4 | 38.7 | 51.2 | 71.3 | 44.2 |
| 7500 | 28.2 | 52.0 | 30.6 | 45.9 | 29.9 | 33.0 | 69.3 | 39.5 | 51.7 | 71.8 | 45.2 |
| 10000 | 29.3 | 54.3 | 30.6 | 48.5 | 30.8 | 33.2 | 69.7 | 40.7 | 50.6 | 74.4 | 46.2 |
| 12500 | 28.7 | 57.8 | 30.7 | 48.1 | 31.1 | 32.6 | 72.0 | 40.4 | 52.7 | 77.4 | 47.2 |
| 15000 | 31.1 | 59.6 | 31.9 | 50.4 | 31.8 | 34.4 | 71.9 | 40.5 | 50.8 | 78.0 | 48.0 |
| 17500 | 32.6 | 60.9 | 31.9 | 51.5 | 32.2 | 33.8 | 72.3 | 39.7 | 52.5 | 78.9 | 48.6 |
| 20000 | 33.2 | 62.5 | 32.5 | 51.6 | 32.4 | 34.6 | 72.4 | 39.7 | 51.7 | 80.7 | 49.1 |
| 22500 | 34.7 | 63.6 | 32.9 | 53.3 | 32.9 | 34.8 | 73.1 | 40.3 | 54.2 | 80.5 | 50.0 |
| 25000 | 34.4 | 63.9 | 32.6 | 53.0 | 33.1 | 34.4 | 73.1 | 39.3 | 52.7 | 81.5 | 49.8 |

**Table 18: Detailed evaluation results of existing base models trained on different corpora and trained using different techniques.**
| ARC-C | ARC-E | CSQA | HellaSwag | MMLU | OBQA | PiQA | SIQA | WinoG | SciQ | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TinyLlama-1.1B (trained on 3T tokens) |  |  |  |  |  |  |  |  |  |  |
| 31.5 | 59.0 | 35.5 | 57.8 | 32.8 | 33.4 | 72.8 | 40.0 | 56.0 | 82.4 | 50.1 |
| OLMo-1B (trained on 2T tokens) |  |  |  |  |  |  |  |  |  |  |
| 31.4 | 59.7 | 38.9 | 61.9 | 32.2 | 38.4 | 76.1 | 41.5 | 53.9 | 78.8 | 51.3 |
| Pythia-1.4B |  |  |  |  |  |  |  |  |  |  |
| 28.7 | 56.9 | 34.7 | 51.7 | 31.5 | 36.0 | 71.8 | 40.8 | 55.1 | 79.3 | 48.7 |
| Pythia-2.8B |  |  |  |  |  |  |  |  |  |  |
| 32.9 | 61.0 | 36.5 | 60.4 | 33.3 | 35.0 | 73.5 | 41.1 | 57.0 | 83.1 | 51.4 |
| ShearedLlama-1.3B (pruned from Llama-2-7B) |  |  |  |  |  |  |  |  |  |  |
| 22.4 | 39.7 | 29.3 | 36.0 | 26.4 | 28.4 | 62.6 | 39.9 | 52.0 | 71.4 | 40.8 |
| ShearedLLama-1.3B (pruned from Llama-2-7B, and further trained on $50$B tokens) |  |  |  |  |  |  |  |  |  |  |
| 29.0 | 58.3 | 34.8 | 59.6 | 32.0 | 35.0 | 74.6 | 41.0 | 56.3 | 82.3 | 50.3 |
| InstructLM-1.3B (LLM data synthesis) |  |  |  |  |  |  |  |  |  |  |
| 28.1 | 57.9 | 32.5 | 52.3 | 30.0 | 34.0 | 74.5 | 39.9 | 56.1 | 86.9 | 49.2 |
| Cosmo-1.8B (LLM data synthesis) |  |  |  |  |  |  |  |  |  |  |
| 33.4 | 57.0 | 31.2 | 55.1 | 32.4 | 35.2 | 71.4 | 42.0 | 54.7 | 84.4 | 49.7 |

#### D.4 Evaluation Results of Continual Pre-training in Sec 3.4

We provide full ablation results for each base model, as shown in Table 19. We can observe that ProX-D+C consistently improves average performance over ProX-D across various base models. Although the performance gain from ProX-D+C compared to ProX-D is less pronounced than the improvement of ProX-D over continual pre-training on raw OpenWebMath, this is both understandable and expected. ProX-D+C does not significantly reduce the token count beyond the reductions achieved by ProX-D alone. Given the scale of the OpenWebMath corpus, a more aggressive token removal strategy could potentially diminish the diversity of unique tokens below the threshold necessary for robust pre-training. This observation underscores the delicate balance between data refinement and maintaining sufficient linguistic variety for effective language model training, particularly when working with limited-scale corpora.

**Table 19: Full ablation results on OpenWebMath Continual Pre-training (CPT). All models are tested using few-shot CoT prompts. Llemma and InternLM2-Math are continual pre-trained models from CodeLlama [Rozière et al., ] and InternLM2 [Team, ] with public available data, respectively. DeepSeek-LLM denotes an internal DeepSeek model, and the model trained on OpenWebMath introduced by Shao et al. []. Note that the unique tokens and training tokens in the column refer exclusively to the token numbers from math-specific corpora (calculated by corresponding tokenizers). ^†: MQA evaluation of InternLM2-Base is based on an alternative prompt due to non-prediction issues with the original prompt. The bolded entries represent the best results within the same base model and CPT experiments.**
| Model | Size | Method | Uniq<br>Toks | Train<br>Toks | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Existing Continual Pre-training for Reference |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| DeepSeek-LLM | 1.3B | - | - | - | 2.9 | 3.0 | - | - | - | - | - | 19.5 | 15.6 | - |
| 1.3B | - | 14B | 150B | 11.5 | 8.9 | - | - | - | - | - | 29.6 | 31.3 | - |  |
| CodeLlama (Base) | 7B | - | - | - | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| 34B | - | - | - | 31.8 | 10.8 | 61.9 | 66.0 | 83.4 | 51.6 | 23.7 | 43.0 | 53.1 | 47.3 |  |
| Llemma | 7B | - | 55B | 200B | 38.8 | 17.2 | 56.1 | 69.1 | 82.4 | 48.7 | 41.0 | 45.4 | 59.4 | 50.9 (+21.8) |
| 34B | - | 55B | 50B | 54.2 | 23.0 | 67.9 | 75.7 | 90.1 | 57.9 | 49.8 | 54.7 | 68.8 | 60.1 (+12.8) |  |
| InternLM2-Base | 7B | - | - | - | 27.0 | 6.6 | 49.0 | 59.3 | 74.8 | 40.1 | 20.9^† | 19.0 | 28.1 | 36.1 |
| 20B | - | - | - | 50.6 | 18.8 | 72.5 | 75.9 | 93.9 | 45.4 | 33.1 | 53.7 | 59.4 | 55.9 |  |
| InternLM2-Math | 7B | - | 31B | 125B | 41.8 | 14.4 | 61.6 | 66.8 | 83.7 | 50.0 | 57.3 | 24.8 | 37.5 | 48.7 (+12.6) |
| 20B | - | 120B | 500B | 65.4 | 30.0 | 75.7 | 79.3 | 94.0 | 50.9 | 38.5 | 53.1 | 71.9 | 62.1 (+6.2) |  |
| Applying Data Refinement Approaches |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| TinyLlama (Base) | 1.1B | - | - | - | 2.8 | 3.2 | 10.9 | 18.0 | 20.2 | 12.5 | 14.6 | 16.4 | 21.9 | 14.7 |
| TinyLlama (CPT) | 1.1B | - | 15B | 15B | 6.2 | 4.8 | 22.3 | 36.2 | 47.6 | 19.3 | 11.6 | 20.7 | 25.0 | 21.5 (+8.1) |
| 1.1B | Rho | 15B | 9B^∗(^6^66Rho-1 only counts the selected tokens that are used for training (loss calculation).) | 7.1 | 5.0 | 23.5 | 41.2 | 53.8 | - | 18.0 | - | - | - |  |
| 1.1B | Rule | 6.5B | 15B | 4.5 | 2.8 | 17.5 | 29.4 | 39.3 | 15.1 | 12.4 | 19.4 | 25.0 | 18.4 (+3.7) |  |
| 1.1B | ProX-D | 5.4B | 15B | 9.3 | 7.4 | 23.4 | 41.9 | 55.6 | 22.1 | 14.6 | 24.1 | 25.0 | 24.8 (+10.1) |  |
|  | 1.1B | ProX-D+C | 5B | 15B | 9.0 | 5.6 | 23.8 | 41.9 | 56.9 | 22.2 | 15.6 | 26.8 | 31.2 | 25.7 (+11.0) |
| Llama-2 (Base) | 7B | - | - | - | 14.1 | 3.8 | 39.5 | 51.6 | 63.6 | 30.9 | 12.5 | 32.9 | 34.4 | 31.5 |
| Llama-2 (CPT) | 7B | - | 15B | 10B | 29.6 | 13.6 | 49.2 | 61.9 | 78.4 | 36.3 | 31.9 | 40.5 | 43.8 | 42.8 (+11.3) |
| 7B | ProX-D | 5.4B | 10B | 30.3 | 16.0 | 54.2 | 63.8 | 79.5 | 37.3 | 37.2 | 44.2 | 46.9 | 45.5 (+14.0) |  |
|  | 7B | ProX-D+C | 5B | 10B | 30.6 | 16.8 | 50.2 | 63.7 | 79.3 | 37.3 | 40.1 | 43.8 | 53.1 | 46.1 (+14.6) |
| CodeLlama (Base) | 7B | - | - | - | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| CodeLlama (CPT) | 7B | - | 15B | 10B | 31.1 | 14.8 | 51.4 | 62.1 | 81.2 | 33.6 | 30.4 | 40.5 | 43.8 | 43.2 (+14.1) |
| 7B | ProX-D | 5.4B | 10B | 38.1 | 17.0 | 54.2 | 67.0 | 83.1 | 40.9 | 39.8 | 43.7 | 50.0 | 48.2 (+19.1) |  |
|  | 7B | ProX-D+C | 5B | 10B | 35.6 | 17.6 | 55.8 | 67.9 | 82.7 | 41.3 | 38.9 | 42.6 | 62.5 | 49.4 (+20.3) |
| Mistral (Base) | 7B | - | - | - | 40.6 | 11.4 | 65.4 | 68.5 | 87.0 | 52.9 | 32.3 | 50.0 | 56.2 | 51.6 |
| Mistral (CPT) | 7B | - | 15B | 10B | 44.4 | 19.2 | 65.2 | 69.6 | 88.4 | 46.6 | 43.1 | 50.8 | 65.6 | 54.8 (+3.2) |
| 7B | ProX-D | 5.5B | 10B | 47.8 | 24.8 | 63.5 | 72.4 | 88.9 | 48.3 | 48.2 | 54.1 | 62.5 | 56.4 (+4.8) |  |
|  | 7B | ProX-D+C | 4.7B | 10B | 51.0 | 22.4 | 64.9 | 72.9 | 89.2 | 49.8 | 53.0 | 54.2 | 75.0 | 59.2 (+7.6) |

Besides, we report the detailed dynamic evaluation results of our continual pre-training experiments on OpenWebMath:

- •
Tables 20, 21, 22, and 23 present the evaluation results for TinyLlama-1.1B.
- •
Tables 24, 25, and 26 present the evaluation results for Llama-2.
- •
Tables 27, 28, 29 present the evaluation results for CodeLlama.
- •
Tables 30, 31, and 32 show the evaluation results for Mistral-7B.

**Table 20: Full evaluation results of TinyLlama-1.1B continual pre-training on OpenWebMath with raw data. Note that about 1B tokens are trained per 500 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 2.8 | 3.2 | 10.9 | 18 | 20.2 | 12.5 | 14.6 | 16.4 | 21.9 | 14.7 |
| 500 | 1.9 | 3.4 | 16.3 | 23.9 | 30.3 | 13.9 | 10.3 | 14.8 | 18.8 | 14.8 |
| 1000 | 3.1 | 2.2 | 16.6 | 25.6 | 32.4 | 12.5 | 12.0 | 16.6 | 25.0 | 16.2 |
| 1500 | 2.7 | 3.0 | 17.6 | 28.5 | 34.5 | 13.9 | 8.7 | 14.1 | 15.6 | 15.4 |
| 2000 | 4.5 | 3.2 | 16.4 | 28.5 | 39.0 | 15.1 | 10.2 | 16.6 | 34.4 | 18.7 |
| 2500 | 4.9 | 3.4 | 19.3 | 31.0 | 39.2 | 16.0 | 12.1 | 18.6 | 9.4 | 17.1 |
| 3000 | 4.1 | 5.2 | 19.1 | 32.0 | 43.0 | 15.3 | 9.6 | 16.1 | 18.8 | 18.1 |
| 3500 | 4.9 | 3.6 | 19.7 | 31.4 | 40.4 | 18.1 | 11.3 | 19.6 | 15.6 | 18.3 |
| 4000 | 4.8 | 4.8 | 19.5 | 33.8 | 44.5 | 16.4 | 10.7 | 19.9 | 12.5 | 18.5 |
| 4500 | 5.4 | 4.8 | 20.2 | 35.0 | 45.2 | 17.9 | 12.7 | 21.0 | 18.8 | 20.1 |
| 5000 | 5.5 | 4.6 | 22.3 | 34.6 | 42.9 | 16.0 | 10.6 | 21.7 | 28.1 | 20.7 |
| 5500 | 4.9 | 5.8 | 23.6 | 35.2 | 44.0 | 20.4 | 11.0 | 21.1 | 21.9 | 20.9 |
| 6000 | 6.1 | 4.4 | 22.8 | 36.2 | 45.4 | 17.8 | 12.7 | 21.4 | 15.6 | 20.3 |
| 6500 | 6.3 | 3.6 | 23.2 | 37.3 | 48.0 | 19.7 | 10.3 | 21.0 | 18.8 | 20.9 |
| 7000 | 6.1 | 4.6 | 22.2 | 36.6 | 46.9 | 19.4 | 12.0 | 21.5 | 21.9 | 21.2 |
| 7500 | 6.2 | 4.8 | 22.3 | 36.2 | 47.6 | 19.3 | 11.6 | 20.7 | 25.0 | 21.5 |

**Table 21: Full evaluation results of TinyLlama-1.1B continual pre-training on OpenWebMath with data after rule-based filtering. Note that about 1B tokens are trained per 500 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 2.8 | 3.2 | 10.9 | 18 | 20.2 | 12.5 | 14.6 | 16.4 | 21.9 | 14.7 |
| 500 | 3.4 | 3.6 | 13.6 | 22.5 | 25.9 | 13.1 | 14.2 | 13.5 | 28.1 | 15.3 |
| 1000 | 3.0 | 2.8 | 14.1 | 22.5 | 27.8 | 11.4 | 11.0 | 16.4 | 12.5 | 13.5 |
| 1500 | 3.6 | 3.2 | 13.6 | 24.0 | 31.2 | 13.9 | 9.2 | 18.0 | 18.8 | 15.1 |
| 2000 | 3.5 | 2.4 | 15.0 | 25.1 | 33.0 | 12.5 | 10.6 | 13.9 | 15.6 | 14.6 |
| 2500 | 3.3 | 1.6 | 15.0 | 25.3 | 33.5 | 13.7 | 11.1 | 18.1 | 25.0 | 16.3 |
| 3000 | 3.5 | 3.0 | 16.4 | 25.5 | 33.4 | 14.1 | 10.2 | 18.4 | 18.8 | 15.9 |
| 3500 | 3.2 | 3.4 | 17.2 | 27.0 | 37.7 | 14.6 | 11.2 | 13.3 | 25.0 | 17.0 |
| 4000 | 3.5 | 3.6 | 15.6 | 26.2 | 36.5 | 13.4 | 12.1 | 15.9 | 18.8 | 16.2 |
| 4500 | 4.1 | 3.8 | 15.6 | 27.9 | 38.2 | 14.9 | 11.6 | 17.1 | 18.8 | 16.9 |
| 5000 | 4.2 | 3.6 | 18.6 | 28.7 | 37.7 | 14.3 | 12.7 | 17.5 | 21.9 | 17.7 |
| 5500 | 4.1 | 3.8 | 16.3 | 29.3 | 38.4 | 14.7 | 10.8 | 17.5 | 18.8 | 17.1 |
| 6000 | 4.3 | 3.6 | 16.0 | 28.7 | 39.1 | 13.5 | 12.8 | 19.5 | 21.9 | 17.7 |
| 6500 | 4.2 | 3.2 | 16.4 | 29.5 | 39.0 | 15.1 | 11.7 | 17.9 | 21.9 | 17.7 |
| 7000 | 4.0 | 4.0 | 16.2 | 29.6 | 37.9 | 16.0 | 13.8 | 17.8 | 21.9 | 17.9 |
| 7500 | 4.5 | 2.8 | 17.5 | 29.4 | 39.3 | 15.1 | 12.4 | 19.4 | 25.0 | 18.4 |

**Table 22: Full evaluation results of TinyLlama-1.1B continual pre-training on OpenWebMath with data after ProX-D. Note that about 1B tokens are trained per 500 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 2.8 | 3.2 | 10.9 | 18 | 20.2 | 12.5 | 14.6 | 16.4 | 21.9 | 14.7 |
| 500 | 3.3 | 2.8 | 17.7 | 29.0 | 38.7 | 12.4 | 9.5 | 15.7 | 15.6 | 16.1 |
| 1000 | 4.6 | 4.0 | 18.1 | 31.6 | 41.9 | 15.9 | 11.9 | 18.2 | 25.0 | 19.0 |
| 1500 | 5.2 | 5.4 | 21.1 | 32.9 | 43.1 | 15.3 | 11.1 | 20.4 | 12.5 | 18.6 |
| 2000 | 6.8 | 5.8 | 20.2 | 33.5 | 46.6 | 18.2 | 10.7 | 20.3 | 12.5 | 19.4 |
| 2500 | 7.1 | 3.8 | 20.7 | 37.0 | 48.6 | 18.3 | 12.0 | 21.4 | 18.8 | 20.9 |
| 3000 | 7.4 | 4.4 | 22.9 | 37.1 | 50.5 | 18.3 | 12.3 | 21.2 | 25.0 | 22.1 |
| 3500 | 8.8 | 4.8 | 22.8 | 39.4 | 53.3 | 19.2 | 12.0 | 22.8 | 34.4 | 24.2 |
| 4000 | 8.6 | 4.6 | 24.0 | 38.7 | 51.4 | 18.8 | 14.8 | 24.4 | 18.8 | 22.7 |
| 4500 | 8.6 | 4.2 | 24.2 | 39.2 | 53.6 | 20.4 | 13.5 | 23.9 | 18.8 | 22.9 |
| 5000 | 8.9 | 5.2 | 24.0 | 40.0 | 52.6 | 20.0 | 13.6 | 23.9 | 18.8 | 23.0 |
| 5500 | 8.0 | 6.2 | 23.2 | 41.4 | 55.0 | 22.3 | 14.3 | 24.9 | 25.0 | 24.5 |
| 6000 | 8.3 | 5.2 | 22.2 | 39.8 | 54.0 | 24.3 | 12.6 | 25.1 | 31.2 | 24.7 |
| 6500 | 9.4 | 5.6 | 24.4 | 40.2 | 54.5 | 20.3 | 13.0 | 24.9 | 31.2 | 24.8 |
| 7000 | 9.2 | 5.8 | 25.8 | 40.6 | 55.3 | 22.5 | 12.5 | 24.5 | 21.9 | 24.2 |
| 7500 | 9.3 | 7.4 | 23.4 | 41.9 | 55.6 | 22.1 | 14.6 | 24.1 | 25.0 | 24.8 |

**Table 23: Full evaluation results of TinyLlama-1.1B continual pre-training on OpenWebMath with data after ProX-D+C. Note that about 1B tokens are trained per 500 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 2.8 | 3.2 | 10.9 | 18 | 20.2 | 12.5 | 14.6 | 16.4 | 21.9 | 14.7 |
| 500 | 4.3 | 5.0 | 16.4 | 28.8 | 36.4 | 15.3 | 11.4 | 18.5 | 15.6 | 16.9 |
| 1000 | 5.5 | 3.8 | 20.5 | 34.6 | 44.6 | 15.3 | 12.1 | 19.6 | 28.1 | 20.5 |
| 1500 | 5.2 | 4.4 | 21.4 | 34.5 | 44.7 | 16.1 | 11.2 | 21.4 | 34.4 | 21.5 |
| 2000 | 6.3 | 5.4 | 20.1 | 33.7 | 46.2 | 19.4 | 10.5 | 21.2 | 12.5 | 19.5 |
| 2500 | 7.8 | 5.4 | 22.1 | 37.0 | 49.5 | 17.9 | 13.3 | 22.9 | 21.9 | 22.0 |
| 3000 | 6.4 | 3.4 | 23.0 | 38.6 | 51.1 | 18.5 | 12.6 | 24.3 | 18.8 | 21.9 |
| 3500 | 8.5 | 4.6 | 24.1 | 40.2 | 53.8 | 22.1 | 12.5 | 23.1 | 25.0 | 23.8 |
| 4000 | 8.2 | 6.0 | 24.1 | 41.0 | 52.4 | 19.8 | 10.2 | 26.1 | 31.2 | 24.3 |
| 4500 | 8.3 | 5.4 | 24.1 | 41.3 | 54.4 | 20.6 | 15.2 | 24.2 | 28.1 | 24.6 |
| 5000 | 8.5 | 7.0 | 26.0 | 40.5 | 54.9 | 21.7 | 13.9 | 25.5 | 34.4 | 25.8 |
| 5500 | 8.7 | 4.0 | 23.2 | 41.1 | 54.8 | 20.5 | 14.4 | 26.5 | 21.9 | 23.9 |
| 6000 | 8.3 | 5.0 | 24.8 | 41.3 | 54.3 | 23.2 | 14.0 | 25.3 | 25.0 | 24.6 |
| 6500 | 8.6 | 6.4 | 24.5 | 41.6 | 55.1 | 22.2 | 14.4 | 26.5 | 25.0 | 24.9 |
| 7000 | 8.9 | 6.0 | 23.4 | 40.5 | 53.4 | 22.0 | 15.8 | 27.3 | 28.1 | 25.0 |
| 7500 | 9.0 | 4.4 | 23.8 | 41.9 | 56.4 | 22.2 | 15.6 | 26.8 | 31.2 | 25.7 |

**Table 24: Full evaluation results of Llama-2 continual pre-training on OpenWebMath with raw data. Note that about 1B tokens are trained per 1000 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 14.1 | 3.8 | 39.5 | 51.6 | 63.6 | 30.9 | 12.5 | 32.9 | 34.4 | 31.5 |
| 1k | 17.2 | 3.6 | 39.1 | 50.4 | 63.0 | 30.2 | 18.9 | 31.8 | 31.2 | 31.7 |
| 2k | 19.7 | 6.0 | 43.9 | 55.5 | 68.3 | 32.9 | 19.0 | 33.0 | 37.5 | 35.1 |
| 3k | 19.6 | 8.6 | 42.9 | 56.3 | 68.4 | 32.2 | 17.4 | 34.6 | 40.6 | 35.6 |
| 4k | 21.8 | 8.8 | 44.6 | 57.3 | 72.0 | 28.9 | 23.6 | 35.8 | 40.6 | 37.0 |
| 5k | 22.6 | 10.4 | 45.9 | 57.0 | 73.5 | 31.5 | 23.9 | 39.0 | 43.8 | 38.6 |
| 6k | 24.5 | 10.0 | 44.9 | 57.6 | 73.7 | 35.5 | 25.8 | 36.1 | 43.8 | 39.1 |
| 7k | 23.3 | 10.4 | 46.5 | 59.0 | 75.3 | 32.9 | 27.7 | 39.0 | 50.0 | 40.5 |
| 8k | 29.0 | 12.4 | 46.4 | 59.7 | 77.0 | 33.1 | 30.2 | 38.8 | 50.0 | 41.8 |
| 9k | 26.1 | 12.8 | 48.8 | 59.9 | 74.3 | 35.0 | 28.3 | 39.2 | 50.0 | 41.6 |
| 10k | 29.6 | 13.6 | 49.2 | 61.9 | 78.4 | 36.3 | 31.9 | 40.5 | 43.8 | 42.8 |

**Table 25: Full evaluation results of Llama-2 continual pre-training on OpenWebMath with ProX-D. Note that about 1B tokens are trained per 1000 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 14.1 | 3.8 | 39.5 | 51.6 | 63.6 | 30.9 | 12.5 | 32.9 | 34.4 | 31.5 |
| 1k | 17.1 | 7.2 | 39.8 | 51.6 | 68.4 | 31.4 | 21.4 | 35.2 | 40.6 | 34.7 |
| 2k | 21.9 | 9.2 | 43.2 | 57.0 | 72.8 | 33.1 | 24.0 | 37.6 | 56.2 | 39.4 |
| 3k | 20.5 | 10.8 | 45.7 | 58.6 | 76.2 | 35.3 | 25.8 | 38.3 | 53.1 | 40.5 |
| 4k | 27.2 | 11.8 | 45.7 | 58.7 | 76.6 | 35.9 | 29.2 | 41.0 | 31.2 | 39.7 |
| 5k | 28.9 | 14.2 | 49.3 | 60.2 | 77.9 | 38.8 | 32.8 | 41.7 | 53.1 | 44.1 |
| 6k | 31.9 | 15.0 | 51.5 | 62.0 | 79.0 | 39.2 | 33.3 | 41.4 | 68.8 | 46.9 |
| 7k | 31.5 | 16.8 | 51.9 | 63.2 | 77.9 | 36.5 | 35.9 | 43.8 | 43.8 | 44.6 |
| 8k | 30.3 | 13.8 | 51.9 | 63.7 | 80.6 | 38.3 | 36.1 | 41.3 | 59.4 | 46.2 |
| 9k | 30.6 | 14.0 | 52.7 | 62.6 | 78.7 | 37.5 | 36.1 | 43.2 | 43.8 | 44.4 |
| 10k | 30.3 | 16.0 | 54.2 | 63.8 | 79.5 | 37.3 | 37.2 | 44.2 | 46.9 | 45.5 |

**Table 26: Full evaluation results of Llama-2 continual pre-training on OpenWebMath with ProX-D+C. Note that about 1B tokens are trained per 1000 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 14.1 | 3.8 | 39.5 | 51.6 | 63.6 | 30.9 | 12.5 | 32.9 | 34.4 | 31.5 |
| 1k | 18.8 | 6.8 | 40.1 | 54.4 | 66.1 | 29.7 | 22.9 | 35.6 | 53.1 | 36.4 |
| 2k | 23.1 | 8.6 | 45.7 | 56.5 | 72.7 | 30.7 | 25.1 | 35.6 | 46.9 | 38.3 |
| 3k | 23.4 | 11.8 | 47.9 | 59.1 | 74.6 | 30.4 | 28.2 | 38.3 | 59.4 | 41.5 |
| 4k | 25.2 | 14.2 | 49.0 | 57.8 | 72.7 | 32.8 | 33.1 | 40.7 | 40.6 | 40.7 |
| 5k | 24.4 | 13.6 | 48.0 | 58.7 | 72.1 | 28.9 | 33.0 | 40.6 | 50.0 | 41.0 |
| 6k | 29.6 | 12.8 | 46.1 | 63.4 | 75.6 | 33.7 | 31.6 | 42.8 | 53.1 | 43.2 |
| 7k | 29.9 | 13.6 | 50.5 | 61.5 | 75.2 | 36.4 | 34.5 | 41.7 | 53.1 | 44.0 |
| 8k | 30.2 | 15.8 | 50.8 | 63.7 | 77.1 | 37.7 | 36.3 | 43.4 | 43.8 | 44.3 |
| 9k | 34.0 | 15.4 | 52.1 | 62.4 | 79.3 | 35.9 | 40.2 | 44.0 | 56.2 | 46.6 |
| 10k | 30.6 | 16.8 | 50.2 | 63.7 | 79.3 | 37.3 | 40.1 | 43.8 | 53.1 | 46.1 |

**Table 27: Full evaluation results of CodeLlama-7B continual pre-training on OpenWebMath with raw data. Note that about 1B tokens are trained per 250 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| 250 | 16.7 | 8.2 | 45.2 | 52.2 | 65.3 | 33.9 | 16.0 | 28.8 | 43.8 | 34.5 |
| 500 | 18.3 | 7.8 | 43.1 | 53.9 | 69.0 | 29.3 | 15.3 | 22.5 | 37.5 | 33.0 |
| 750 | 20.2 | 8.0 | 45.2 | 54.2 | 71.9 | 29.9 | 17.1 | 31.2 | 37.5 | 35.0 |
| 1000 | 24.7 | 9.8 | 40.6 | 58.6 | 72.7 | 29.3 | 20.7 | 31.9 | 34.4 | 35.9 |
| 1250 | 24.3 | 10.4 | 44.0 | 57.5 | 74.8 | 29.2 | 21.4 | 36.1 | 50.0 | 38.6 |
| 1500 | 26.2 | 13.2 | 48.4 | 58.8 | 75.4 | 29.4 | 28.1 | 34.9 | 50.0 | 40.5 |
| 1750 | 25.5 | 11.8 | 49.1 | 58.7 | 76.6 | 32.4 | 26.7 | 37.3 | 43.8 | 40.2 |
| 2000 | 28.0 | 13.6 | 46.3 | 61.7 | 80.0 | 33.8 | 29.4 | 37.2 | 50.0 | 42.2 |
| 2250 | 27.7 | 13.6 | 48.9 | 62.2 | 80.3 | 32.5 | 28.9 | 39.1 | 59.4 | 43.6 |
| 2500 | 31.1 | 14.8 | 51.4 | 62.1 | 81.2 | 33.6 | 30.4 | 40.5 | 43.8 | 43.2 |

**Table 28: Full evaluation results of CodeLlama continual pre-training on OpenWebMath with ProX-D. Note that about 1B tokens are trained per 250 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| 250 | 21.1 | 9.2 | 48.7 | 56.1 | 71.3 | 33.4 | 22.2 | 34.1 | 50.0 | 38.5 |
| 500 | 23.7 | 11.6 | 49.8 | 57.4 | 74.7 | 32.9 | 28.5 | 35.8 | 59.4 | 41.5 |
| 750 | 25.1 | 15.4 | 48.1 | 58.9 | 78.8 | 36.8 | 29.4 | 37.6 | 53.1 | 42.6 |
| 1000 | 28.4 | 14.2 | 50.9 | 61.2 | 79.8 | 36.7 | 27.7 | 37.6 | 50.0 | 42.9 |
| 1250 | 33.0 | 15.2 | 49.3 | 62.9 | 81.1 | 33.4 | 32.8 | 41.0 | 46.9 | 44.0 |
| 1500 | 36.0 | 15.0 | 54.2 | 65.0 | 81.0 | 39.3 | 34.1 | 42.0 | 62.5 | 47.7 |
| 1750 | 34.7 | 14.6 | 53.1 | 63.6 | 83.3 | 40.6 | 35.9 | 43.4 | 62.5 | 48.0 |
| 2000 | 35.7 | 17.6 | 53.3 | 65.4 | 83.5 | 42.4 | 37.1 | 42.4 | 56.2 | 48.2 |
| 2250 | 37.2 | 18.8 | 54.5 | 65.4 | 83.2 | 41.9 | 41.0 | 44.9 | 71.9 | 51.0 |
| 2500 | 38.1 | 17.0 | 54.2 | 67.0 | 83.1 | 40.9 | 39.8 | 43.7 | 50.0 | 48.2 |

**Table 29: Full evaluation results of CodeLlama continual pre-training on OpenWebMath with ProX-D+C. Note that about 1B tokens are trained per 250 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 11.8 | 5.0 | 44.2 | 50.7 | 62.6 | 30.6 | 14.3 | 20.4 | 21.9 | 29.1 |
| 250 | 18.1 | 10.2 | 46.0 | 54.5 | 71.9 | 33.0 | 21.3 | 34.4 | 50.0 | 37.7 |
| 500 | 22.4 | 10.0 | 50.3 | 59.7 | 76.4 | 31.3 | 26.1 | 36.0 | 59.4 | 41.3 |
| 750 | 26.8 | 11.4 | 51.2 | 61.0 | 78.5 | 34.9 | 26.4 | 38.0 | 53.1 | 42.4 |
| 1000 | 29.0 | 14.4 | 54.1 | 62.8 | 80.1 | 36.9 | 34.2 | 40.4 | 62.5 | 46.0 |
| 1250 | 31.4 | 15.0 | 51.7 | 63.8 | 81.1 | 37.2 | 32.5 | 41.4 | 75.0 | 47.7 |
| 1500 | 31.5 | 17.4 | 53.4 | 64.4 | 80.7 | 39.6 | 35.4 | 41.6 | 71.9 | 48.4 |
| 1750 | 33.7 | 15.2 | 50.6 | 64.3 | 81.5 | 39.2 | 36.1 | 40.5 | 53.1 | 46.0 |
| 2000 | 36.2 | 16.0 | 54.7 | 65.1 | 83.1 | 39.9 | 39.1 | 43.4 | 71.9 | 49.9 |
| 2250 | 37.1 | 16.6 | 55.3 | 65.6 | 82.4 | 41.3 | 36.5 | 42.7 | 75.0 | 50.3 |
| 2500 | 35.6 | 17.6 | 55.8 | 67.9 | 82.7 | 41.3 | 38.9 | 42.6 | 62.5 | 49.4 |

**Table 30: Full evaluation results of Mistral-7B continual pre-training on OpenWebMath with raw data. Note that about 1B tokens are trained per 1000 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 40.6 | 11.4 | 65.4 | 68.5 | 87.0 | 52.9 | 32.3 | 50.0 | 56.2 | 51.6 |
| 1k | 31.6 | 12.0 | 56.5 | 66.0 | 80.1 | 43.9 | 27.1 | 45.1 | 56.2 | 46.5 |
| 2k | 32.4 | 10.8 | 54.7 | 63.5 | 82.6 | 40.8 | 31.6 | 45.7 | 59.4 | 46.8 |
| 3k | 33.6 | 14.8 | 60.4 | 64.7 | 84.5 | 43.5 | 33.1 | 47.2 | 68.8 | 50.1 |
| 4k | 35.1 | 14.8 | 58.7 | 65.2 | 84.4 | 41.2 | 38.5 | 47.3 | 62.5 | 49.7 |
| 5k | 33.4 | 16.0 | 59.3 | 65.0 | 83.8 | 46.7 | 34.6 | 49.1 | 62.5 | 50.0 |
| 6k | 38.7 | 16.6 | 61.5 | 68.1 | 86.1 | 47.4 | 35.3 | 48.5 | 37.5 | 48.9 |
| 7k | 39.6 | 17.2 | 60.5 | 68.2 | 86.2 | 44.4 | 38.5 | 49.3 | 53.1 | 50.8 |
| 8k | 44.0 | 16.4 | 64.5 | 69.8 | 88.7 | 45.5 | 41.3 | 50.6 | 59.4 | 53.4 |
| 9k | 43.9 | 19.4 | 63.7 | 69.7 | 87.6 | 44.9 | 42.9 | 51.0 | 62.5 | 54.0 |
| 10k | 44.4 | 19.2 | 65.2 | 69.6 | 88.4 | 46.6 | 43.1 | 50.8 | 65.6 | 54.8 |

**Table 31: Full evaluation results of Mistral-7B continual pre-training on OpenWebMath with ProX-D. Note that about 1B tokens are trained per 1000 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 40.6 | 11.4 | 65.4 | 68.5 | 87.0 | 52.9 | 32.3 | 50.0 | 56.2 | 51.6 |
| 1k | 36.8 | 14.6 | 57.2 | 66.1 | 83.1 | 45.7 | 32.6 | 47.7 | 59.4 | 49.2 |
| 2k | 38.5 | 17.0 | 57.9 | 69.0 | 86.3 | 44.7 | 33.6 | 49.2 | 56.2 | 50.3 |
| 3k | 40.0 | 19.0 | 59.3 | 68.7 | 87.0 | 46.8 | 41.0 | 48.0 | 68.8 | 53.2 |
| 4k | 38.5 | 20.4 | 59.3 | 66.2 | 85.1 | 42.6 | 42.8 | 49.5 | 68.8 | 52.6 |
| 5k | 42.5 | 20.2 | 63.0 | 70.5 | 86.6 | 47.2 | 43.4 | 49.8 | 62.5 | 54.0 |
| 6k | 46.8 | 17.8 | 62.5 | 72.7 | 88.2 | 51.2 | 47.7 | 51.3 | 56.2 | 54.9 |
| 7k | 47.5 | 22.4 | 64.1 | 71.8 | 89.1 | 51.4 | 47.9 | 52.4 | 65.6 | 56.9 |
| 8k | 44.6 | 23.8 | 63.2 | 70.8 | 87.7 | 47.6 | 49.1 | 54.1 | 65.6 | 56.3 |
| 9k | 46.6 | 24.6 | 61.6 | 72.3 | 86.4 | 46.9 | 49.8 | 53.2 | 65.6 | 56.3 |
| 10k | 46.7 | 22.6 | 63.5 | 72.4 | 88.9 | 48.3 | 48.2 | 54.1 | 62.5 | 56.4 |

**Table 32: Full evaluation results of Mistral-7B continual pre-training on OpenWebMath with ProX-D+C. Note that about 1B tokens are trained per 1000 steps.**
| Train<br>Steps | GSM8K | MATH | SVAMP | ASDiv | MAWPS | TAB | MQA | MMLU<br>STEM | SAT<br>MATH | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 40.6 | 11.4 | 65.4 | 68.5 | 87.0 | 52.9 | 32.3 | 50.0 | 56.2 | 51.6 |
| 1k | 30.9 | 16.0 | 60.1 | 64.5 | 85.3 | 40.8 | 33.9 | 48.0 | 59.4 | 48.8 |
| 2k | 40.3 | 17.6 | 63.0 | 66.3 | 86.2 | 48.0 | 33.9 | 48.7 | 53.1 | 50.8 |
| 3k | 42.4 | 17.8 | 59.6 | 69.1 | 85.7 | 50.1 | 38.5 | 49.9 | 59.4 | 52.5 |
| 4k | 43.8 | 20.4 | 63.7 | 69.3 | 88.2 | 46.2 | 46.3 | 50.9 | 65.6 | 54.9 |
| 5k | 42.5 | 18.4 | 59.3 | 69.6 | 87.9 | 44.3 | 46.1 | 51.9 | 65.6 | 54.0 |
| 6k | 47.7 | 21.8 | 62.7 | 71.7 | 89.2 | 47.9 | 48.4 | 54.0 | 68.8 | 56.9 |
| 7k | 46.8 | 21.6 | 62.9 | 72.1 | 88.4 | 50.1 | 46.4 | 52.5 | 68.8 | 56.6 |
| 8k | 48.4 | 21.6 | 65.0 | 72.7 | 89.2 | 51.1 | 49.4 | 52.9 | 65.6 | 57.3 |
| 9k | 48.5 | 24.8 | 64.4 | 72.6 | 88.3 | 50.7 | 48.1 | 53.4 | 62.5 | 57.0 |
| 10k | 51.0 | 22.4 | 64.9 | 72.9 | 89.2 | 49.8 | 53.0 | 54.2 | 75.0 | 59.2 |

### Appendix E Analysis

#### E.1 Case Studies

We provide several cases to qualitatively illustrate the refinement effect of ProX, as shown in Tables 33-34. For the general domain, using RedPajama-V2 as an example, we observe that ProX can drop low-information documents, remove meaningless content such as navigation bars, and replace URL links (see Table 33). In the mathematics domain, ProX demonstrates the ability to eliminate documents with minimal relevance to mathematical reasoning and remove less important elements like functional buttons (see Table 34). These refinements enhance the quality and relevance of the processed data across different domains.

**Table 33: Cases from RedPajama-V2 after applying ProX. Text in red indicates content to be removed or replaced. “...” denotes omitted content due to limited space.**
| Case 1 |
| --- |
| TagCollegeEducationJournalismWar<br>: Michael Lewis<br>ContributorMichael Lewis<br>Michael Lewis is possibly the most entertaining nonfiction writer alive. If that’s not true it’s at least close to true. Liar’s Poker, Moneyball, The Blind Side, his NYT article about Jonathan Lebed (Google it): what’s not to love?<br>504: How I Got Into College<br>Act Two: My Ames is True<br>Writer Michael Lewis tells the story of a man named Emir Kamenica, whose path to college started with fleeing the war in Bosnia and becoming a refugee in the United States. Then he had a stroke of luck: a student teacher read an essay he’d plagiarized from a book he’d stolen from a library back in Bosnia, and was so impressed that she got him out of a bad high school and into a much better one.<br>Act Three<br>Michael Lewis’ story continues, and he figures out why Emir Kamenica insists on remembering, and telling, the story of his life the way he does — even when he finds out that some of the facts may be wrong. |
| Output by ProX:<br>drop_doc() |
| Case 2 |
| Home > Staff > Staff search > Dr Tim Overton<br>Dr Tim Overton BSc PhD<br>School of Chemical EngineeringSenior Lecturer<br>Telephone (+44) (0) 121 414 5306Emailt.w.overton@bham.ac.uk<br>AddressSchool of Chemical EngineeringUniversity of Birmingham<br>B15 2TT<br>Dr Tim Overton is a biochemist and molecular microbiologist who is interested in applying molecular biology and single-cell techniques to understand and develop bioprocesses. He is active in microbial flow cytometry research and collaborates widely with bioprocess engineers, molecular microbiologists, cell biologists and environmental microbiologists to develop new methods of answering fundamental questions on a single-cell level.<br>His research also focuses on using bacteria to make useful products such as protein drugs and small molecules, and the bacterial responses to stress encountered in such processes. Current and recent research funding has come from the BBSRC, TSB and EU FP7. He is the director of the MSc in Biochemical Engineering. Pages: 1 3 4<br>...<br>Google scholar: http://scholar.google.co.uk/citations?user=tF_eBKEAAAAJ<br>... |
| Output by ProX:<br>keep_doc()<br>remove_lines(line_start=0, line_end=5)<br>normalize(source_str="http://scholar.google.co.uk/citations?user", target_str="")<br>normalize(source_str="Pages: 1 3 4", target_str="")<br>... |

**Table 34: Cases from OpenWebMath after applying ProX. Text in red indicates content to be removed or replaced. “...” denotes omitted content due to limited space.**
| Case 1 |
| --- |
| ## unhybridized pi bonds<br>$sp,sp^{2},sp^{3},dsp^{3},d^{2}sp^{3}$<br>Tatiana 4B<br>Posts: 30<br>Joined: Fri Sep 28, 2018 12:28 am<br>### unhybridized pi bonds<br>...<br>### Re: unhybridized pi bonds<br>I am not too sure in my knowledge about this, but I think that both have hybridized orbitals. Since hybridization is defined as the phenomenon of intermixing of the orbitals such as sp, sigma and pi bonds are just different types of covalent bonds formed depending on the way the atomic orbitals hybridize with each other. Sigma bonds are a result of when the overlap of orbitals of two atoms takes place along the line joining the two orbitals, while pi bonds are when two atoms overlap due to the sideways overlap of their ’p’ orbitals.<br>Hannah Yates 1K<br>Posts: 59<br>Joined: Fri Sep 28, 2018 12:27 am<br>### Re: unhybridized pi bonds<br>I am also not too sure on my answer, but I am pretty sure that a sigma bond has just hybridized orbitals, but the reason a pi bond can form is because of an extra (not hybridized) p orbital. This allows for a double and triple bond to form. |
| Output by ProX:<br>drop_doc() |
| Case 2 |
| Solution - Trigonometric Identities<br>Account<br>Register<br>Share<br>Books Shortlist<br>ConceptTrigonometric Identities<br>Question<br>Prove the following trigonometric identities:<br>$(\text{i})\text{ }\frac{\sin\theta}{1-\cos\theta}=\text{cosec}\theta+\cot\theta$<br>Solution<br>You need to to view the solution<br>Is there an error in this question or solution?<br>Reference Material<br>Solution for concept: Trigonometric Identities. For the course CBSE<br>S |
| Output by ProX:<br>keep_doc()<br>remove_lines(line_start=0, line_end=7)<br>remove_lines(line_start=18, line_end=24) |

#### E.2 Computing Overhead Analysis

According to Kaplan et al. [], both training and inference computational FLOPs for Transformer-based Language Models (denoted as $C_{\text{train}}$ and $C_{\text{inference}}$) can be approximated as the product of model parameters (non-embedding parameter) $N$ and the number of tokens $D$.
This can be expressed as:

$$ $C_{\text{train}}\approx 6\cdot{N}D_{\text{train}},$ (9) $$

$$ $C_{\text{inference}}\approx 2\cdot{N}\left(D_{\text{prefill}}+D_{\text{decode} }\right).$ (10) $$

In ProX, we go through two data refining stages before final training, which incurs additional inference-time computational FLOPs. Suppose the refining model parameter for each stage is denoted as $N_{\text{refine}}$, and the raw data size in tokens is $D_{\text{raw}}$.

For the first document-level stage, the computational cost can be approximated as:

$$ $C_{\text{doc}}\approx 2\cdot N_{\text{refine}}\left(D_{\text{raw}}+D_{\text{ output}}\right)\approx 2\cdot N_{\text{refine}}D_{\text{raw}},{~{}~{}~{}~{}( \text{suppose }D_{\text{output}}\ll D_{\text{raw}})}$ (11) $$

resulting in a new pool of data sized $D_{\text{doc}}$.

Similarly, for the second chunk-level stage, the computational cost is:

$$ $C_{\text{chunk}}\approx 2\cdot N_{\text{r}}\left(D_{\text{doc}}+D_{\text{ output}}\right)\approx 2\cdot N_{\text{r}}D_{\text{doc}},{~{}~{}~{}~{}(\text{ suppose }D_{\text{output}}\ll D_{\text{doc}})}$ (12) $$

which produces the final refined data size of $D_{\text{ProX}}$.

Thus, the total computational overhead for ProX can be calculated as the sum of the two stages:

$$ $C_{\text{{ProX}}}=C_{\text{doc}}+C_{\text{chunk}}\approx 2\cdot N_{\text{doc_ refine}}D_{\text{raw}}+2\cdot N_{\text{chunk_refine}}D_{\text{doc}}.$ (13) $$

In general, we use refining models with same sizes, so the final inference overhead can be estimated as

$$ $C_{\text{{ProX}}}\approx 2\cdot N_{\text{refine}}(D_{\text{raw}}+D_{\text{doc} }).$ (14) $$

Additionally, we omit the FLOPs for fine-tuning since they are negligible compared to the large-scale pre-training and inference FLOPs.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please Check Weakness.
soundness: 2
strengths: **A highly complete paper, and I appreciate it along with your hard work.**

1. The paper is clearly articulated with intuitively designed charts that help readers easily understand the problems being addressed, the research motivation, and the methods employed. 

2. It is detailed in scope, including appendices with textual explanations of document/block-level programming, algorithm flowcharts, prompts used, pretraining details, baseline, downstream task introductions, and case analyses, all of which are comprehensive. 

3. The experiments are rich in content, especially the appendices, and features beautifully crafted charts. 

4. The field this paper focuses on is crucial to the LLM community -- "how to improve the quality of pretraining data."
summary: This paper presents Programming Every Example (PROX), a framework that redefines language model pre-training by treating data refinement as a programming task. PROX allows even small language models (as few as 0.3B parameters) to perform fine-grained data operations, achieving performance comparable to human experts. Models trained with PROX-curated data outperform those trained on original or traditionally curated data by more than 2% across ten benchmarks. It is effective with different model sizes and corpora, demonstrating significant potential in domain-specific tasks and reducing computational costs.
weaknesses: 1. **This paper raises several concerns for me:**

    ``The methodology can be summarized as follows: it utilizes LLAMA's annotated pairs (doc, program) to fine-tune a small model Prox with approximately 0.3B parameters, serving as a proxy. Based on a vast corpus of pre-trained documents, Prox generates Python function calls to conduct document-level encoding (discard or retain) and chunk-level encoding (discard, retain, normalize).``

- a. Firstly, while the goal of the paper is to balance data processing efficiency and enhance data quality, the introduction of Prox as a proxy to call Python functions might add extra computational cost, potentially undermining the practical of this approach. A comparative analysis of time overhead with other methods would strengthen the argument for the effectiveness of this approach.

- b. Furthermore, if direct application of Python functions achieves the document-/chunk-level heuristic optimizations as proposed, how does this alternative compare to Prox in terms of performance? 
Because after all, the processes of discarding or retaining documents and chunks, and normalizing chunks (such as top menus, navigation bars, buttons, HTML elements, links, and footers) can be addressed using existing Python rules (as referenced with C4, Gopher, and RedPajama). 

- c. Additionally, how do the document- and chunk-level programming rules differ from existing heuristic rules, particularly in relation to the above mentioned processes (at paragraph b) and education scores, considering related research such as [1] and [2] have already proposed similar concepts?

2. The experimental results in the paper need further development. In Table 2, only a 750M LLAMA model is used for pretraining on 26B data, while the minimal data volume and model size in standard practice are at least a 1.3B model and 30B data. I believe Prox could demonstrate more significant improvements in in-context learning (ICL) capabilities with larger data scales and model sizes.

3. Some experimental comparisons require clarification. For example, in Table 5, the comparison on domain-specific data tasks should not be between CPT and the BASE model. Instead, it should be between randomly sampled data and Prox-refined data of equivalent volume size, with two BASE models trained from scratch.

4. The current paper only examines mathematics as a specific domain. Expanding the scope to include more vertical fields such as finance, education, or medicine would be beneficial.

> [1] Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. 

>[2] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high-quality data for training language models. In International Conference on Machine Learning (ICML), 2024.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: Did you compare Prox-D against an n-gram based filtering method like fastText? DataComp-LM (Li et. al.) find that fastText-based quality filtering was the best performing method they tried, outperforming the Gopher and C4 filtering rules, which are also used as baselines in this paper.
soundness: 4
strengths: - The paper introduces ProX, which is a novel method to improve training data quality using small language models to synthesize and execute short data refinement programs, rather than using static rules or large model data synthesis.
- Framing data filtering and synthetic data generation as a programming task that can be performed by small models is a creative contribution that has not been explored in previous work
- Prox-D and Prox-D+C outperform all heuristic and model based data selection baselines and improve pretraining efficiency. The empirical validation is comprehensive, testing the method across multiple model sizes (350M to 1.7B), different pre-training corpora (RedPajama-V2, C4, FineWeb), and domain-specific applications (OpenWebMath).
- The method shows significant gains in accuracy and efficiency for domain specific continual pretraining, with up to a 20x reduction in compute. The effectiveness of synthesizing example-specific filtering programs shows especially here, as the method can synthesize domain-specific filtering rules.
- Although there is additional computational overhead from running inference of the refining model to generate the programs, this rapidly decreases as a proportion of the total pretraining FLOPs as the pretrained model gets larger. The method shows a 67% FLOPs reduction in pretraining to a given accuracy for a model as small as 1.7B.
- The paper is well structured and clearly written, with nice looking and clear figures to explain the method and results.
summary: The authors propose ProX, a method for data filtering and synthetic data generation for improving the quality of large pretraining corpora. ProX proposes to fine-tune a small language model to produce a “program”, which is a sequence of one or more filtering or normalization operations applied at the document and line level. These programs are synthesized and executed for each document in the corpus, which results in a smaller higher quality subset. The authors demonstrate that ProX yields significant improvements in pretraining and domain-specific continual pretraining in both accuracy and training efficiency across numerous baselines.
weaknesses: - The paper is missing a qualitative analysis of the programs generated by ProX. It could benefit from:
  - Examples of wrong programs generated by ProX and an analysis of the common failure modes
  - Statistics on the complexity of the programs (e.g. distribution of function calls per document)
  - An analysis of how program quality varies across different domains
  - A discussion of potential safety issues since programs may modify data in unintended or harmful ways

- The paper could benefit from comparison to simpler baselines, like binary classifiers (e.g. fastText) for quality filtering.

- The paper could explore expanding the space of programs, for example by inclusion of additional operations like text transformations, or composing multiple operations.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Generalizability to Highly Refined Datasets: How would ProX perform when applied to datasets that are already highly refined, such as FineWeb-edu or DCLM-base? Are there diminishing returns when starting with higher-quality corpora?
2. Evaluation Choices: Why did the authors prioritize zero-shot performance for their evaluation? Wouldn't few-shot performance offer a more nuanced understanding of ProX’s impact, especially for instruction-following tasks?
3. Compute Cost of Refining Model Training: Why did the authors not also use an existing pre-trained model for the refining task? This could have significantly reduced the compute cost. Additionally, what measures were taken to address the long context requirements of creating refinement programs for lengthy documents?
soundness: 3
strengths: * Novel Chunk-Level Rewriting Technique: A novel and fine-grained approach that moves beyond binary keep/dismiss judgments, allowing deterministic edits to individual chunks of text. This method is validated through robust ablation studies, proving its efficacy in improving pre-training data quality.
* Efficient Use of Compute: By leveraging small language models (0.3B parameters) for data refinement, ProX achieves substantial performance improvements with lower computational costs. This efficiency is especially noteworthy in domain-specific tasks like OpenWebMath, where ProX yields remarkable accuracy gains.
* Comprehensive Ablations: The authors tested ProX across multiple model sizes (0.3B to 1.7B) and pre-training corpora (C4, RedPajama-V2, FineWeb), with significant performance enhancements demonstrated consistently. The results show that even smaller refining models can produce high-quality pre-training data.
* Important Practical Implications: The framework offers a scalable solution to pre-training data refinement, which can be particularly valuable in scenarios where human expert intervention is impractical. ProX’s deterministic chunk-level edits contribute to a more nuanced and flexible data curation process.
summary: The paper "Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale" introduces ProX, a framework leveraging small language models to refine large-scale pre-training data through program generation. ProX aims to enhance the quality of training data by executing fine-grained operations on individual examples. The authors demonstrate significant improvements in pre-training efficiency and model performance across various benchmarks and model sizes. Notably, ProX can achieve remarkable gains in domain-specific continual pre-training tasks, as the authors show with OpenWebMath.
weaknesses: * Base Dataset Selection and Novelty: The exclusive reliance on older datasets like C4 and RedPajama limits the relevance of the findings. While the results on OpenWebMath are impressive, it would be more compelling to see ProX applied to modern datasets such as FineWeb-edu or DCLM-base, which have undergone extensive refinement.
* Document-Level Refinement: The document-level component of ProX appears to be a straightforward application of FineWeb-edu’s filtering techniques, reducing its novelty. The primary innovation lies in the chunk-level refinement, which could be more explicitly distinguished from the document-level methods.
* Evaluation Methodology: The emphasis on zero-shot performance may not fully capture the advantages of ProX, especially since the refining models are not fine-tuned for instruction-following tasks. Few-shot performance with higher n might provide more insightful comparisons.
* Exclusion of Base Model Training FLOPs: The paper excludes the computational cost (approximately 5.3e19 FLOPs) for training the base model used to create the refining model. This is significant, as it constitutes nearly half the total compute, raising concerns about the true efficiency gains. The authors should have explored using an existing pre-trained model to substantiate ProX’s cost-effectiveness.
* Context Window Considerations: Given that ProX needs to refine long documents, the context window of the refining model is a critical factor. The paper does not provide sufficient details on how the model handles long contexts, which could impact its practical utility.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Each time the pre-trained model is updated, the entire pre-training dataset needs to be reprocessed, which is highly GPU-intensive. Do the authors have any alternative solutions to address this computationally expensive process?
soundness: 2
strengths: The advantage of this paper is its novelty in transforming data cleaning into a programmable task, which shifts away from traditional, rigid heuristic-based methods. By treating data refinement as a series of programmatically generated actions, the framework enables language models to apply tailored operations to each example, such as string normalization and noise filtering, with high precision.
summary: The authors introduce a framework called PROX for refining data quality in large language model (LLM) pre-training. This framework claims to address limitations in traditional, heuristic-based data-cleaning methods, which often lack flexibility and are labor-intensive.
weaknesses: 1. The first limitation is that the amount of pre-training data used in the authors' experiments is insufficient, falling far short of what would be needed to fully support their claims. It’s highly possible that, with more extensive training, the advantages of the proposed method might diminish or even disappear.
2. The second limitation is that using a model—especially a smaller one—for data selection introduces issues like hallucinations, bias, and the omission of less common information during the pre-training phase. These issues are difficult to resolve and can become deeply embedded in the foundation model. In fact, pre-training should ideally be a robust modeling of the real world, and relying on a smaller model for data refinement may skew this representation, undermining the model’s ability to capture a true and comprehensive understanding of diverse real-world contexts.

Meta-review: This paper tackles the vital issue of enhancing pretraining data quality in the large language model (LLM) community. The authors introduce Prox, a small model with approximately 0.3 billion parameters, which fine-tunes using LLAMA's annotated pairs (document, program). Prox generates Python function calls for document-level and chunk-level encoding to determine which data to retain or discard.

Reviewers have raised concerns about the feasibility of applying Prox in real-world scenarios due to its computational costs. To strengthen their argument, the authors should provide concrete evidence demonstrating Prox's effectiveness compared to other LLM corpus filters, such as DCLM, which is known for its superior performance. This could involve comparing models trained with datasets filtered by Prox against those using the same training corpus as outlined in the DCLM paper.

Decision: Reject