Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
Authors: Fan Zhou
Zengzhi Wang
Qian Liu
Junlong Li
Pengfei Liu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Pre-training
Data Refinement
Data Engineering
Primary area: deep_learning->large_language_models

Abstract: Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these fixed rules lack the flexibility to address the unique characteristics of individual examples, yet crafting sample-wise rules is impractical for human experts. In this paper, we show that even small language models, with only 0.3B parameters, can exhibit substantial data refining capabilities. We propose Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, and enables the model to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experiments show that models trained on ProX-refined data consistently outperform other baselines across 10 benchmarks, demonstrating effectiveness across model sizes (up to 1.7B) and pre-training corpora (C4, RedPajama-V2, FineWeb, FineWeb-Edu, and DCLM).
ProX also shows great potential in continual pre-training: on math domain, ProX boosts 7B models by up to 20% within 10B tokens—results typically achieved with much larger scale training (e.g., 200B tokens).
We believe ProX offers a way to curate high-quality pre-training data, and finally contributes to efficient LLM development.

Programming Every Example:
Lifting Pre-training Data Quality Like Experts at Scale
Fan Zhou 12
Zengzhi Wang" 12
Qian Liu
Junlong Li
Pengfei Liu 1 124
Abstract
creative writing (Yuan et al., 2022), complex reason- ing (Wei et al., 2022; Kojima et al., 2022), and agentic task planning and execution (Fan et al., 2022; Park et al., 2023), Behind these, massive, high-quality pre-training corpora form the backbone of these models, equipping them with the essential knowledge and reasoning abilities crucial for a wide range of downstream tasks (Penedo al., 2024a).
Large language model pre-training has tradition- ally relied on human experts craft heuristics for improving the corpora quality, resulting in numer- ous rules developed to date. However, these fixed rules lack the flexibility to address the unique characteristics of individual examples, yet craft- ing sample-wise rules is impractical for human experts. In this paper, we show that even small language models, with only 0.3B parameters, can exhibit substantial data refining capabilities. We propose Programming Every Example (PROX), novel framework that treats data refinement as a programming task, and enables the model to refine corpora by generating and executing fine- grained operations, such as string normalization. for each individual example at scale. Experiments show that models trained on PRoX-refined data consistently outperform other baselines across 10 benchmarks, demonstrating effectiveness across model sizes (up to 1.7B) and pre-training cor- pora (C4, RedPajama-Va V2, Wibe Febe Edu, and DCLM). PROX also shows great po- tential in continual pre-training: on math domain, PROX boosts 7B models by up to 20% within 10B tokens results typically achieved with much larger scale training (e.g., 200B tokens). We be- lieve PROX offers a way to curate high-quality pre-training data, and finally contributes to effi- cient L.I.M development.
The Internet offers vast amounts of data, but much of it is noisy and unrefined, requiring extensive cleaning and quality enhancement before being applied for pre-training, Previous works focus primarily on designing heuristic- pipelines to lift data quality, such as document filtering (Rae et al., 2021; Penedo et al., 2024a; Soldaini et al., 2024) and perplexity- beasee scoring methods (Together, 2023), relying heavily human expertise and manual adjustments (Zhang et al., 2024a). While widely adopted, these labor-intensive solutions are inherently limited by rule coverage and their in- ability to address every specific case. Recently, some efforts have explored leveraging L.LMs for high-quality data acqui- sition. On the one hand, language models have been applied for data filtering or selection (Xie et al., 2023; Wettig et al., 2024; Yu et al., 2024; Dubey al., 2024), but their role is largely limited to identifying low-quality documents with- out enabling fine-graines refinements (e.g., string-level). On the other hand, LLMS are also being used to generate high- quality data directly, i.e., data synthesis (Gunasckar et al., 2023; Li al., 2023; Ben Allal al., 2024). Unlike filtering, synthesis methods actively create or refine data to produce documents, but they require substantial computational resources, limiting the methods' scalability. Despite the success, these methods can also inherit issues from LLMs like hallucination (Maini et al., 2024), and assessing their correctness and completeness an interpretable manner remains a challenge (Liu et al., 2024a).
1. Introduction
Large Language Models (LLMs) have made significant strides in capabilities (Meta, 2024; Achiam et al., 2023; Anthropic, 2024; Reid et al., 2024), excelling in tasks such
At this intersection of data processing efficiency and data quality improvement, propose PROX, model-based framework for pre-training-level data refinement. PROX focuses on refining corpora using smaller models at scale, offering a more efficient alternative. As shown in Figure 2, in practice, PROX first adapts small base language models (e.g., < 1B) to data refining tasks through fine-tuning them on seed data. The refining models in PROX then determine
'Equal contribution 'Shanghai Jiao Tong University "Generative AI Research Lab (GAIR) *Sea AI Lab 'Shanghai Artificial Intelligence Laboratory. Correspondence to: Pengfei Liu <pengfei sttpeei@ssuu..du.......
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Apply ProX on Feb
Apply ProX Open Mab
Average Performance (%)
55
54
Math Performance (%)
20 lessitraining steps
50
training steps
6 6.2%
45
34B
2.0%%
40
46
35
44
1.78 model trained on ProX
OL-
Cont, trained on PreX trined on Ongg
M-ATTL
1.78 model trained on Orig.
Pythin- .BB
CadeLLaMA
TinyLIaM4-1.IB
Liemma
1.9k
Fizie (x1e")
80
FT.PP
Left: Pre-training from scratch on general domain.
Right: Continual pre-training on math domain.
Figure I: Training FLOPs vs. downstream performance. Although these corpora have been processed through expert-crafted rules, applying PROX still yields significant improvements over these baseline models trained with the original corpora. Moreover, models trained on PROX curated data achieve competitive performance with much fewer training FLOPs.
the appropriate operations for each document in the pre- training corpora via versatile programs, such as document filtering, string normalization and noisy line removal. The generated programs are then executed by a pre-defined ex- ecutor, producing refined corpus ready for pre-training. this way, PROX empowered with language models au- tonomously refine pre-training corpora, leveraging flexible function calls to enhance data quality.
cess of transforming d into d, where d exhibits higher quality. While it is challenging to formally define "higher quality" for pre-training data, we assume it can be described through qualitative improvements, such as the removal of adver- tisements, meaningless URL links, random code gibberish, and content lacking educational value, just as shown on the left side of Figure 2. Specifically, we formulate this refin- ing process the generation of a data processing program Z, conditioned on d. The refined document d is then pro- duced by executing program Z on the original document d. For instance, the "string normalization' can be a very fine-grained process transforming noisy strings into clean ones with executor & and program Zoormalize
Experimental results demonstrate that the proposed PROX framework consistently lifts data quality for pre-training. Specifically, PROX achieves an average improvement of 2.5% over the original corpus on 10 downstream bench- marks and outperforms state-of-the-art data selection meth- ods by over 1.0% ($3.2). Furthermore, PROX demon- strates broad applicability across model sizes from 0.3B to 1.7B and achieves consistent performance gains across diverse pre-training corpora of varying quality, includ- ing RedPajama- V2 (Together, 2023), C4 (Raffel et al., 2020), Web, FineWeb-EAe (Penedo et al., 2024a), and DCLM (Li al., 2024) ($3.3). In domain-specift con- tinual pre-training, shown in Figure refined OpenWebMath (Paster et al., 2024) boosts performance across 9 mathematical tasks (e.g., +20.3% on CODEL- LAMA-7B) while significantly enhancing efficiency, achiev- ing similar downstream results with up to 20x less com- puting ($3.4). Quantitative analysis suggests scaling re- finement FLOPs achieves comparable performance with much lower training costs, offering an efficient path for LLM pre-training ($4).
EZZormalize, d) (s
(1)
where 8/ = normalize(s:) 8; needs normalization; oth- erwise, 5/ = st. Here, (s1, $2,.... a) is the orig- inal document represented a sequence of strings, and normaliza () is our normalization function that maps certain strings to their normalized versions for simplicity. Moreover, document filtering is special case of refining transformation, where executing Zitter removes the entire document, i.e., E(Ziterr d) = Ø. In this way, data quality improvements like cleaning or normalizing can be unified into standardized functions executed by E(Z,d) where Z encodes function calls heuristics for each specific task.
2.2. PROX Framework
Overview As shown in Figure given any document d as input, PROX utilizes the language model itself with param- eter 0 to generate the data refinement program Z = (d)).. The snippet is executed within the executor ε, producing the refined document d = (d) d). We include two stages in the PROX framework, aiming to refine the data progressively, from rough fine-grained. These two stages
2. Programming Every Example
2.1. Data Refinement Task Formulation
Given any document in the corpus d E D, such as an HTML
extract or textbook, we define data refinement as the pro-
2

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
ProX: Programming Every EXample
Pre training Corpora
Refined Corpora
[020] Activities I Tachnical Reports Aboutt WaC1 [001] A description of the type of authorized interactions can have with resource. Examples
ProX programs
[001] A description of the type authorized
Base
Model
Chunk level Program
interactions can have with resource. Examples include read, write, execute, add, modily, and delete.
induude read, wrile, execute, add, modify, and delate. ClickHare
normalies Glickeeee', Sinest, remove, Inast D8, 1099
[106] htppt/t.coo ca/3U0722y33
Adapt
Doc-level Program
[109] Search English sta starting with the lottor
keep dlo()
+
Noisy Mixed-quality
Refining Model
Clean & High-quality
Generate
Excccute
Figure 2: An overview of PROX framework: (1) adapt a base language model for data refinement; (2) generate elaborate programs for each document by PROX refining models, including document-level filtering and more fine-grained chunk- level refining; (3) execute the programs with the docs via a
Python executor, producing the refined high-quality corpora.
Table 1: PROX program design of document and chunk level refining stage. doc and chunk will also be sent into the corresponding functions as inputs for execution.
ing specific lines with remove 1 lines () and replacing strings with normalize (), providing flexibility in mod- ifying content rather than dropping the whole document. For high-quality chunks that require no modifications, we use the keep.chunk () function. As shown in Table 1, while the individual functions may seem straightforward, their design space is flexible and capable of expressing complex rules developed by human experts. We believe expert-crafted rules can be projected into the program space of PROX, demonstrating that our approach simplifies and enhances the rule creation process, offering more systematic and scalable refinement capabilities.
Stage
Function Interface
Document Level
drop doc () -> <None> "Delete the whole doc."
kesp_doc () <atr> "Return the orignal doc." !!!
remove_lines (line_start, line_nnnneeeee
<str>>
"Delete noisy lines from chunk."
b 1 -iin_starrt stt.stttttnntttt <in>>, index of the first line to be removed
Line www.eenddddn...... index of the last line removed
Chunk Level
nornal (source str, target str) + <str>
" "Replace strings with nornalized ones." "0
b source.ste sstttrr...ssss......... the noisy string pattern b targgt.ettrsttee the string for replacement
keep-chund () - <str> "Return the orignal chunk." "
PROX Execution During the execution stage, the execu- tor & executes the generated program snippets 2 to refine the document. PROX integrates Pythonic grammars, wrapping operations into functions with parameters, implemented in Python for later execution. For example, in Figure 2, the document contains noisy patterns such as navigation bars, meaningless HTML links, and page indexes. The refin- ing model generates programs to remove these lines and patterns. At the document- and chunk-level programming stages, PROX uses two refining models to generate pro- grams with various function calls (see Table 1). We believe this sequential approach ensures structured and effective refinement, first addressing larger document noise, and then focusing on finer-grained cleaning.
are referred to as document-level programming and chunk- level programming, illustrated in Figure 2. In each stage, the PROX refining model will generate programs Zane and Zehunk to refine corpora at varying levels of granularities.
PROX Program Design Designing an effective program space is key to maximizing language model capabilities. For large-scale pre-training corpora, we prioritize: (1) lightweight models capable of recognizing specific patterns, and (2) simplicity and efficiency despite higher computa- tional costs compared to heuristic- based pipelines. To bal- ance functionality and resource constraints, we let models generate function calls without detailed implementations, ensuring effective document manipulation while maintain- ing coherence. We present the function definitions in Ta- ble 1, which also constitutes the program space of PROX.
2.3. Model Adaptation for PROX
It is generally difficult for off-the-shelf models to directly generate perfect PROX programs. In fact, generating such custom API calls is relatively challenging even for the most powerful LLMs the current stage (Zhuo et al., 2024). Thus, it is necessary curate some seed data to adapt the model for these scenarios. Under such consideration, we employ advanced LLMs to annotate these operations via zero-shot and shw-shot prompting, and then adapt our small models to these tasks by supervised fine-tuning (SFT). As presented in Figure 3, we first apply additive scale scoring
The most fundamental operations we aim to perform on document are deletion and replacement. In PROX, we incorporate these types of operations across different stages to refine the corpus at different granularities: (1) In the document-level programming stage, we define the functions drop doc () to delete a document and keep doc () to retain it. (2) At the chunk-level programming stage, we split lengthy documents into smaller chunks and apply fine- grained operations to them. These operations include delet-
keep.doc
3

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Seed Documents
LLM
Document- Document-Program Pairs
Base Model
Refining Model
level Program crop_ docc
Fine-tume
Inference At Scale
(Scoring Critiques]
Synthesize
[Function Definition]
dropp dop_docctext str): "delete doc from corpus" pass
Chunk Program normalizet * (
Pretraining Corpora
Zero-shot
/ Few-shot
Figure 3: The illustration of the model adaptation in PROX. We employ powerful LLMs (LLAMA-3) annotate random seed documents with valid programs and use doc-program pairs fine-tune small base language model, obtaining the refining model suitable for fine-grained data refining tasks.
prompts (Yuan et al., 2024; Penedo et al., 2024a), to split the corpus into kept and dropped documents, then use LLMs to annotate fine- tieeggained programs for kept documents. Specifi- cally, we leverage the LLAAA-3-700 model (Dubey al., 2024) for seed data annotation, and the seed documents are randomly sampled from the original pre-training corpus. In PROX, this annotation is performed only once, and all mod- els are adapted with the same curated data. To ensure the reliability of the collected data, we also conduct necessary checks for grammar correctness and control the removal ratio threshold. Detailed procedures for program synthesis and proceesocesnng can be found in $A.1.
3.1. Experiment Setup
Pre-training Corpora For the general domain, we be- gin with RedPajama-V2 (Together, 2023), a preprocessed large-scale dataset of 30 trillion tokens from diverse Internet sources, ready for pre-training. We further apply PROX on the C4 corpus (Raffel et al., 2020) with 198 billion tokens and the recent high quality datasets including Web (as well as FineWeb-Edul (Penedo et al., 2024a) and DCLM (Li et al., 2024). For specific domain experiments, we use WebMathhtth (Paster et al., 2024), a math-focused dataset with 15 billion tokens. Table 8 ($B.2) reports full details of our training corpus.
For simplicity and transparency, we use a small base lan- guage model (e.g., 0.3B parameters) that we trained from scratch on approximately 26B tokens of original unrefined data, ensuring it is overtrained beyond the Chinchilla opti- mal points (Hoffmann et al., 2022). This model serves as the base model for ProX adaptation and also as the com- parison baseline in subsequent experiments. The adapted models' performance will then be evaluated using the F1 score the held-out validation dataset, both of which were derived from the seed data we collected earlier. We select the seahhor-perrorming checkpoints and employ them to generate programs Z, for each document or chunk. These programs together with the documents are then executed us- ing the corresponding function implementation, resulting in the final processed corpus. Please refer to the appendix for more training details ($A.2), implementation for calculating the F1 score ($ ($,.)), and large-scale inference ($A.4).
Base Models Our experiments are conducted on various sizes of language models. (1) To verify different stages' ef- fectiveness of PROX, we employ 0. 7B sized model sharing LLAMA-2 architecture (Touvron et al., 2023b), denoted as TLM-S, used for both pre-training from scratch and refining, We also compare PROX with data selection methods using PYTHIA-410 B's architecture (Biderman et al., 2023), as those employed in MATES (Yu et al., 2024). (2) For further evaluation of PROX using different refining and base model sizes, we scale model sizes from 0.3B (0.5 smaller, denoted as TLM-XS) to 1.7B (2x larger, denoted TLM- M). (3) For domain-specific continual pre-training, we select TINYLLAMA-L.I (Zhang et al., 2024b), LLAMA-2 (Tou- vron et al., 2023b), CODELLAMA (Rozière et al., 2023) and MISTRAL-7B (Jiang et al., 2023) as representative base models for their adequate training and solid performance. Detailed specifications and training recipes are provided in $B.3, especially in Table 9 and Table 10.
3. Experiments
In this section, we first describe our experimental setup ($3.1), then verify the effecctivnness of each PROX refining stage and compare it with various data selection methods tailored for pre-training ($3.2). We then apply PROX to various model sizes and corpora to demonstrate its broad applicability ($3.3). Finally, we apply PROX to the mathe- matical domain, showing its superiority in domain-specifff continual pre-training ($3.4).
Baselines To ensure a fair comparison within the same experiment, we maintain consistent training hyperparam- eters across most of the baselines, differing only in data refining and selection pipelines. We compare PROX with various baseline methods, including: (1) heuristic filtering rules used to create Gopher (Rac et al., 2021). C4 (Raf- fel et al., 2020), and FineWeb (Penedo et al., 2024a)), (2) fasttext-based filtering, trained on our PROX's 's seed data, (3) existing data selection techniques, including DSIR (Xie

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 2: Detailed performance on 10 downstream tasks. All models use the same TLM-S architecture and are trained on RedPajama- V2. The doc-level (PROX-D) and chunk-level (PRoX-C) refining are done by fine-tuning the raw data pre-trained model as a refining model. Bolded entries represent the best results. #Win represents the number of tasks where the method achieved the best performance.
Method
ARC-C
ARC-E
CSQA
HellaS
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
#Win
Raw
26.1
44.3
29.7
39.1
27.3
29.2
66.9
39.0
52.0
67.4
42.1
0/ 10
Applying Rule-based filtering on Raw Data:
Go
Gopher rules, C4
C4 rules, Fw
Fine rules.
Go
25.7
44.0
31.3
40.2
27.3
29.0
66.3
39.0
51.2
68.9
42.3
0/10
C4
25.0
46.0
31,0
40.5
27.1
29.2
68.5
40.5
51.7
66.6
42.6
2/10
FW
25.2
46.8
32.6
39.6
27.2
29.0
66.5
39.4
52.4
69.2
42.8
2/
Go+C4+FW
25.2
43.9
30.0
41.9
27.5
31.0
67.0
39.9
51.9
65.3
42.3
0/10
FASTTEXT
26.9
49.9
29.5
39.0
28.5
31.8
64.7
39.6
52.1
70.4
43.3
2/10
Applying PROX (ours) on Raw Data: D = Doc-level Programming, C = Chunk-leve Programming.
PROX-D PROX-D4C
26.6
49.7
30.1
40.5
29.4
30.4
66.3
39.0
51.2
71.6
43.5
1/10
26.4
51.9
30,9
42.4
29.4
31.6
67.9
40.0
52.2
73.5
44.6
3/10
et al., 2023), DsDm (Engstrom et al., 2024), MATES (Yu
45.0
Average Performance (%)
et al., 2024), QuRating (Wettig et al., 2024), and (4) LLM synthesis approaches (such as INSTLM (Cheng et al., 2024)
42.5
and COSMO (Ben Allal et al., 2024)) or LLM Pruning (SHEAREDDLAMA (Xia et al., 2024)). For domain-specific continual pre-training we also compare with strong open-
40.0
ProX-D+C
ProX-D
37.5
sourced models such LLEMMA (Azerbayev al., 2024),
FasiText
and INTERNLM2-MATE (Ying al., 2024). Please refer to $C for full baseline details.
Rule
35.0
Raw
0.0
2.5
5.0
7.5
10.0
12.5
We compare the trained models' perfor- mance over vast of datasets for comprehensive evaluation: (1) For general pre-training, we evaluate performance across ten selected tasks using lighteval's implementation (Fourrier et al., 2023); we have also included LM-eval haal-harnesss (Bi- derman et al., 2024) for fair comparison with data selection methods. (2) For -omaii--peccfic continual pre-training evaluation, we integrate nine mathematical related tasks and report few-shot of-thugh (CoT) (Wei et al., 2022) performance. The selected evaluation benchmarks, number of evaluation examples, and full details can be found in $D.
Evaluation Setup
Training Step (K)
Figure 4: Downstream performance w.r.t. different training steps: first 0.5K, then evenly from 2.5K to 12.5K. Rule: the best performing Fine rule in Table 2.
and different rule-based baselines. Moreover, we visualize the dynamic benchmark performance in Figure 4, implying the consistent improvement of PROX over all baselines. See $E.1 for full detailed results all intermediate checkpoints.
These results show that PROX is highly effective, outper- forming the raw corpus with average boost of 2.5%, including significant boosts such as 7.6% on ARC-E, and 3.3% on HellaSwag. Such improvements were achieved even on benchmarks that are typically prone to performance instability, such as SIQA, WinoGrande, and CSQA. By con- trast, rule-based methods demonstrate relatively marginal overall improvement. For instance, Gopher rules achieve only 0.2% boost, while C4 shows modest 0.5% improve- ment. Furthermore, combining all three rules (as is done in constructing the official Fine corpus), does not lead to any larger enhancement in overall performance.
3.2. Verifying PROX's effectiveness
Verifying Effectiveness for Each PROX Operation We first conduct series of experiments to verify the effec- tiveness of each PROX operation. We begin by training TLM-S on the RedPajama-V2 raw data for approximately 26B tokens (or 12.5K steps) as the initial baseline. Follow- ing Wettig et al. (2024) and for convenience, we sequen- tially apply the document-level and chunk-level refining pipelines by fine-tuning the 0.7B model itself. We then perform harge-scale program synthesis and execution using the refining models, resulting in Daoe and Daretchunk- Such 2-stage synthesis requires approximately 192 A100 GPU hours for processing 60B tokens of data. The downstream performance is presented Table 2. including base models trained on the data produced by PROX refinement methods
Comparing with Data Selection Methods Apart from comparing with heuristic methods, we also include existing representative model-based data selection methods tailored for pre- tre-training corpora to verify PROX's s effectiveness. In Table 3, we report both (-shot and 2-shot performance un-
5

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 3: Comparison with different data selection methods on 8 benchmarks using C4 corpus and PYTHIA architecture.
Table 4: Refining model's performance on valid set and token retention ratio of original corpus.
Total FLOPs
Size
Doc-level
Chunk-level
Kept Ratio
Method
O-shot
2-shot
#Win
(0.3
82.6
75.2
23.2%
(x1e19)
(0.7
81.3
75.6
25.6%
Model Architecture: PYTHIA-4104
M (1.7 B)
83.7
77.3
28.8%
Random
6.4
42.7
43.8
0/8
DSIR
6.4
42.5
43.7
1/8
Raw
ProX-(xs) ProX-(s) ProX-(m)
DsDm
10.7
43,4
44.1
0/8
Base Model Size
46
QuRaring
26.4
43.5
44.6
0/8
x
39.6
42.3
41.9
41.9
MATES
8.1
44.0
45.0
0/8
44
Performance
PROX (ours)
13.2
46.2
47.5
7/8
v
42.5
43.9
44.6
43.5
Model Architecture: PYTHIA-IB
-42
Random MATES
20.0
17.7
44.7
45.8
45.4
46.4
0/8
1/8
E
43.4
46.0
46.2
45.7
-40
PROX (ours)
21.9
46.8
48.4
7/8
Figure 5: PROX'S ' effect over different model sizes.
der the same settings used in MATES (Yu et al., 2024). While we merely apply document-level stage (i.e., PROX- D) which is indeed similar to data selection methods, we can see that PRoX outperforms the strongest data selection method MATES, by 2.2% and 2.5% in 0-shot and 2-shot average performance for 410M model, and by 1.0% and 2.0% for 1B model. Additionally, PROX achieves the best performance on 7 out of 8 benchmarks tested, demonstrat- ing its superiority over existing data selection methods. Full evaluation results are provided in Table 13 ($F.2).
favorable balance between data quality and quantity. These additional tokens likely provide more knowledge during pre-training without compromising downstream benchmark performance, showcasing an effective trade-off between data refinement and information preservation.
PROX works well across pre-training corpora. To as- sess the applicability of PROX across various pre-training corpora, we extend our experiments beyond RedPajama-Y to C4 and the recently released top-quality corpus includ- ing FineWeb, FineWeb-Edu, and DCLM. We apply exactly the same PROX-xs refining models detailed in Table 4 to these corpora without constructing new seed data. We con- ducted larger-scale experiments by training 1.7B models from scratch for about 50B tokens, again achieving no- table improvements: On ten downstream benchmarks, mod- els trained on PROX's curated data showed improvements of +2.0% on RedPajama- V2, +2.9% on C4. +2.4% on Febe + +0.9% on FineWeb-Edu and 1.77% on DCLM, as shown in Figure 6.
3.3. Applying PROX across model sizes and corpora
In this section, we demonstrate that PROX can effectively benefit models beyond scales and across different corpora, and greatly improves the training efficiency.
PROX works well across different scales. We train se- ries of models from 350M to L.7B (i.e., TLM-XS, TLM-S, and TLM-M) on the same 26B tokens used in $3.2, and then fine-tune these models on doc-level and chunk-level tasks, obtaining refining models with diffcrent sizes. We then apply these models in both doc-level and chunk-level refining stages and use the curated data for from-scratch pre- training. We report the adaptation performance refining tasks of different refining model sizes in Table 4. According to the validation performance, adaptation works well across all model sizes, all achieving > 80% Fl on document-level refinement, and > 75% F1 on chunk-level refinement. We further train models of different sizes from scratch using data produced by refining models of varying sizes. In Fig- ure the heatmap indicates that all refining models of three sizes improve data quality over data with consistent performance boost of 2% over all base model sizes. While TLM-XS XS curated data shows slightly better downstream per- formance, it has lower token-level retention ratio (23.2% vs. 28.8%) compared to larger models as reflected in Ta- ble This implies that moderately larger models suggest a
ProX brings greater training efficiency. To demonstrate the non-trivial nature of these results, we compared models trained on PROX curated data against various models trained by different approaches. These include models like TINYL- LAMA-1. (trained on 3T tokens, about 60x of our train- ing tokens and 40 x training FLOPs), SHEADLLAMA- 1.3B (pruned from LLAMA-2-7BB with extra 50B tokens training), and models using L.LM data synthesis, such as INSTRUC- TIONLM-1.33 and COsMo-1.8B. Our results, including TLM-M (PROX) and TLM-M (Raw), are presented along- side all these baselines in Figure 6. On FineWeb, which is recognized for its high-quality data, TLM-M using PROX- refined data performs comparably to pruned models like SHEADLLAMA- 1.33 and TINYLLAMA-I.BD despite their reliance on additional pruning techniques or much larger datasets. Moreover, using much less computing overhead for data refinement, our models even outperform models that
6

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 5: Math domain continual pre-training (CPT) results with few-shot CoT evaluation performance averaged over 9
Baw Deta
L.M Syuthess Muski Pruning
52-
51.0
mathematical reasoning benchmarks.
49.8
50.3
49.7
Average Porformmen (%)
50-
Clyyaomaa/ 78-17 50 1%
Unig
Train
Math
48A
Model
Size
CPT
Method
Toks
Toks
AVG. Perf.
48.0
48-
3134
LLEMMA
7B
S5B
200B
50.9 (+21.8)
46.0
45.5
INTERNLM2
7B
7B
<
X
31B
125B
48.7 (+12.6)
36,1
1.1B
X
14.7
L.B
15B
15B
21.5 (+6.8)
TINYLLAMA
1.1B
Rule
6.5B
15B
18.4 (43.7)
1.1B
PROX
5B
15B
25.7 (+11.0)
42 MedPajana
DULM
IsstLM St.lome
7B
X
31.5
LLAMA-2
7B
15B
10B
42.8 (+11.3)
Figure 6: Performance of original data and PROX curated data trained models across different datasets using 50B tokens and comparison with existing models trained using different techniques like LLM data synthesis and direct model pruning. Inst-LMM: INSTRUCTIONLM-1133 Cosmo: COSMO-L.8B; S-Llama: SHEAREDDLAMM-I13B
7B
PROX
5B
10B
46.1 (414.6)
7B
X
29.1
CODELLAMA
7B
15B
10B
43.2 (+14.1)
7B
PROX
5B
49.4 (+20.3)
7B
X
51.6
MISTRAL
7B
15B
10B
54.8 (+3.2)
7B
PROX
4.7B
10B
59.2 (+7.6)
dation compared to continual pre-training on the original corpus. This suggests that universal heuristics are inef- fective across all domains, highlighting the need for au- tomated pipelines like PROX. Moreover, compared with existing math continual pre-training models like LLFEMAA and INTERNLM2-MATH typically requiring hundreds of billions of training tokens, PROX demonstrates remarkable efficiency gains. A more controlled comparison further high- lights this: LLEMMA-7B, based on CODELLAMA-7B. was trained on 200B tokens; whereas PROX, also starting from CODELLAMA-7B reaches similar performance (50.9% vs. 49.4% 49.4%) with just 10B tokens of training, indicating 20x reduction in training computes.
rely heavily on data synthesis with LLMs, underscoring the PROX's efficiency. Notably, models like INSTRUCT-LM- 1.3B, trained on 100 billion tokens from a fine-tuned MIS- TRAL-7B synthesizer, and CosMO-1 1.8, trained on 180B tokens (including 25B tokens synthesized by MIXTRAL- 8x7B), require significantly more compute than PROX. In fact, their computational cost of data synthesis has far sur- passed the training overhead.
3.4. Applying PROX to Specific Domains
We also demonstrate the potential of PROX in the contin- ual pre-training scenario, specifically, in the mathematical domain. We apply the very same pipeline as in general domains to the Open WebMath corpus (Paster et al., 2024), aiming to further mine and refine the high-quality and clean data from the crawled vast web pages. We apply PROX-xs series for refining as described in 3.3 and adapt them math seed data for the document-level and chunk-level refin- ing tasks. Finally. about 5.5B tokens remain after document- level refining, and about 4.7B after chunk-level refining. We present the final mathematical evaluation results of models trained on Math in Table 5, with comprehensive evaluation results and full ablation studies presented in $E.4.
4. Analysis
Impact on the Original Data We compare the docu- ment's token length distribution of the original corpus with that of the PRoX-refined corpus in Figure 7. In the general domain, the refined data exhibits a noticeable shift in the average number of tokens per document. Before refinement, documents with fewer than 100 tokens make up signif- icant portion of the corpus; after applying the PROX, the majority of them contain more than 200 tokens, with an average number of tokens per document increasing from 1217 to over 2000. This shift, however, is not observed in OpenWebMate (Middle part in Figure 7). One possible reason for this outlier is that the OpenWebMate corpus is collected mostly from sources different from the general domain, e.g., online forums like Stack Exchange, and aca- demic publisher websites such as Arxiv. And noises of these sources can be quite different from general domains. Further analysis and case studies on these documents are provided in F. $F.2, and F..3.
PROX boosts continual pre-training efficiency vastly. Without any domain-specific design, Table 5 shows that pre- training on Open WebMath refined by PROX brings 11.0% average performance improvements for TINYLLAMA- 1. IB, 14.6% for LLAMA-2, 20.3% for CODELLAMA, 7.6% for MISTRAL, which clearly exceeds the improvements of all baselines, including their counterparts pre-trained on the original corpus. Notably, applying rule- rule-based filtering does not improve performance; instead, it causes a 3. 1% degra-
7

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
2.26
Original Data
Original Data
Thrin FLOP% (X1e20)
AW
Train FLOPs
ProX Refined Data
ProX Refined Data
20
ZZ
Infer FL.OPs
Densiry
ToksT
Toks: 1815.8
is
135
Avk.Toks: 1217.5 Awg..Tok::20000
Tok:
Toks: 1734,9
1.0
0.73
0.5
10
10'
10
10
(ftakks
10
10"
(fooks
0.0
0.38
1.7B
Figure 7: Left and Middle: Comparison of doc's token length distributions between original and PROX-refined data from RedPajama- V2 and Open WebMath, respectively. Right: Total FLOPs comparison for achieving comparable downstream performance with/without PROX refining: 0.3B (Avg. Perf = 40.5), 0.7B (41.6), and 1.7B (42.9).
Table 6:
Analysis of programs generated by PROX: error
(Wenzek et al., 2020; Smith et al., 2022). The remaining documents will then undergo several quality checks with heuristic rules like overall length, symbol-to-word ratio to determine whether they are kept, or aborted (Zhang et al., 2024a; Dou et al., 2024; Qiu et al., 2024). Finally, these documents are deduplicated with algorithms such as Min- Hash (Broder, 1997). In PROX, we use small language models for further data refining, outperforming heuristic rules with acceptable computational overhead.
program ratio and program complexity (functioo calls).
Error Ratio
Avg. Func Calls
Domain
Doc-level
Chunk-level
Chunk-level
General
0.04%
0.36%
3.7
Math
0.06%
0.11%
2.7
PROX Generated Program Analysis PROX maintains extremely low failure rates (< 0.4%) across both refining stages and domains while demonstrating computational ef- ficiency, requiring only 3.7 and 2.7 average function calls for general and math domains respectively. This indicates PROX is well-suited for small models, achieving high relia- bility under resource constraints.
Data Selection Methods Data selection is more com- monly applied in the later stages of large-scale data pre- processing. In supervised fine-tuning. it typically involves selecting much smaller subset of samples while maintain- ing performance (Liu et al., 2024b). Recent efforts have extended these selection strategies to pre-training (Engstrom et al., 2024; Xie et al., 2023; Ankner et al., 2024; Sachdeva et al., 2024). Wettig et al. (2024) train a rater model to score documents on four quality criteria; MATES (Yu et al., 2024) apply LMs to estimate data influence and enables dynamic selection schema. Moreover, as mentioned in LLAMA-3 (Meta, 2024), LLAMA-2 (Touvron et al., 2023b) are used as text-quality classifiers that underpin LLAMA- 3's 's training data. In PROX, we provide more fineggriined operations within documents for further improvements.
Computing Overhead Analysis Although PROX demon- strates very promising results in downstream tasks, such large-scale inference will require an extra computing budget. Gladly, the relative computational cost for PROX will keep decreasing when developing larger models. In Figure 7, we calculate the FLOOPs consumed by checkpoints with similar downstream performance, both with and without PROX. As model size increases, the proportion of inference FL.OPs for applying PROX decreases. Surprisingly, for the largest 7. model, we achieve performance comparable trained on the original data, but with only 58% of baseline's 's FL.OPs. This demonstrates that refining methods like PROX not only enhance data quality but also provide efficiency for developing LLMs, reinforcing the value of allocating additional resources to refining pre-training data.
model pre-
6. Conclusion
We introduced PROX, a framework that uses small language models to refine pre-training data through program genera- tion and execution. Extensive experiments show that train- ing on PROX curated data greatly improves performance on various downstream benchmarks, and holds effective across different model sizes and datasets. For the math domain, models trained on PROX curated data also yield significant improvements with 20x less training tokens. Further analy- sis also shows applying PROX can achieve similar results with less compute for large-scale LLM pre-training These results demonstrate PROX's 's potential to enhance data qual- ity while reducing costs in training LLMs. We believe that PROX paves the way for developing more efficient LLMs, and scaling computing for data refinement may further ac- celerate progress in future exploration.
5. Related Works
Pre-training Data Processing It has been a common prac- tice to execute rrepprcesssing before training due to the noisy nature of raw data from the Internet, which can hurt model performance (Touvron et al., 2023a; Together, 2023; Penedo et al., 2024a). The pipeline usually starts with docu- ment preparation, such as URL filtering and text extraction
'The train FLOPss for the hase model ( 5.3 x 104 used create the refining model are excluded. This is because any pre-trained LLM can theoretically serve as the hase for refinement.
8

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Impact Statement
In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Anthropic, A.
The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. URL https: /ttp:://wwwwwwr odn anthrapic.c.ccomm. de8ba9b01che http:///.w...aa.acaoooooo/oooaa. Model_Carde Salaadd__aaud________ .df..
Acknowledgement
We extend our profound gratitude to Shanghai AI Lab and Sea Al Lab for generously providing valuable computational resources, which were instrumental in the realization of this project. Our sincere thanks also go to Mingxuan Wang and Jiaze Chen from ByteDance for their crucial support. We are deeply thankful to Ethan Chern from Shanghai Jiao Tong University and Yuqing Yang from University of Southern California for their early discussions and insightful contribu- tions, and equally grateful to Zhoujun Cheng from UC San Diego Yiheng Xu and Tianbao Xie from University of Hong Kong, and Terry Yue Zhuo from Monash University for their valuable feedback, Guilherme Penedo and Loubna Ben Allal from Hugging Face for their guidance hyper- parameter tuning, to Zhibin Gou from Tsinghua University for providing advise on continual pre-training, to Lyuman- shan Ye for helping with illustrations and color scheme de- sign. Finally, special thanks go to Peiyuan Zhang from UC San Diego, representing the TinyLlama team, for providing great open pre-training framework and supporting series of acceleration operators. These collective wisdom and unwa- vering support have been pivotal to our project. This project is supported by SJTU SEIEE ByteDance Large Language Model Joint Laboratory, Shanghai Artificial Intelligence Laboratory.
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S. M., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. In The Twelfth International Conference Learning Representations, 2024. URL https:// openreview. net forum? i 1d=4hggR9155..
Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. Cosmopedia, Febru- ary 2024. URL https:/ htttps://hugeee httgh/hggggggccceeec co/ settastts//uggiingaaec.c coopoooga.....cooooooooo........
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. PMLR, 2023.
Biderman, S., Schoelkopf, H., Sutawika, L.,, Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J. Z., Foster, C., Jaiswal, M., Lee, W. Y., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang, J.. Skowron, A., Tan, S., Tang. X., Wang, K. A., Winata, G. I., Y Yvon, F,, and Zou, A. Lessons from the trenches reproducible evaluation of language models, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint 2:XIv:2303.08874,,0000 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-74327 0229-743,,00 2020.
AI, L.
Litgpt.
attps://gitee com/ 2023.
Broder, A. Z. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21-29. IEEE, 1997.
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, 2019.
Cheng, D., Gu, Y., Huang, S., Bi, J., Huang, M., and Wei, F. Instruction pre-training: Language models are supervised multitask learners. arXiv preprini arXiv:2406.. 14491, 2024.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter
Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-bant based pruning with small reference models.

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
of the Association for Computational Linguisties: Hu- man Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, 2019.
Welbi, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint
Hu, S., Tu, Y., Han, X., He, C.,, Cui, G., Long, X., Zheng, Z.., Fang, Y., Huang, Y., Zhao, W., et al. Miniepm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint arXiv:2404.0633 06395, 2024.
2018.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint wtt:::::110014111 2021.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, al. Mistral 7b, arXiv preprint 2023.
Dao, Flaetiitt:: Faster attention with better paral- lelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. 2020.
arXiv preprint arXiv:2001.088.00
Dou, L., Liu, Q. Zeng, G., Guo, J., Zhou, J., Lu, W., and Lin, M. Sailor: Open language models for south-east asia. CoRR, abs/2404. 03608, 2024. doi: 10.48550/ARXTV. 2404.03608. URL https::///ee org/ 005///10.8855 48550/ arXiv. 2404. 03608.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint trXiv:2407.21788 2024.
Sencel-Kedziorski R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. Mawps: A math word problem reposi- tory. In Proceedings of the 2016 conference of the north american chapter of the association for computational lin- guistics: human language technologies, pp. 1152-11577 2016.
Engstrom, L., Feldmann, A., and Madry, A. Dsdm: Model- aware dataset selection with datamodels. arXiv preprint arXiv:2401.12906 2024.
Kwon, W., Li, Z., Zhuang. S., Sheng, Y.. Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, I, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023,
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scall scale knowledge. Advances in Neural Informa- fion Processing Systems, 35: 11833-183622 2022.
Li, J., Fang, A., Smymis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E.,, Keh, S., Arora, K., et al. Datacomp- Im: In search of the next generation of training sets for lan- guage models. arXiv preprint arXiv:2406. 11744 2024.
Fourrier, C., Habib, N., Wolf, T. and Tunstall, L. Lighteval: A lightweight framework for 11m evaluation, 2023. URL https://githin com/ huggingface lighteval.
Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05466 2023,
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306. /644, 2023.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., et al. Rho- Not all tokens are what you need. arXiv preprint arXiv:2404.0790 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, Measuring mathematical problem solving with the math dataset. In Tiitty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021.
Liu, J., Cui, I.., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124 2020.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E.,
Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., al. Best practices and
Cai, T., Rutherford, E.,, Casas, D. d. I.., Hendricks, L. A.,
10

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503, 2024a.
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Liu, W. Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? comprehensive study of auto- matic data selection in instruction tuning, In The Twelfth International Conference on Learning Representations, 2024b. URL https: http//ppereeieeeeeeeeeeeeee wwweerrrview.ne//frrreeeeeeeeeee 1d=BTKAeLQIM..
Paster, K. Santos, M. D., Azerbayev, Z., and Ba, Open- webmath: An open dataset of high-quality mathemat- ical web text. In The Twelfth International Confer- ence Learning Representations, 2024. URL https: /oopnnevvew. net forum?iddjKHmm
Lu, P., Qiu, L.. Chang, K.-W., Wu, Y. N.. Zhu, S.-C., Ra- jpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured mathemat- ical reasoning. In International Conference on Learning Representations (ICLR), 2023.
Patel, A., Bhattamishra, S., and Goyal, N. Are nip models really able solve simple math word problems? In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, 2021.
Luukkonen, R., Komulainen, V., Luoma, J., Eskelinen, A., Kanerva, J., Kupari, H.-M., Ginter, F., Laippala, V., Muen- nighoff, N., Piktus, A., et al. Fingpt: Large generative models for a small language. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 27 10-2726, 2023.
Penedo, G., Kydlfček, H., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L.,, Wolf, T., al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 17557, 2024a.
Maini, P., Seto, S., Bai, H., Grangier, D., Zhang, Y., and Jaitly, N. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401. 163800 2024.
Penedo, G., Kydlfčck, H., Cappelli, A., Sasko, M., and Wolf, T. Datatrove: large scale data pro- cessing, 2024b. URL https: attps://github.co con/ buuginggface/dateoo
I6380,
Mehta, S., Sekhavat, M. H., Cao, Q.. Horton, M., Jin, Y., Sun, C., Mirzadch, I., Najibi, M., Belenko, D., Zatloukal, P., et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.1466 2024.
Qiu, J., Lv, H., Jin, Z., Wang, R., Ning, W., Yu, J., Zhang, C., Chu, P., Qu, Y., Peng. R., et al. Wanjuan-cc: A safe and high-quality open-sourced english webtext dataset. arXiv preprint arXh::2402.19282 2024.
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2 12. /446, 2021.
Meta. Introducing meta 1lama 3: The most capable openly available I1m to date, 2024. URL https: // ai meta. com/blog hllgg/mes /meta- 1lama-3 3.
Miao, S.-Y., Liang, C.-C., and Su, K.-Y. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, 2020.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1667, 2020.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier. J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2381-2399 Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/1/DD8-12220 1260. URL https: wwweelanhholooy..rogo/ooo
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap. T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055: 2024.
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J.. Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Canton-Ferrer, C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usumier, N., Scialom, T., and Synnaeve, G. Code llaa: Open foundation models
Muennighoff. N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 0155503885377777 2023.

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
for code. CoRR, abs/2308. 12950, 2023. doi: 10.48550/ ARXIV.2308. 12950. URL https::/ttsdet.. http:://ooooooo//// 18550/arXiv.. 2308. 12950..
Volume I (Long and Short Papers), pp. 4149-4158, Min-
neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: 10.18653//1/N119-142 URL
attps://aclanthol..0g htt.:///.....ooooooooorrrrrra//.
Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L.. Chi, E. H., Caverlee, J., McAuley, J., and Cheng, D. Z. How to train data-efficient llms. arXiv preprint FrXiv:2402.09666 2024.
Team, I. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Together. Redpajama: an open dataset for training large lan- guage models, October 2023. URL pttttp::/githhi com/togethe wwwphherroooutttee Ree/PaedPaaaaa jaaa-aaaaaaaaaaaaaaa
Sakaguhhi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. /397/, 2023a.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interac- tions. Xiviv preprint /904,00990 2019.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhan- dari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11999 2022.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- of-thouugt prompting elicits reasoning large language models. Advances in neural information processing systems. 35:24824-24837. 2022.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkin son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muen- nighoff, N., Naik, A., Nam, C., Peters, M., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, E., Zettlemoyer, L., Smith, N., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K. Dolma: an open corpus of three trillion to- kens for language model pretraining research. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15725-15788, Bangkok, Thailand, August 2024. Asso- ciation for Computational Linguistics. URL ht https: //aclanthology, org/ 22/2 acl- long. (al.nn.8400 840.
Welbl, J., Liu, N. F., and Gardner, M. Crowdsoure- ing multiple choice science questions. arXiv preprint MxXiv:177.006 2017.
Wenzek, G., Lachaux, M., Conneau, A., Chaudbary, V. Guzmán, F., Joulin, A., and Grave, E. Ccnet: Ex- tracting high quality monolingual datasets from web crawl data. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., and Piperidis, S. (eds.), Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pp. 4003 4012. European Language Resources Association, 2020. URL. https: / aaacllantthh ogy org/ 2020.
1rec 1. 494/.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y.
Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Wettig, A., Gupta, A., Malik, S., and Chen, D. QuRating: Selecting high-quality data for training language mod- els. International Conference on Machine Learning (ICML), 2024.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseQA: A question answering challenge targeting commonsense knowledge. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies,
Xia, M., Gao, T., Zeng Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Xie, S. M., Santurkar, S., Ma, T., and Liang, S. Data selection for language models via importance resampling- Advances in Neural Information Processing Systems, 36; 34201-3422) 2023.
Ying, H., Zhang, S., Li, L.., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J., Liu, K., Wang, Z., al. InternIm- math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.000 06032,06632222 2024.
Yu, Z., Das, S., and Xiong, C. Mates: Model-aware data selection for efficient pretraining with data influence mod- cls. arXiv preprint arXiv:24000 06046, 2024.
Yuan, A., Coenen, A., Reif, E., and Ippolito, D. Wordcraft: story writing with large language models. In 27th Inter- national Conference on Intelligent User Interfaces, pp. 841-8522 2022.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, Self-rewarding language models, arXiv preprint 400v:2401.11020, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 19Xiv:1905.077 07030, 2019.
Zhang. G., Qu, S., Liu, J., Zhang, C., Lin, C., Yu, L., Pan, D., Cheng, E., Liu, J., Lin, Q. et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405. 19327, 2024a.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385 2024b.
Zhao, Y. Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmai-
son, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Ex- periences on scaling fully sharded data parallel. Proc. VLDB Endow,, 16413333888-386, aug 2023. ISSN 2150- 8097. doi: 10. 10..14778/3111540.366 1569. //doi. org 10. 11w.1/1477711111 3611569.
1569. URL https:
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., and Luo, Z. Lla- mafactory: Unified efficient fine-tuning 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, N. B., Zhan, H., He, J., Paul, I, et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint 255.:22406.1877,,,. 2024.
13

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
A. PROX Implementation Details
A.I. Supervised Fine-tuning Data Collection
In this section, we elaborate on the detailed prompts used to generate the SFT data for model adaptation. In principle, We apply the same prompts for general domain corpora (including C4 (Raffel et al., 2020), RedPajama- (Together, 2023), Web (Penedo et al., 2024a)) and mathematical corpus (OpenWebMath (Paster et al., 2024)). All seed data is randomly sampled from the raw corpora.
Document-level Programming We apply two zero-shot scoring prompts to evaluate and assign a combined score to each web document before synthesizing the (doc, program) pair. One of the prompts is the same as the one used in FineWeb-Edu which is prompt let the model decide the educational score. Additionally in PROX, we add new format scoring prompt, focusing on the format and structure of the document. Both prompts follow the additive style proposed by Yuan et al. (2024). Given these prompts, the language models generate short critiques and assign score between 0 and 5.
In Fine Web,Edu, documents are retained only the educational score (Edu Score) is greater than 2. However, this approach too aggressive when attempting to preserve a larger portion of the tokens. For instance, Fine retains only 1.3 trillion tokens out of the original 15 trillion in the Win corpus. To recall more documents, we relax the filtering criteria by incorporating the format score follows:
Edu Score 2 3,
keep document;
Filtering Criteria
Edu Score = 2 and Format Score 2 4, keep document; drop document.
(2)
Edu Score < 2,
Finally, we use LLAMA-2 1---7-B-IISTRUE to annotate 51K data, splitting 5K for validation. 2
The FineWeb- prompt and our format scoring prompts are presented in Figure 8.
Chunk-level Programming We apply chunk-level programming for more fine- tne-graiied operations. We find three very popular patterns that keep occurring in all corpus: (1) menu, navigation bars at the top of the document; (2) button, html elements, links; (3) footers.
In general, LLMs work well given within few-shot examples. But to generate these program snippets more accurately, we apply few-shot prompting with LLAMA-3-70B- INSTRUCT for each type of noise. We merge these programs aiming to clean different types of noises, perform some grammar checking, and make them the final data for training and validation during the chunk-level refining stage. The annotated source comes from the same seed document used in the previous document filtering stage, accumulating to about 57K data, of which 5K is split as validation.
After the release of LLAMA-3 33. INSTRUCT, We also try to use only one prompt aiming to remove all the noises. However, we find such practices lead to aggressive removal of the original document, often making the document less coherent. Finally, we decide to only keep the head part and tail part of the program generated by LLAMA-3. 1 --00BB- INSTRUCT, which is previously mentioned in FinGPT (Luukkonen al., 2023), and merge with the previous programs generated by LLAMA-3- 3-B0--7000-0 --0---INTTUCC... INSTRUCT.
The few-shot prompts used to generate program snippets are presented in Figure 9, Figure 10 and Figure 11.
Comparison with FineWeb-Edu's Approach Compared with the recently released FineWeb-Edu, which also uses model-based scoring by applying a BERT model evaluate documents, we find that our relaxed design retains more tokens without compromising overall data quality. Specifically, FineWeb-Edh retains about 1.3 trillion tokens out of a 15 trillion token corpus (less than 9%), while PROX curation typically keeps 23% to 28%, providing up to 3x more unique tokens for training.
In the earlier stage of experiments, we found that dataset of thousands of data points (i.e., 5K) is also sufficient to equip the model with the "programming" abilities. This generally holds true for both document-level and chunk-level programming tasks. Scaling the dataset size could enhance the model's robustness across various documents so we finally enlarge the pool to over 50K.
14

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Edu Scoring Prompts (Penedo et al., 2024a)
Below is extract from web Evaliate whether the has high educational valuae and could be useful educationa setting teaching from primary school grade school levels using adfirive spoint scoring syseem deseribed below. Points accumolated haved sntisfaction of each criterion:
Add point the extract provides some basic information relevant to educational topies. even if it includes some irreevant or non-scadmmi content like advertieemnnss and pro- addresses cersin elements pertinent education but does align closely with educational standards. might mix educationnl content with point
motional material, Add unother point the
disorgamized munner and insoberent writing style. extract is approprizne for olucational ue and introdaces key concepts relevent to school curicula. is coherent though may not be comprehemiise or could include some extrancous information. may resemble an incrodactory section of tesbooo or basic tutocial that is suitable for learning but has notable limitations like treating concepts that are too complex for grade school students. for level higher than grade school, exhibiing clear and comsistent wriing ssyle. Il could be similar solutions, minimal inelevant information, the concepts aren't advanced grade
non edureddccaiiolll material, offering a superficial overview potentially
loptcs, or preseming information
Grunt fourhh puint the extract highly relevant and benelicial for educationl chupter from textbook
usceiill offerin substaniial oducational content incholing
school students The content is coherent, focused, and valuable for structured loarning.
Bestow a fifth poin, if the exinect is vuistanding in iis educational vulue, perfectly suited for teaching either primury school grade school. It follows detaile reasoning, the writing style is easy follow and offers profound thorough insighis the subject matier, devoid any educational complex content.
The
|EXAMPLEE.
After cxamining the extract:
Briefly juctify
100 wards.
Conclade with the score using the format: "Educational score: (total points;"
Format Scoring Prompts
Evaluate the provided web extraction sample. Points accumolated based surisfaction each cratetio::
6. Start with points.
1. Add I point if the extract contains some rendable consent, even if includes a significant amoune of HT ML tags, navigation clements, other web page artifacts. The main content should be idenrifiblee, atheit mixed with
exnrct shows signs bosic cleaning. Most obvious HTML tags have been ressoved, though soms may remainn The text strueture hegins to emerge. but non-cantest elements (e.g. focter links, button text may still be present. The writing syle may be disjointed due to remmants of page structure.
another point
Award third point if the extract lorplyy cleinel HTMI and most non-coetent elements, The main hody the cratent intser and coherent, Some extmieous infoemation isolated URLa,, timestamps. image
may persist, doesn't significnally impede rendability.
resembles rough draft the original
Grant a fourth point if extract highly refined, with cleae paragraph stnicture and farmutting Almost all HTML, tigs and non-content elements have been eliminsted. Minimal moise renains. The contest flows well and resds like neaé-linal drift, with comvistet formatting and style.
5. Bestow fitth point the extraction is flawless. The content is enturely clean, presevving the original structure (paragraphs, hendiags, lists) without any HTML tags web elements. No information present. The reads
profesionally edited document, perfectly capturing original
The extract:
JEXAMPLE.-
After eximining the extract:
Briefly justify soore, up words.
Conclude with the score using the format: "Extractinn Quality Soome: jiotal points,"
Figure 8: Edu scoring prompts used in Web (Penedo et al., 2024a) and newly proposed "format scoring" prompts for PROX.
Moreover, we conducted preliminary study by training 0.7 billion parameter models on these data. We found that models trained on our curated data achieved similar downstream performance, as shown in Table 7. Therefore, we believe our current strategy is more suitable for large-scale pre-training, as it is capable of retaining more tokens while maintaining very high data quality.
Table 7: Comparing Fine Web-Edu with our strategy on TLM-S.
Methods
Kept Ratio
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
#Win
FineWeh- Edu FineWeh- PROX
8.6%
30.3
58.7
29.0
42.0
30.4
31.8
67.7
50.4
73.3
45.2
5/10
28.0%
27.7
55.7
30.4
44.2
29.5
31.0
68.8
39.3
52.2
72.8
45.2
5/10
15

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Navigation Removal Prompts
You'r te teked with pentrating Python prngamm to clein wich test strings by ramoving eovigstion bors. The with lest will he presented with line numbers starting from [000]", Your takk use the following pre-detined functions to clesa the text.
"python
deř unbouch di
vanlaave
uneaurched,, bagging high quuil dey
def renove. 1 ieeeeesttrr Istart: int, end: int)
ranranove Vines From "etarr"" ontil "end", inpluding "wad", aand".nt
Your identiff navigation buns items the beginning the lest and them using 15nee Function, the doeen'' cuntiin navipation bar are some examptes guide you
the
indicate clesning necessory. line contoins navigation, escape overkilling.
Example l:
[6oc]
[030] Beno I Producte About Us Contact
1001] Nelooms
website
[002] Bare's main cencent...
[/doc]
Progran:
''Eython
1inov (start"O,
Esample
[60c]
341 479 Bottnan v. Unicad States
341 us 079 Hoffnan v. United States 341 U.S. 479 2.5d.
BOZEKANV, UNIDED STATES.
8z. William A. Gray, Philadeiphio, Pe.r for petiticner.
Mz. John F. Davis, Mashington, D.C., for respendent.
[/doc]
Progean:
""python
untouch_doe
Example
[000|Police Seasot Tuabridge Wells House
Remaine Tip
Updated: 16/06/2012 10:44 reddit stunble [03i]Posted: 16/04/2C12 10:44
[032]Crine Body Buried In House, Buried Body, Buried Remains, Tip-Off, UK Nows, UK 2olice,
[003]Detectione seaschingg the gasdena house following intosnation
may
there.
[/doc]
Progran:
"'Eythonn
untouch_doe()
Example
[60c]
[030]Boow > Bollywood Neva Bollywood Staza clash on Indian TV Bollywood Stars clash en Indian tV
[031]ay Lekha Hadtavan05::7 Betting big on the festive seasen, ceneral entertainnent charnels (GECs)
lsunching csicbrity-driven so happen.
media
audicaes
that
[002]Tha fourth season of Bieg Boss on Colors is sinost cortaia to clash with the fourth season of Kaun Banega Crorepati (XBC) Sony Entertairment televinicn (SEI) in the second week ot October.
proparty, Haster Chef, on zir Cctobar. Bewever, the chaaral yet disclose the launch date. :034]Big budget ahows Like these ase often losa naking propesiLions for channels, as the opecatinng cost is vezy high and adver:lsenent revenues do not suffice to oover the cost.
:00s]Ancthor
Akshay
expected
[0ores IBNS
[/due]
Progran:
"""python
untoudl_dbe
For given analyze
these's navigation har Ton, use 'untouch, doc 1)* to indicate skaningg needed.
beginning.
"normell
Example: JEXAMPLE).
After esamining the web text: Briefly describe if the web euract contains saviguton bar begining (10 lines). decide
puge navigasion When the whole line savigation har, call "renove liness the line continns other information, "nornall ze" remove port of it.
your program using
"gython Iyouz codc]"
Figure 9: Few-shot navigation bar removal prompts.
16

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
URL Removal Prompts
You'r re lasked with generating Pythom programs io clean web text strings by removing hilp lines, The web will be presented with numbers staring from " 1000, Your task use the following pre defined functions clean the text:
"python
def unteuch_ (1
renioava
dec uncoushed, tagging
quality doc. n.x
def Hancs int, end: int):
renzunove noiay lines Izon "scaxe" untzi "ond", including "sad".nex
nosnal iun MMLLE whay Larget_ates atc=*")e
eencuen nolay strings into normallzed acringa,
Your goal is to idemiity hup links from the test remove them ming the "remove (inopp) or "aocnaliza 1)" function, text doesa' cooniin hipp lines, use the "untouch_doz1)) function indicase that cleaning secessary.
examples guide
Example
[000]
0131 http:///gropss ggogg google.com le..oe///oo.. con.//oooouuiiiooooooooooooooo.e group/ boowoombs nuxtast
[014] Breeking ing Neves Major Rivent Unlddd
(155] http://code. wwt.g/...o/g///aeeeoooo
//doc
Program:
'>python
the whole [013)
the 1ine-1013:
aanooe_iinsssstarrrr cnd-13)
the whole line-1 (015] http, 5o remove the line- 1015,
remove Fines astart=l end=1s)
Example
660c]
[000]
Jane Climate Global Ecosystens
charge inntinues to be a preesing fssue...
[/doc]
Program:
** bython
ontowch_doo
Example 3:
[600]
022 Box-xoM
0022 pe http:::////.../ooee. www.eiggogogooggggg...e henogg//e.coggooop////ooeoe inuxlast edited by Puppyt Mon 06 don 2011, 00:23; edited
02311 would like to see comething like Jitsi
Plua some at hee incorporated into puppy discro.
[/doc]
Program:
""python
the http 1ink in line 22 and line 24 comes with other text, use cornslize to ONLY remove the lins
without touching "ttt:tttttt"t////rrereaa
necnalize (sourea_str-
cargst_sts-"nn)
nernalize (swuros_strr sttp"tttp:////////// //www fitei orrg/. target_ str=*"]
For each gives web test, anayyee the conteet and determine there's navigation har menu items at the beginning. presest, use "cenove 1 iree () "nornall Low ioe not, use untouch__ooc ()" indictte that cleaning needed.
remove them.
Example: EEXAMPLE.
etamining
remone together with htp.
Hriefly deseribe if the web exteet contains top links; and make sure remove them will not influence the main coutent.
Program only contain sequences of function callings and comments, other codes. number, your program using same "ytton Iyous code]""
line munher with
int number of given
minus
Figure 10: Few-shot URL removal prompts.
17

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Footer Removal Prompts
You're usskdd with generating Python peograms to clean web test strings by removing footer sections, references. The web text will be presented with line mumbers starting from: [C00]" Your task following pro-defined functions
"pyyton
untouth doutt vengaave
uncoushed, taggang clean high quality doc.
def senove Lines (staztz ink, end: int):
noney
"end", inptuding nes
def nornalian izn 20 alay Larget_ates atz=th): vancurn notay strings lnto normsl Fize stringe. nea
Your idearity fnoter sections from the text and remove them using the "seaavee Linea 1)" function, Footers and referenees typically the end of the text and may contain infommation copyright cleaning nocessary.
navigation links. doesn' contain any
function indicate
Here are some exumples to guide you.
Example I:
[6oc]
doncactrates signirisent sindings. :014] Research: Inacitute. ziches reserved.
[013]
[015] Centact: sesbessarrh-inttttiee
[016] Fcllow uz social media: #ReccarchInst
[/doc]
Program:
*>python
Remove the foszez section starting from. lina 14 (start cnd-16)
Example
.000] Impact
Caange Ecosystens
:031] By De. Jane Snith
[032] Climate charge continucs to be pressing issue...
Further
needed tully understand its implications.
[/doc]
Program:
""python
untouch_doc
...
Example
[020] you for seading our aessicster.
:021] Stay informed with our latest updates!
[022]
Insubscribe Frivacy Policy
Servize
:024] HovsletterCo, Main Bt, Anytown, USA
[/doc]
Program:
***
Semove the footes saction starting
dividar
scmove_ Lines (start 22, end-247
For esch given web test, molyze the contest and determine there a footer section reforence. If persent, use "renove, 14 miee (1" romove it If not, use ountouch, doc 1)" indiiiiate that no cleaning needed.
Example: [EXAMPLE,
etamining
Briefly describe if the web eurnet contains footer section referenees ensure that removing it will not influence the main cumtent, not, simply call "untouch__oo"""
The should only contain sequences of function callk and comments, other code. mumhers,
numbers
number given
GGive your program using
pythun lyous code]"".
Figure 11: Few-shot footer removal prompts.
18

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
A.2. Supervised Fine-tuning Details
Training Parameters We use llama-fcctoyy (Zheng al., 2024) as our main code base for the Adaptation Stage. We apply full parameter supervised fine-tuning our base models: train the whole seed dataset for 3 to 5 epochs, with batch size as 64, and cosine learning rate schedular (Ir from le-5 - le-6). Also, find that the base model converges quite fast these tasks, thus we do not apply further tuning over hypee-parametees, and keep the same training configurations for all the adaptation tasks.
A.3. Evaluation Metrics for PROX Refining Tasks
Document-lewe Refining Task The document filtering task is indeed equal to a binary classification problem, where documents are classified as either to be kept (1) or dropped (0). We evaluate the performance using the Fl score, calculated as follows:
Precision Recall
F1 2
(3)
Precision + Recall
where:
TP
TP
Precision = TP + FP'
Recall
(4)
TP + FN
The Fl score ranges from 0 to and we assume higher score indicates better classification performance.
Chunk-level Refining Task This task actually contains two parts: line removal and string normalization. However, we find it rather hard evaluate the normalization task, so we use the line removal accuracy to reflect the refining performance. We propose a line-wise Fl score metrie:
The Fl score is computed by comparing the predicted noisy lines with the labeled noisy lines. First, extract the noisy line indexes from both the prediction and the label. Then, we calculate the overlap between these two sets. The true positives (TP) are the number of lines in this overlap. False positives (FP) are the predicted noisy lines that are not in the labeled set, and false negatives (FN) are the labeled noisy lines that are not in the predicted set. The calculation is actually simple:
TP (True Positives)
Predicted Noisy Lines O Actual Noisy Lines
(5)
FP (False Positives)
Predicted Noisy Lines Actual Noisy Lines
(6)
FN (False Negatives)
Acttal Noisy Lines
Predicted Noisy Lines
(7)
Then we same calculation of Fl score mentioned before, i.e., Fl
2.TP TT
19

---

else Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
A.4. PROX Inference scale
Thanks to the Datatrove project (Penedo et al., 2024b), we able efficiintly split, and load the whole corpus each worker (which normally equals the number of GPUs since small models do not require tensor parallelism). We use the vllm (Kwon et al., 2023) to perform large-scale inference.
For thunk-wise programming, we will split the original document into several chunks, controlling the tokens of each chunk less than the context window. In practice, we normally replace the token count process with a word count process to save time and control the window size as 1, 500. The general algorithm implemented as below:
Algorithm Document Chunk Splitting Algorithm
Require: Document D, context window size W
Ensure: Set of chunks C
1: C' < 0, c c 0
2: for each line in D do
3:
if TokenCounnt (c 1) & W then
4:
c+c
Add line to current chunk
5: else
6:
c then
7:
C {c}
Save current chunk
8: end if
9: if TokenCount()) < W then
10:
D Start new chunk
11:
C (FagAAAsSSipp)))))
Flag long line
c+0
14:
end if
15:
end if
16: end for
17:
if c ( then
18:
C
Add the final chunk
end if
return C
20

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
B. Pre-training Details
B.1. Training Infrastructure
Thanks to LitGPT (AI, 2023), and TinyLlama (Zhang et al., 2024b), we are able to flexibly train all our base models. We inherit several fused kernels from the TinyL which is installed from the FlashAttention (Dao, 2024) including fused rotary positional embedding (RoPE) (Su et al., 2024), layer normalization, and cross-entropy loss to help saving memory. We mainly apply FSDP strategy (Zhao et al., 2023) to enable training larger scale models multiple nodes.
Code Base
B.2. Pre-training Corpora
Due computing constraints and for fair comparison purposes, we cannot exhaustively train over the whole corpora. Thus, apply random sampling for some of the pre-training corpora and make them pre-training data pools.
For RedPajama-Ye 22,
randomly download 70 file shards, obtaining a total data pool consisting about 500B tokens, evenly separate it into dumps, with each containing about 62.5B tokens; due computing constraints, use only dump for verifying effectiveness (Section 3.2) and use dumps for scaling the training to 50B tokens (Section 3.3);
For C4, we download the whole dataset, which contains about 198B tokens;
For Fine we download the official 350B sample;
For OpenWebMath, we download the whole dataset.
We report the corpora details applied in each experiment in Table 8.
Table 8: The detailed breakdown for pre-training corpora in all experiments.
Section
Experiments
Source
Data Description
Corpora Size (B)
Train Tokens (B)
Epoch
data size
62.5
0.42
after rule-based filtering
31.5
0.83
Section 3.2
Table 2, Figure
RedPajama-V2 V2
PROX-D
19.0
26.2
after PROX-D+C
16.0
random
Section 3.2
Table
C4
after PROX-D
41.5 (GPT-NeoX)
26.2
0.63
other baselines
raw data size
62.5
0.42
after PROX-D+C (using PROX-xs)
14.5
1.80
Scction 3.3
Figure 5
RedPajama V2
after PROX-D+C (using PROX-s)
16.0
26.2
1.64
after PROX-D+C (using PROX-m)
18.0
1.46
raw
198.0
0.53
C4
after PROX-D+C (using PROX-xs)
raw data size after PROX-D+C (using PROX-xs)
123.5
0.42
Scction
Figure 6
RedPajama-V2
29
52.4
1.81
raw data
79.0
0.66
FineWeb
after PROX-D4 -DC (using PROX-xs)
18.0
2.91
duta size after rule-based filtering
15.0
1.05
6.5
2.40
Section 3.4
Table 5, I.IB model
MeatWhhaath
15.7
after PROX-D after PROX-D+C
4.7
3.49
duta size after PROX-D D
15.0
0.70
Scction 3.4
Table 5, 7B model
OpenWehMath
5.5
10.5
1.91
after PROX-D+C
2.23
21

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
B.3. Model Configuration and Training Parameters
Table 9: The details of the pre-training experiments' model architecture.
Model
Hidden Size
Intermediate Size
Context Len
Heads
Layers
Vocab Size
Params (w/o embed)
Training From Scratch
TLM-XS
1,280
2,048
2,048
16
24
32,000
354,284,801 (313,324,8010)
TLM-S
1,536
4.864
2,048
24
24
32,000
758.982,144 (709,830,144
TLM-M
2,048
8,192
2,048
32
24
32,000
1,741,785,088 (1.676,249,088)
PYTHIA-410M
1,024
4,096
1,024
16
50,304
405.334,016 (353.822,720)
PYTHIA-IB
2,048
8,192
1,024
50,304
L011.781,632 (908,759.040)
Continual Pre-truining
TINYLLAMA-I.18
2,048
5,632
2,048
32
22
32,000
1, 1,100,048, 384 (1,034,512,,88 (14434)))884)
LLAMA-2-7B
4,096
11,008
4,096
32
32
32,000
6,738,415.61 415,616 (6,607,343,6)
CODELLAMA-78
4,096
11,008
4,096
32
32,016
6,738,546,66 688
MISTRAL-7B
4,096
14,336
4,096
32/8 (GQA)
32,000
7,241,732,096 (7,1 (7.160,660,09)
Table 10: Training hyper- paper-parameterss of all base models.
Context Length
Warmup Steps
Weight Decay
LR
Model
Batch Size
Max Steps
Optimizer
LR
Scheular
Training from Scratch
TLM-XS
1,024
2,048
12,500
500
0.1
Adam
cosine
5c-4 +
TLM-S
1,024
2,048
12,500
500
0.1
AdamW
cosine
5e-4 + 5c-6
TLM-M
1,024
2,048
12,500/2,5000
500
0.1
AdamW
cosine
3e-4 -> 3e-5
PYTHIA-410M
1,024
50,200
2,000
Adam WSD
1e-3 6.25e-5
PYTHIA-IB
512
1,024
50,200
2,000
0.1
Adam
WSD
le-3 + 6.25e-5
Continual Pre-training
TINYLLAMA-I.1B
2,048
1,024
7,500
Adam
cosine
8e-5
LLAMA-2-7B
4096
256
15,000
leady step (((000)
0.1
AdamW
cosine
8c-5 +
CODELLAMA-TB
4096
1024
3,750
(eatly 2,500
0
0.1
AdamW
cosine
3e-4 + 3e-5
MISTRAL-7B
4,096
256
15,000
loudy
0
0.1
AdamW
cosine
2e-5 -> 26-6
Base Model Selection
Our pre-training experiments conducted using various sizes of decoder-only language models.
To verify different stages' effectiveness of PROX, we employ a 750M sized model sharing LLAAM--2 architecture (Tou- vron et al., 2023b), denoted as TLM-S, used for both pre-training from scratch and refining. We also compare PROX with architecture (Biderman et al., 2023), as those employed in MATES (Yu
data selection methods using et al., 2024).
For further evaluation of PROX using different refining and base model sizes, we scale the model sizes from 350M (0.5x smaller, denoted as TLM-xS) and 1.7B (2x larger, denoted as TLM-M), all based on the LLAMA-2 architecture
3. For domain-speciff continual pre-train we select TINYLLAMA-LA 1. (Zhang et al., 2024b), LLAMA-2 (Touvron et al., 2023b), CODELLAAA (Rozière et al., 2023) and MISTRAI.-7B (Jiang et al., 2023) as representative base models for their adequate training and solid performance.
Model Architecture The models we used in general and continual pre-training are presented at Table 9 with detailed architecture configuration.
Training Hyperparameter Choice We primarily use a cosine learning rate scheduler and follow established settings used in Zhang et al. (2024b) and Lin et al. (2024). The default configurations for each experiment can found below and we elaborate on full details in Table 10.
22

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
1. For general pre-training experiments, the learning rate Se-4 for TLM-XS and TLM-s, 3e-4 for TLM-M; the maximum sequence lengths are uniformly set to 2048, and the global batch size is 2M tokens.
2. Additionally, we align all our comparison with their existing data selection methods, as previously shown in Table 3. In this case, we switch to the wurmup-stable-decu: (WSD) learning rate scheduler (Hu al., 2024), implemented in MATES. For a fair comparison with baselines implemented in MATES, apply the exact same WSD Schedular (Hu al., 2024):
with those used MATES (Yu et al., 2024) to facilitate a direct
if < W W < t <
w
(r())=
(8)
0.5 S)/D 7
f
where W equals to 2000, S equals to 50000, D equals 200.
For continual pre-training experiments, we set different hyperparameters different base models, as shown Table 10. We apply early-stop mechanism mentioned in INTERNLM2-MATH (Ying al., 2024) for 7B model experiments. We
mainly refer to these settings to the setup reported in Rho-1 (Lin et al., 2024) and LLEMMA (Azerbayev al., 2024). We
do not warmup in continual pre- pre-training experiments.
23

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
C. PROX Baseline Selection
To ensure a fair comparison w.r.t training cost, we keep most of the training hyperparameters, such as training steps and batch size, consistent across baselines, with only the data refining and selection pipelines differing. We compare PROX to series of baselines:
1. In $ 3.2, to verify PROX's effectiveness, we first compare with PROX with regular pre-training over the raw RedPajama-Ve V2 data. We also introduce heuristic baselines used to curate the FineWeb corpora, which is the com- bination of three filtering strategies from C4 (Raffel et al., 2020), Gopher (Rae et al., 2021), and newly crafted rules (as Web rules). We also reproduce the fastext classifier filtering which is reported as the strongest baseline in DCLM (Li et al., 2024) our own corpus. Apart from rule-based baselines, niques proposed in previous works, including (1) importance resampling: DSIR (Xie et al., 2023); (2) model-based selection: DsDM (Engstrom et al., 2024), MATES (Yu et al., 2024), and QuRating (Wettig et al., 2024).
also introduce existing data selection tech-
2. In 3.3, to test PROX on different model sizes and training corpora, we scale the TLM-M's training tokens to 50B over RedPajama- V2, C4, and Fine To show PROX efficiency, we then directly compare with models covering a variety of pre-training approaches including (1) large-scale pre-training: TINYLLIAMA-1 1-1.111 BB (Zhang al., 2024b) trained on 3T tokens; (2) model pruning from existing models: (SHEADLLAMA (Xia et al., 2024) pruned from LLAMA-2 and trained on extra 50B tokens); (3) LLM synthesis (INSTRUCTIONLN) LM-1.3 (Cheng et al., 2024) trained on MISTRAL-7B generated data and COSMO-1.C. 1.8B (Ben Allal et al., 2024) trained on MIXTRAL-887BB generated data).
3. 3.4's specific domain continual pre-training, apart from standard continual pre-training on TINYLLAMA-I, 1. 1B, LLAMA-2-7B, CODELLAMA-7B. and MISTRAL-7B, additionally introduce with well-known and strong baselines trained on public (or partially public) data, including Rho-1 (Lin et al., 2024), INTERNLM2-MATH (Ying et al., 2024), LLEMMA (Azerbayev et al., 2024), and an internal checkpoint reported in DEEPSEEK-M (Shao et al., 2024).
24

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
D. Downstream Tasks Evaluation
D.1. General Pre-training Evaluation
Lighteval Configurations We mainly borrow the evaluation benchmarks from FineWeb's nine selected "early signal" tasks (Penedo et al., 2024a), and use the implementation of lighteval (Fourrier et al., 2023) to test all our base models. We also introduce SciQ (Welbl et al., 2017) which is widely used in previous works and proved good testbed (Mehta et al., 2024; Wettig al., 2024). By default, we report the normalized zero-shot accuracy. All nine benchmarks are listed below:
ARC (Clark al., 2018): including ARC-Easy (ARC-E) and ARC-Challenge (ARC-C)
CommonSense QA (Talmor et al., 2019) (CSQA)
HellaSwag (Zellers et al., 2019)
MMLU (Hendrycks et al., 2021)
OpenBook QA (Mihaylov et al., 2018) (OBQA)
PIQA (Bisk al., 2020)
SocialIQA (Sap al., 2019) (SIQA)
WinoGrande (Sakaguchi et al., 2021) (WinoG)
SciQ (Welbl et al., 2017)
We use the same configuration used in FineWeb's, which randomly picks 1, 000 samples for each dataset (for MMLU, it selects 1, 000 samples for each of the 57 subsets), and reports the normalized accuracy. This average performance is calculated over the nine benchmarks, where ARC-C and ARC-E are considered as two separate benchmarks, and MMLU is treated as single benchmark. This approach differs slightly from the aggregation score calculation in FineWeb, as we believe MMLU's performance is relatively unstable, and we aim to give equal weight to all benchmarks, preventing MMLU from becoming a dominant factor. For the original lighteval scores, please refer to the $E.1, where we include dynamic result curve that clearly illustrates the fluctuations in each benchmark.
We choose present zero-shot evaluation mainly following settings used in all Fine Web's ablation experiments (Penedo et al., 2024a). We find the FineWeb evaluation maintains very stable performance curve when training tokens gradually accumulate. Also, it is very time-efficient for fast evaluation regarding our extensive pre-training experiments(20+ final runs, with hundreds of intermediate checkpoints). We also present few-shot evaluation results in Table 11. Also, we find that not all benchmarks show better performance given few-shot prompts. For example, we do not observe a very clear performance boost on HellaSwag, MMLU, PIQA, and WinoGrande, Similar observation can also be noticed in recent works (Mehta al., 2024; Muennighoff al., 2023), where 0-shot Hellaswag and O-shot WinoGrande show very close performances with 5-shot ones.
Based on these findings and considerations, we present zero-shot evaluation results in Table 2, Figure 4 and use it as our default evaluation metrics.
LM-Eval Harness Configurations We also include the Im-evel-harness (Biderman et al., 2024) for zero-shot and few-shot performance, for fair comparison with different data selection methods including DSIR (Xie et al., 2023), DsDm (Engstrom ct al., 2024), Qurating (Wettig et al., 2024) MATES (Yu et al., 2024). Similar to lighteval configuration, we include:
ARC: including ARC-E and ARC-C
HellaSwag
LogiQA (Liu et al., 2020)
25

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
OpenBook QA (OBQA)
PIQA
WinoGrande (WinoG)
SciQ
We exclude the BoolQ (Clark et al., 2019) tasks from MATES (Yu et al., 2024), leaving eight tasks in total. This decision was made because we observed that the BoolQ benchmark performance exhibited severe fluctuations and showed notable declining trend in the early stages. Therefore, we decided to exclude it from our evaluation set. Such a similar trend also observed earlier the OpenELM work (Mehta et al., 2024). We report both zero-shot and two-shot performance. the metries include normalized accuracy, we use that measure; otherwise, we use accuracy.
D.2. Continual Pre-training Evaluation
We evaluate all benchmarks implemented in the math-eval -hareesssssssssssese repository," 4 including:
Math (MATH) (Hendrycks et al., 2021)
GSM8K (Cobbe et al., 2021)
SVAMP (Patel et al., 2021)
ASDiv (Miao et al., 2020)
MAWPS (Koncel-Kedziorski al., 2016)
MathQA (MQA) (Amini al., 2019)
TableMWP (TAB) (Lu et al., 2023)
SAT MATH (Azerbayev al., 2024)
We use few-shot CoT prompting (Wei et al., 2022) when evaluating these tasks, and report the accuracy of each task.
E. Full Evaluation Results
E.1. Detailed Performance on 10 Benchmarks in Sec 3.2
We report full evaluation results of checkpoints saved at different training steps in Section 3.2. We present the results for 0.7B models trained data curated by different methods Table 12, including models trained on raw data, rule-based filtered data, fasttext-filtered data, and data curated by PROX.
http://wwwinnoomaatthooooooooeo htt uat on ha rness
thub. htt///ubiiaaa
26

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 11: Few-shot performance on 10 selected tasks. All models use the same TLM-S architecture and are trained on RedPajama-V V2. The doc-level (PRoX-D) and chunk-level (PROX-C) refining are done by fine-tuning the raw data pre-trained model refining model same as Table
Method
ARC-C ARC-E
CSQA
HellaS
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
Raw
25.5
50.3
33.2
39.9
27.8
29.2
67.8
38.7
52.4
71.5
43.6
Rule-based
26.2
50.9
34.1
41.8
27.8
29.2
66.8
40.5
52
72.8
44.2
PROX-D
29.1
55.7
35.6
41.8
29.4
29.2
66.8
38.3
51.3
77
45.4
PROX-D+C
27.2
59.9
38.3
42.8
29.7
31.4
67.1
40.3
50.2
75.8
46.3
27

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 12: Full evaluation results TLM-S.
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
Raw Data
2500
22.1
39.0
27.6
31.6
25.9
26.6
61.2
37.3
48.9
59.1
37.9
5000
24.4
41.2
28.8
34.8
26.7
27.0
64.9
39.3
50.4
61.9
39.9
7500
26.5
43.9
29.5
37.2
27.2
29.0
64.8
38.7
50.8
68.2
41.6
10000
25,8
43.5
29.1
38.8
27.4
29.8
66.9
39.0
51.2
66.2
41.8
12500
26.1
29.7
27.3
29.2
39.0
52.0
42.1
Gopher
2500
22.3
39.4
26.6
31.3
25.6
27.0
61.1
38.9
51.3
58.6
38.2
5000
25.1
41.4
29.8
34.3
26.4
27.2
64.5
39.6
52.1
62.9
40.3
7500
26.5
43.0
30.5
38,5
27.2
28.8
65.7
38.2
53.7
66.4
41.8
10000
26.2
31.8
39.2
27.5
29.4
38.9
51.3
42.3
12500
25.7
44.0
31.3
40.2
27.3
29.0
66.3
39.0
51.2
68.9
42.3
C4
2500
22.6
40.6
28.8
31.3
26.2
27.4
61.7
39.3
51.2
57.1
38.6
5000
22.9
41.6
29.3
36.0
26.8
27.6
64.7
40.2
50.9
63.6
40.4
7500
24.2
44.2
29.5
39.2
27.2
28.4
66.2
40.9
51.6
63.8
41.5
10000
24.6
44.8
30.4
39.5
27.0
29.4
68.7
40.9
51.7
63.9
42.1
12500
25.0
46.0
31.0
40.5
27.1
29.2
68.5
40.5
51.7
66.6
42.6
Fine
2500
23.2
39.4
27.2
31.8
25.6
26.2
62.6
39.0
51.4
57.1
38.3
5000
24.2
42.3
29.8
27.0
28.4
64.3
38.9
51.4
61.4
7500
24.4
44.1
30.4
37.8
27.2
28.2
66.1
39.5
50.8
66.2
41.5
10000
23.6
46.6
32.0
39.6
27.0
27.8
66.3
39.2
53.1
70.5
42.6
12500
25.2
46.8
32.6
39.6
27.2
29.0
66.5
39.4
52.4
69.2
42.8
Gopher
C4 Fine
2500
23.6
39.3
27.6
32,1
25.8
26.0
61.7
39.8
50.9
55,4
38.2
5000
23.9
40.9
29.0
36.2
26.9
26.8
65.3
39.3
52.7
62.4
40.3
7500
25.6
42.2
30.7
39.7
27.0
28.4
66.0
40.2
51.8
60.9
41.2
10000
25.8
43.3
30.8
41.4
27.5
29.8
66.9
39.5
51.8
63.1
42.0
12500
25.0
43.9
30.0
41.9
27.5
31.0
67.0
39.9
51.9
65.3
42.3
PROX-D
2500
25.6
43.2
27.7
32,9
27.2
27.0
61.3
39.4
50.6
63.0
39.8
5000
25.4
46.2
28.4
35.7
28.1
28.8
64.7
39.3
53.3
64.2
41.4
7500
26.9
29.1
39.2
28.6
30.8
38.8
51.2
43.1
10000
26.7
48.2
30.5
39.9
28.6
28.6
66.2
39.7
51.9
71.2
43.2
12500
26.6
49.7
30.1
40.5
29.4
30.4
66.3
39.0
51.2
71.6
43.5
PROX-D+4
2500
24.9
43.4
27.3
32,1
26.9
28.2
60.9
38.8
51.2
60.8
39.5
5000
24.9
49.6
28.8
36.8
27.9
30.6
64.7
38.8
51.1
66.9
42.0
7500
25.5
51.2
30.8
38.8
28.4
31.2
67.3
40.2
50.3
71.7
43.5
10000
26.2
51.7
30.8
39,9
29.0
32.6
39.7
51.7
73.7
44.4
12500
26.4
51.9
30.9
42,4
29.4
31.6
67.9
40.0
52.2
73.5
44.6
28

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
E.2. Detailed Performance on Benchmarks used in Data Selection Experiments
The full benchmark performance used in data-sclection method comparison experiments is presented in Table 13.
Table 13: Detailed evaluation results for different data selection methods.
Method
ARC-C
ARC-E
HellaSwag
LogiQA
OBQA
PIQA
WinoGrande
SciQ
AVG
PYTHIA - 41
O-shot
Random
25.6
40.2
39.7
24.7
29.4
67.1
50.6
64.1
42.7
DSIR
23.8
39.9
39.6
27.0
28.4
66.8
51.5
63.1
42.5
DsDm
24.7
41.7
40.3
27.5
29
68.1
50.1
65.4
43.4
QuRating
25.4
42.0
40.7
25.3
30.2
67.5
52.1
64.8
43.5
MATES
25.0
41.8
41.0
25.7
30.8
68.7
52.7
66.0
44.0
PROX
27.2
48.9
43.1
26.9
31.8
68.4
54.1
69.5
46.2
PYTHTA- CM 2-shot
Random
25.3
42.6
39.9
24.1
28.6
66.9
52.2
70.6
43.8
DSIR
23.6
42.0
39.8
26.1
28.6
66.1
51.6
71.4
43.7
DsDm
23.6
44.2
40.1
23.5
29.2
66.5
51.5
74
44.1
QuRating
23.6
43.9
40.4
26.1
30.2
67.4
51.4
74.1
44,6
MATES
25.3
43.8
40.6
24.9
30.6
67.1
53.4
74.1
45.0
PROX
27.0
52.7
42.6
23.7
32.8
68.2
53,9
78.9
47.5
PYTHIA- 1B 0-shot
Random MATES
25.6
43.7
43.8
27.5
31.8
68.9
50.7
65.8
44.7
25.9
44.9
45.3
28.7
32.2
69.5
52.4
67.3
45.8
PROX
26.2
49.1
46.6
24.8
32.2
70.3
54.2
70.9
46.8
PYTHIA 1B 2-shot
Random MATES
25.5
45.1
42.9
24.6
30.0
68.3
52.1
74.6
45,4
26.8
44.8
25.2
30.6
68.7
51.6
75.7
46.2
PROX
27.3
54.5
46.2
26.6
32.2
69.0
53.9
77.4
48.4
ARC-E
wag
MMLU
50
40-
Peffmmaae
45
30
Perfemmance S2
10
Pri. D-C
FruE DHC
Pric D-
PaO. DHC
Prik DHC Prot-D
Pak-
FredDD
Prak-D
Paak-
22
Bule
Mulk
25
30-
TokenotB)
Triniisa
Thiniux TokenstB
Tuinnin Tokers WinoC:
Training Tokera
OBQA
PIQA
SIQA
SciQ
32
Puttemers
30
reformntr
59
60
28.
80
Trux
ProX ProX
50
DiC
DIC.
26
PriX Rulo
55
Kak
Jule
Bale
Pao
Rale
PriX
24
Ria
37
29
20
20
10
20
Tratzing TokenotB)
Tisiaing Tokens/B)
Truinks TokenuB)
Training Tokens
Trahting Tokens/B)
Figure 12: Visualization of dynamic performance on ten benchmarks. Rule: the best performing FineWeb rule in Table 2.
29

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
E.3. Detailed Performance in Sec. 3.3
In 3.3, we test PROX's effectiveness using different sizes refining models, and also train series of models by using these curated data. We report these detailed results in Table 14, Table 15 and Table 16.
Table 14: Full evaluation results of TLM-XS trained on different PROX model curated data.
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
TLM-X trained Raw data
2500
22.5
38.5
27.0
29.1
25.8
25.0
60.2
38.8
50.4
58.6
37.6
5000
23.6
39.2
28.7
33.1
26.1
26.6
62.2
39.5
49.9
66.2
39.5
7500
23.8
42.7
28.0
33.4
26.0
26.2
64.0
39.3
51.5
67.0
40.2
10000
23.8
41.2
27.8
35.0
26.6
28.0
65.3
40.9
50.1
65.9
40.5
12500
22.6
41.9
29.7
32.8
26.2
26.4
62.2
39.3
51.3
63.3
39.6
TLM-XS trained PROX-xs data
2500
24.8
43.5
26.5
30.3
26.8
26.6
59.3
38.6
50.8
60.7
38.8
5000
23.7
44.3
28.1
33.8
27.3
28.8
61.3
38.9
50.9
70.2
40.7
7500
24.1
46.0
29.2
35.0
27.7
30.6
63.4
38.7
52.0
70.4
41.7
10000
25.3
46.1
28.3
35.7
28.1
29.2
64.4
38.5
51.2
70.6
41.7
12500
25,9
47.5
29.2
36.7
28.1
30.2
64.6
38.0
51.7
71.4
42.3
TLM-XS trained PROX-s data
2500
23.5
41.9
24.9
30.4
26.6
27.6
62.0
37.8
49.3
61.4
38.5
5000
24.7
44.5
27.0
33.8
27.5
28.0
62.4
38.0
50.6
67.0
40.3
7500
25.3
45.3
27.3
34.0
27.9
29.2
63.4
37.7
52.9
68.7
41.2
10000
25.6
45.7
27.6
35.6
28.6
30.2
63.6
37.4
52.0
71.1
41.7
12500
26.4
46.7
27.5
37.2
28.1
29.8
62.8
37.8
52.2
70.1
41.9
TLM-XS trained on PROX
curated data
2500
22.9
41.3
26.5
31.1
26.9
27.0
62.2
37.6
50.6
62.4
38.9
5000
25.8
44.0
27.3
34.0
27.1
29.6
63.1
38.5
51.8
64.9
40.6
7500
26.0
45.3
28.5
27.7
29.8
63.6
39.4
51.3
68.5
41.7
10000
26.0
46.6
28.8
37.3
27.6
30.6
63.3
38.7
51.6
70.3
42.1
12500
26.5
46.4
29.1
37.6
28.1
29.4
64.1
38.7
51.5
68.0
41.9
We also further scale PROX to other two pre-training corpora, C4 and FineWebb. We also scale our training to about 50B tokens, and directly compare with existing well-trained models developed by different research groups. We report our detailed results in Table 17, Table 18 and Table 19. We also present other models' results in Table 20.
30

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 15: Full evaluation results TLM-S trained on different PROX model curated data.
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
TLM-S trained on Raw data
2500
22.1
39.0
27.6
31.6
25.9
26.6
61.2
37.3
48.9
59.1
37.9
5000
24.4
41.2
28.8
34.8
26.7
27.0
64.9
39.3
50.4
61.9
39.9
7500
26.5
43.9
29.5
37.2
27.2
29.0
64.8
38.7
50.8
68.2
41.6
10000
25.8
43.5
29.1
27.4
29.8
66.9
39.0
51.2
66.2
41.8
12500
26.1
44.3
29.7
39.1
27.3
29.2
66.9
39.0
52.0
67.4
42.1
TLM-S trained on PROX-x xs curated data
2500
23.8
44.1
26.5
33.5
26.9
29.4
60.7
38.9
50.6
62.1
39.6
5000
26.8
48.1
28.4
36.7
28.0
30.6
64.0
38.6
50.3
65.6
41.7
7500
26.9
49.0
30.6
39.5
28.2
29.6
65.3
39.6
52.2
69.6
43.0
10000
26.7
51.3
29.4
40.1
28.3
31.8
64.1
39.3
51.4
69.9
43.2
12500
26.8
52.1
30.2
41.8
28.5
31.6
65.5
39.5
51.9
70.8
43.9
TLM-S trained PROX-s curated data
2500
24.9
43.4
27.3
32.1
26.9
28.2
60.9
38.8
51.2
60.8
39.5
5000
24.9
49.6
28.8
36.8
27.9
30.6
64.7
38.8
51.1
66.9
42.0
7500
25.5
51.2
30.8
38.8
28.4
31.2
67.3
40.2
50.3
71.7
43.5
10000
26.2
51.7
30.8
39.9
29.0
32.6
68.6
39.7
51.7
73.7
44.4
12500
26.4
51.9
30.9
42.4
29.4
31.6
67.9
40.0
52.2
73.5
44.6
TLM-s s trained on PROX
curated data
2500
25.3
45.3
27.5
32.2
26.7
27.0
62.4
38.7
50.6
60.8
39.6
5000
26.1
45.4
28.6
37.2
27.4
27.8
65.7
38.9
50.9
65.6
41.4
7500
27.1
47.5
30.6
41.0
28.6
29.2
66.8
39.3
51.1
69.9
43.1
10000
26.7
50.5
30.7
41.5
28.4
30.2
67.0
40.1
49.9
70.9
43.6
12500
27.4
50.7
30.6
42.0
28.8
30.2
67.4
39.4
48.8
70.1
43.5
31

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 16: Full evaluation results of TLM-M trained on different PROX model curated data.
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
TLM-S trained on Raw data
2500
23.5
41.5
27.5
32.9
26.4
25.2
62.1
39.4
51.5
65.1
39.5
5000
24.0
42.1
29.6
37.6
27.6
27.2
65.0
39.7
53.2
68.5
41.4
7500
24.3
44.9
28.9
39.3
27.8
27.6
66.4
40.4
51.3
69.2
42.0
10000
46.1
29.6
27.9
28.4
67.5
39.8
51.9
70.9
42.8
12500
26.3
46.8
29.0
43.2
28.3
27.8
68.2
40.5
50.7
72.5
43.3
TLM-M trained on PROX-xx curated data
2500
24.9
49.6
26.5
27.3
30.4
61.8
37.9
51.3
65.1
40.9
5000
26.7
47.6
28.6
39.7
28.5
31.8
65.4
39.5
50.2
70.7
42.9
7500
27.5
52.1
30.4
41.8
29.6
31.8
67.6
39.6
51.7
75.2
44.7
10000
28.4
54.7
29.8
45.2
30.8
31.8
67.9
39.7
52.0
77.7
45.8
12500
28.8
54.2
29.7
46.5
30.9
31.8
68.2
39.9
51.3
78.3
46.0
TLM-M trained PROX-s curated data
2500
25.3
45.7
27.8
34.2
27.8
29.0
64.4
37.5
49.3
66.3
40.7
5000
26.1
49.0
28.8
40.2
29.2
30.8
65.6
39.0
50.5
71.2
43.0
7500
27.7
53.6
31.1
44.1
29.6
34.8
67.6
39.4
52.5
72.2
45.3
10000
27.2
54.0
31.5
45.1
30.3
33.8
67.7
39.7
52.9
74.2
45.6
12500
28.6
56.1
31.8
45.5
30.5
34.4
68.5
39.4
51.3
76.1
46.2
TLM-M M trained PROX
curated data
2500
24.7
44.1
25.9
34.8
27.4
27.8
62.9
38.9
49.2
67.0
40.3
5000
27.7
48.0
26.8
40.5
28.5
30.6
67.4
39,4
50.3
69.1
42.8
7500
26.7
51.9
26.7
42.9
29.3
31.4
69.1
40.3
50.4
73.3
44.2
10000
28.4
52.4
27.9
45.0
29.7
32.0
70.2
40.0
51.9
75.4
45.3
12500
28.3
53.7
28.4
45.9
30.1
33.8
70.6
41.1
52.3
72.5
45.7
32

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 17: Full evaluation results scaling pre-training about 50B tokens RedPajama- V2.
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
TLM-M trained RedPajama- V2 raw data.
2500
24.0
42.9
26.6
33.7
25.9
26.0
62.4
39.4
52.3
64.0
39.7
5000
24.3
45.9
26.4
37.4
27.0
27.6
64.1
39.7
49.5
66.2
40.8
7500
25.1
45.3
28.8
40.3
27.1
29.2
66.3
39.1
51.7
66.9
42.0
10000
25.8
49.3
31.5
42.5
28.0
28.8
66.7
39.6
51.5
74.0
43.8
12500
25.3
50.1
30.2
43.0
28.2
30.0
66.6
39.2
51.1
74.2
43.8
15000
26.2
50.3
31.2
44.3
28.8
28.4
68.2
39.8
51.7
76.2
44.5
17500
25.8
51.1
30.8
44.7
29.0
29.6
67.7
39.2
52.6
75.2
44.6
20000
26.7
52.5
31.7
47.2
28.6
30.4
69.0
39.6
53.0
78.2
45.7
22500
27.4
51.7
47.2
29.3
30.4
69.5
39.5
51.9
78.5
45.7
25000
26.9
51.4
32.4
47.3
29.3
32.2
69.7
39.6
52.1
79.1
46.0
TLM-M trained PROX refined RedPajama- V2 data.
2500
24.8
46.8
27.2
33.8
27.3
28.2
61.3
38.6
50.3
65.1
40.3
5000
26.9
49.3
28.5
40.1
28.0
30.6
66.2
39.7
50.2
70.1
43.0
7500
28.5
53.1
29.2
41.7
29.4
33.2
66.9
39.3
53.0
73.0
44.7
10000
28.2
53.5
30.1
43.6
29.8
31.6
68.4
39.6
52.0
75.3
45.2
12500
29.5
55.3
30.2
46.4
30.5
32.2
68.6
40.2
52.6
76.9
46.2
15000
30.0
57.1
30.2
47.6
30.9
33.0
69.5
39.8
52.2
77.8
46.8
17500
31.5
59.6
29.4
49.5
31.6
33.6
69.4
39.8
53.0
78.9
47.6
20000
31.2
61.2
29.4
50.4
31.4
35.2
70,6
40.1
53.7
79.6
48.3
22500
32.0
61.7
30.2
51.4
31.4
34.0
70.0
39.9
53.2
79.5
48.3
25000
31.1
60.7
29.8
51.0
31.7
33.2
70.9
39.2
53.3
79.1
48.0
Table 18: Full evaluation results scaling pre-training about 50B tokens C4.
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PiQA
SIQA
WinoG
SciQ
AVG
TLM-M trained on C4 raw data.
2500
22.4
39.7
26.8
36.5
26.5
27.6
64.8
40.2
50.1
60.0
39.5
5000
23.9
42.9
27.5
42.3
27.1
29.6
68.2
39.6
50.3
66.6
41.8
7500
44.8
28.2
45.4
27.1
29.2
70.7
40.7
51.6
66.3
42.9
10000
25.5
46.0
32.3
48.2
27.9
31.6
71.1
39.7
52.3
67.6
44.2
12500
25.8
48.8
30.3
49.7
27.9
31.6
71.2
40.9
52.0
69.4
44.8
15000
26.9
48.0
50.5
28.5
31.4
71.9
41.1
51.4
69.7
44.8
17500
26.6
48.8
30.3
52.1
28.6
31.2
73.2
41.6
52.0
70.0
45.4
20000
26.3
50.1
29.7
52.5
28.5
32.6
72.3
41.7
52.3
71.0
45.7
22500
25.8
50.7
31.0
52.9
28.8
33.8
73.0
41.6
53.0
71.5
46.2
25000
25.3
48.8
30.1
52.4
28.8
32.2
72.0
40.6
53.6
71.7
45.5
TLM-M trained PROX refined C4 data.
2500
24.1
45.9
26.0
37.3
27.2
29.0
66.3
39.8
50.8
65.9
41.2
5000
27.3
50.0
26.6
42.4
28.6
33.8
68.1
40.5
53.0
71.9
44.2
7500
28.3
53.7
27.7
47.7
29.3
35.4
71.1
39.3
54.0
73.1
46.0
10000
30.0
54.3
28.1
50.9
30.0
33.6
71.2
40.6
52.0
74.2
46.5
12500
29.3
56.7
27.5
52.3
30.9
33.8
72.8
39.9
52.5
77.5
47.3
15000
29.6
55.9
28.3
53.9
30.6
35.0
72.9
41.0
53.8
75.8
47.7
17500
30.6
55.5
28.7
53.3
31.2
34.2
73.6
40.4
53.4
76.7
47.8
20000
30.0
57.6
28.3
54.9
31.1
37.2
74.6
40.7
53.6
79.4
48.7
22500
30.1
56.7
28.6
55.2
31.4
37.2
73.8
41.6
53.3
77.7
48.6
25000
31.1
56.0
28.4
55.2
31.1
36.2
74.0
41.0
54.1
76.8
48.4
33

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 19: Full evaluation results on scaling pre-training to about 50B tokens on Fine
Train Steps
ARC-C
ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
TLM-M trained on FineWeb raw data.
2500
22.9
41.2
28.9
34.3
26.1
27.6
64.8
39.3
52.1
62.8
40.0
5000
25.5
44.5
30.4
39.8
26.9
32.0
68.4
39.2
52.1
67.2
42.6
7500
26.8
45.6
31.4
44.1
27.6
30.2
70.9
38.8
52.2
70.3
43.8
10000
27.2
46.2
31.3
47.2
28.3
31.6
72.1
38.8
53.4
69.0
44.5
12500
26.4
49.2
32.1
48.7
28.7
31.6
71.5
40.1
52.6
74.7
45.6
15000
27.1
49.6
32.8
49.5
28.9
31.0
72.7
39.0
52.3
77.1
46.0
17500
26.4
50.9
33.8
51.3
29.3
31.0
71.9
39.3
53.0
78.0
46.5
20000
27.1
53.1
33.2
51.2
29.6
32.2
73.4
39.7
52.3
76.3
46.8
22500
27.1
51.2
34.9
51.7
29.5
33.4
73.7
40.1
52.4
78.0
47.2
25000
28.5
52.6
33,9
53.2
29.8
32.6
72.9
40.2
53.0
77.1
47.4
TLM-M trained on PROX refined Feb data.
2500
25.8
46.8
27.4
36.1
27.7
28.8
63.9
39.3
51.9
69.1
41.7
5000
28.5
52.1
28.8
43.5
29.3
32.6
66.4
38.7
51.2
71.3
44.2
7500
28.2
52.0
30.6
45.9
29.9
33.0
69.3
39.5
51.7
71.8
45.2
10000
29.3
54.3
30.6
48.5
30.8
33.2
69.7
40.7
50.6
74.4
46.2
12500
28.7
57.8
30.7
48.1
31.1
32,6
72.0
40.4
52.7
77.4
47.2
15000
59.6
31.9
50.4
31.8
34.4
71.9
40.5
50.8
78.0
48.0
17500
32.6
60.9
31.9
51.5
32.2
33.8
72.3
39.7
52.5
78.9
48.6
20000
33.2
62.5
32.5
51.6
32.4
34.6
72.4
39.7
51.7
80.7
49.1
22500
34.7
63.6
32.9
53.3
32.9
34.8
73.1
40.3
54.2
80.5
50.0
25000
34.4
63.9
32.6
53.0
33.1
34.4
73.1
39.3
52.7
81.5
49.8
Table 20: Detailed evaluation results existing base models trained on different corpora and trained using different techniques.
ARC-C ARC-E
CSQA
HellaSwag
MMLU
OBQA
PIQA
SIQA
WinoG
SciQ
AVG
TINYLLAMA-L.I 1.IB (trained on 3T tokens)
31.5
59.0
35.5
57.8
32.8
33.4
72.8
40.0
56.0
82.4
50.1
OLMo- IB (trained on 2T tokens)
31.4
59.7
38.9
61.9
32.2
38.4
76.1
41.5
53.9
78.8
51.3
PYTHIA-1.44
28.7
56.9
34.7
51.7
31.5
36.0
71.8
40.8
55.1
79.3
48.7
PYTHIA-2.SB
32.9
61.0
36.5
60.4
33.3
35.0
73.5
41.1
57.0
83.1
51.4
SHEAREDLLAMA-I.. (pruned from 1.AMA-2-7BB
22.4
39.7
29.3
36.0
26.4
28.4
62.6
39.9
52.0
71.4
40.8
SHEAREDLLAMA- 1.0MMA-1.3BB 1.3B (pruned from LLAMA-2-7B, and further trained on 50B tokens)
29.0
58.3
34.8
59.6
32.0
35.0
74.6
41.0
56.3
82.3
50.3
INSTRUCTI.M113 (LLMM data synthesis)
28.1
57.9
32.5
52.3
30.0
34.0
74.5
39.9
56.1
86.9
49.2
CosMo-1.8D (LLM data synthesis)
33.4
57.0
31.2
55.1
32.4
35.2
71.4
42.0
54.7
84.4
49.7
34

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
E.4. Evaluation Results of Continual Pre-training in Sec. 3.4
We provide full ablation results for each base model, as shown in Table 21. We can observe that PROX-D+C consistently improves average performance over PROX-D across various base models. Although the performance gain from PROX- D+C compared to PROX-D is less pronounced than the improvement of PROX-D over continual pre-training on raw WpenW,,,ath, this is both understandable and expected. PROX-D+C does not significantly reduce the token count beyond the reductions achieved by PROX-D alone. Given the scale of the OpenWebMath corpus, a more aggressive token removal strategy could potentially diminish the diversity of unique tokens below the threshold necessary for robust pre-training. This observation underscores the delicate balance between data refinement and maintaining sufficient linguistic variety for effective language model training, particularly when working with limited-scale corpora.
Table 21: Full ablation results OpenWebMath Continual Pre-training (CPT), All models are tested using few-shot CoT prompts. LLEMMA and INTERNLM2-N MATII are continual pre-trained models from CODELLAMA (Rozière et al., 2023) and INTERNLM2 (Team, 2023) with public available data, respectively. DEEPSBEK-LAE -LLM denotes an internal DeepSeek model, and the model trained on Open WehMath introduced by Shao et al. (2024). Note that the unique tokens and training tokens in the column refer exclusively to the token numbers from math-specific corpora (calculated by corresponding tokenizers). !; MQA evaluation of INTERNLM2- BASE is based on an alternative prompt due to non non- con-prediction issues with the original prompt. The bolded entries represent the best results within the same base model and CPT experiments.
Uniq Train Toks Toks
MMLU SAT STEM MATH
Model
Size Method
GSM8K MATH SVAMP ASDiv MAWPS TAB MQA
AVG
Existing Continual Pre-training for Reference
1.3B
2.9
3.0
19.5
15.6
DEEPSECK-LLM,
1.3B
14B
150B
11.5
8.9
29.6
31.3
7B
11.8
5.0
44.2
50.7
62.6
30.6
14.3
20.4
21.9
29.1
CODELLAMA (Base)
34B
31.8
10.8
61.9
66.0
83.4
51.6
23.7
43.0
53.1
47.3
7B
5SB
200B
38.8
17.2
56.1
69.1
82.4
48.7
41.0
45.4
59.4
50.9
(+21.8)
LLEMMA
34B
55B
50B
54.2
23.0
67.9
75.7
90.1
57.9
49.8
54.7
68.8
60.1 (+12.8)
7B
27.0
6.6
49.0
59.3
74.8
40.1
20.97
19.0
28.1
36.1
WNTERNLM2-BASH
20B
50.6
18.8
72.5
75.9
93.9
45.4
33.1
53.7
59.4
55.9
7B
31B
125B
41.8
14.4
61.6
66.8
83.7
50.0
57.3
24.8
37.5
448.7 (+12.6)
INTERNLM2-MATE
20B
120B
500B
65.4
30.0
75.7
79.3
94.0
50.9
38.5
53.1
71.9
62.1 (+6.2)
Applying Data Refinement Approaches
TINYLLAMA (Base)
|..BB
2.8
3.2
10.9
18.0
20.2
12.5
14.6
16.4
21.9
14.7
I.IB
15B
15B
6.2
4.8
22.3
36.2
47.6
19.3
11.6
20.7
25.0
21.5 (+6.8)
L.BB RHO
15B
9B*
7.1
5.0
23.5
41.2
53.8
18.0
TINYLLAMA (CPT)
L.I Rule
6.5B
15B
4.5
2.8
17.5
29.4
39.3
15.1
12.4
19.4
25.0
18.4 (+3.7)
LIB PROX-D
5.4B
15B
9.3
7.4
23.4
41.9
55.6
22.1
14.6
24.1
25.0
24.8 (+10.1)
LIB PROX-D+C
SB
15B
9.0
5.6
23,8
41.9
56.9
22.2
15.6
26.8
31.2
25.7 (+11.0)
LLAMA-2 (Base)
7B
14.1
3.8
39.5
51.6
63.6
30,9
12.5
32,9
34.4
31.5
7B
15B
10B
29.6
13.6
49.2
61.9
78.4
36.3
31.9
40.5
43.8
42.8 (+11.3)
LLAMA-2 (CPT)
7B
PROX-D
5.4B
10B
30.3
16.0
54.2
63.8
79.5
37.3
37.2
44.2
46.9
45.5 (+14.0)
7B
PROX-D+C
5B
10B
30.6
16.8
50.2
63.7
79.3
37.3
40.1
43.8
53.1
46.1 (+14.6)
CODELLAMA (Base)
7B
11.8
5.0
44.2
50.7
62.6
30.6
14.3
20.4
21.9
29.1
7B
15B
10B
31.1
14.8
51.4
62.1
81.2
33.6
30.4
40.5
43.8
43.2 (+14.1)
CODELLAMA (CPT)
7B
PROX-D
5,4B
10B
38.1
17.0
54.2
67.0
83.1
40.9
39.8
43.7
50.0
48.2 (+19.1)
7B
PROX-Do D+C
5B
10B
35.6
17.6
55.8
67.9
82.7
41.3
38.9
42.6
62.5
49.4 (+20.3)
MISTRAL (Base)
7B
40.6
11.4
65.4
68.5
87.0
52.9
32.3
50.0
56.2
51.6
7B
15B
10B
44.4
19.2
65.2
69.6
88.4
46.6
43.1
50.8
65.6
54.8 (+3.2)
MISTRAL (CPT)
7B
PROX-D
5.5B
10B
47.8
24.8
63.5
72.4
88.9
48.3
48.2
54.1
62.5
56.4 (+4.8)
7B
PROX-D+C
4.7B
10B
51.0
22.4
64.9
72.9
89.2
49.8
53.0
54.2
75.0
59.2 (+7.6)
Besides, we report the detailed dynamic evaluation results of our continual pre-training experiments on Open WebMath:
"RHO-1 I only counts the selected tokens that are used for training (loss calculation).
35

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Tables 22, 23, 24, and 25 present the evaluation results for TINYLLAMA- I.I......................
Tables 26, 27, and 28 present the evaluation results for LLAMA-2.
Tables 30, 31 present the evaluation results for CODELLAMA.
Tables 32, 33, and 34 show the evaluation results for MISTRAL-7B.
Table 22: Full evaluation results of TINYLLAMA-L. IB continual pre-training on with raw data. Note that about IB tokens are trained per 500 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
2.8
3.2
10.9
18
20.2
12.5
14.6
16.4
21.9
14.7
500
1.9
3.4
16.3
23.9
30.3
13.9
10.3
14.8
18.8
14.8
1000
3.1
2.2
16.6
25.6
32.4
12.5
12.0
16.6
25.0
16.2
1500
2.7
3.0
17.6
28.5
34.5
13.9
8.7
14.1
15.6
15.4
2000
4.5
3.2
16.4
28.5
39.0
15.1
10.2
16.6
34.4
18.7
2500
4.9
3.4
19.3
31.0
39.2
16.0
12.1
18.6
9.4
17.1
3000
4.1
5.2
19.1
32.0
43.0
15.3
9.6
16.1
18.8
18.1
3500
4.9
3.6
19.7
31.4
40.4
18.1
11.3
19.6
15.6
18.3
4000
4.8
4.8
19.5
33.8
44.5
16.4
10.7
19.9
12.5
18.5
4500
5,4
4.8
20.2
35.0
45.2
17.9
12.7
21.0
18.8
20.1
5000
5.5
4.6
22.3
34.6
42.9
16.0
10.6
21.7
28.1
20.7
5500
4.9
5.8
23.6
35.2
44.0
20.4
11.0
21.1
21.9
20.9
6000
6.1
4.4
22.8
36.2
45.4
17.8
12.7
21.4
15.6
20.3
6500
6.3
3.6
23.2
37.3
48.0
19.7
10.3
21.0
18.8
20.9
7000
4.6
22.2
36.6
46.9
19.4
12.0
21.5
21.9
21.2
7500
6.2
4.8
22.3
36.2
47.6
19.3
11.6
20.7
25.0
21.5
36

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 23: Full evaluation results of TINYLLAMA-L. continual pre-training on Open WatMath with data after rule-based filtering. Note that about IB tokens are trained per 500 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
2.8
3.2
10.9
18
20.2
12.5
14.6
16.4
21.9
14.7
500
3,4
3.6
13.6
22.5
25.9
13.1
14.2
13.5
28.1
15.3
1000
3.0
2.8
14.1
22.5
27.8
11.4
11.0
16.4
12.5
13.5
1500
3.6
3.2
13.6
24.0
31.2
13.9
9.2
18.0
18.8
15.1
2000
3.5
2.4
15.0
25.1
33.0
12.5
10.6
13.9
15.6
14.6
2500
3.3
1.6
15.0
25.3
33.5
13.7
11.1
18.1
25.0
16.3
3000
3.5
3.0
16.4
25.5
33.4
14.1
10.2
18.4
18.8
15,9
3500
3,2
3.4
17.2
27.0
37.7
14.6
11.2
13.3
25.0
17.0
4000
3,5
3.6
15.6
26.2
36.5
13.4
12.1
15.9
18.8
16.2
4500
4.1
3.8
15.6
27.9
38.2
14.9
11.6
17.1
18.8
16.9
5000
4,2
3.6
18.6
28.7
37.7
14.3
12.7
17.5
21.9
17.7
5500
4.1
3.8
16.3
29.3
38.4
14.7
10.8
17.5
18.8
17.1
6000
4.3
3.6
16.0
28.7
39.1
13.5
12.8
19.5
21.9
17.7
6500
4.2
3.2
16.4
29.5
39.0
15.1
11.7
17.9
21.9
17.7
7000
4.0
4.0
16.2
29.6
37.9
16.0
13.8
17.8
21.9
17.9
7500
4.5
2.8
17.5
29.4
39.3
15.1
12.4
19.4
25.0
18.4
Table 24: Full evaluation results of TINYLLAMA-L.. 1.11. continual pre-training on with data after PROX-D. Note that about IB tokens are trained per 500 steps.
Train Steps
MMLU STEM
SAT MATH
GSMSK
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
0
2.8
3.2
10.9
18
20.2
12.5
14.6
16.4
21.9
14.7
500
3.3
2.8
17.7
29.0
38.7
12.4
9.5
15.7
15.6
16.1
1000
4.6
4.0
18.1
31.6
41.9
15.9
11.9
18.2
25.0
19.0
1500
5.2
5.4
21.1
32.9
43.1
15.3
11.1
20.4
12.5
18.6
2000
6.8
5.8
20.2
33.5
46.6
18.2
10.7
20.3
12.5
19,4
2500
7.1
3.8
20.7
37.0
48.6
18.3
12.0
21.4
18.8
20.9
3000
7.4
4.4
22.9
37.1
50.5
18.3
12.3
21.2
25.0
22.1
3500
8.8
4.8
22.8
39,4
53.3
19.2
12.0
22.8
34.4
24.2
4000
8.6
4.6
24.0
38.7
51.4
18.8
14.8
24.4
18.8
22.7
4500
8.6
4.2
24.2
39.2
53.6
20.4
13.5
23.9
18.8
22.9
5000
8.9
5.2
24.0
40.0
52.6
20.0
13.6
23.9
18.8
23.0
5500
8.0
6.2
23.2
41.4
55.0
22.3
14.3
24.9
25.0
24.5
6000
8.3
5.2
22.2
39.8
54.0
24.3
12.6
25.1
31.2
24.7
6500
9.4
5.6
24.4
40.2
54.5
20.3
13.0
24.9
31.2
24.8
7000
9.2
5.8
25.8
40.6
55.3
22.5
12.5
24.5
21.9
24.2
7500
9.3
7.4
23.4
41.9
55.6
22.1
14.6
24.1
25.0
24.8
37

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 25; Full evaluation results of TINYLLAMA-L. -.LLMM--I.IBB continual pre-training on Open Math with data after PROX-D+C. Note that about IB tokens are trained per 500 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
2.8
3.2
10.9
20.2
12.5
14.6
16.4
21.9
14.7
500
4.3
5.0
16.4
28.8
36.4
15.3
11.4
18.5
15.6
16.9
1000
5.5
3.8
20.5
34.6
44.6
15.3
12.1
19.6
28.1
20.5
1500
5,2
4.4
21.4
34.5
44.7
16.1
11.2
21.4
34.4
21.5
2000
6.3
5.4
20.1
33.7
46.2
19.4
10.5
21.2
12.5
19.5
2500
7.8
5.4
22.1
37.0
49.5
17.9
13.3
22.9
21.9
22.0
3000
6.4
3.4
23.0
38.6
51.1
18.5
12.6
24.3
18.8
21.9
3500
8.5
4.6
24.1
40.2
53.8
22.1
12.5
23.1
25.0
23.8
4000
8.2
6.0
24.1
41.0
52.4
19.8
10.2
26.1
31.2
24.3
4500
8.3
5.4
24.1
41.3
54.4
20.6
15.2
24.2
28.1
24.6
5000
8.5
7.0
26.0
40.5
54.9
21.7
13.9
25.5
34.4
25.8
5500
8.7
4.0
23.2
41.1
54.8
20.5
14.4
26.5
21.9
23,9
6000
8.3
5.0
24.8
41.3
54.3
23.2
14.0
25.3
25.0
24.6
6500
8.6
6.4
24.5
41.6
55.1
22.2
14.4
26.5
25.0
24.9
7000
8.9
6.0
23.4
40.5
53.4
22.0
13.8
27.3
28.1
25.0
7500
9.0
4.4
23.8
41.9
56.4
22.2
15.6
26.8
31.2
25.7
Table 26: Full evaluation results of LLAMA-2 continual pre-training on OpenWebMatth with raw data. Note that about 1B tokens are trained per 1000 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
14.1
3.8
39.5
51.6
63.6
30.9
12.5
32.9
34.4
31.5
1k
17.2
3.6
39.1
50.4
63.0
30.2
18.9
31.8
31.2
31.7
2k
19.7
6.0
43.9
55.5
68.3
32.9
19.0
33.0
37.5
35.1
3k
19.6
8.6
42.9
56.3
68.4
32.2
17.4
34.6
40.6
35.6
4k
21.8
8.8
44.6
57.3
72.0
28.9
23.6
35.8
40.6
37.0
5k
22.6
10.4
45.9
57.0
73.5
31.5
23.9
39.0
43.8
38.6
6k
24.5
10.0
44.9
57.6
73.7
35.5
25.8
36.1
43.8
39.1
7k
23.3
10.4
46.5
59.0
75.3
32.9
27.7
39.0
50.0
40.5
8k
29.0
12.4
46.4
59.7
77.0
33.1
30.2
38.8
50.0
41.8
9k
26.1
12.8
48.8
59,9
74.3
35.0
28.3
39.2
50.0
41.6
10k
29.6
13.6
49.2
61.9
78.4
36.3
31.9
40.5
43.8
42.8
38

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 27: Full evaluation results of LLAMA-2 continual pre-training on OpenWebMath with PROX-D. Note that about IB tokens are trained per 1000 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
14.1
3.8
39.5
51.6
63.6
30.9
12.5
32.9
34.4
31.5
lk
17.1
7.2
39.8
51.6
68.4
31.4
21.4
35.2
40.6
34.7
2k
21.9
9.2
43.2
57.0
72.8
33.1
24.0
37.6
56.2
39.4
3k
20.5
10.8
45.7
58.6
76.2
35.3
25.8
38.3
53.1
40.5
4k
27.2
11.8
45.7
58.7
76.6
35.9
29.2
41.0
31.2
39.7
5k
28.9
14.2
49.3
60.2
77.9
38.8
32.8
41.7
53.1
44.1
6k
31.9
15.0
51.5
62.0
79.0
39.2
33.3
41.4
68.8
46.9
7k
31.5
16.8
51.9
63.2
77.9
36.5
35.9
43.8
43.8
44,6
8k
30.3
13.8
51.9
63.7
80.6
38.3
36.1
41.3
59.4
46.2
9k
30.6
14.0
52.7
62.6
78.7
37.5
36.1
43.2
43.8
44.4
10k
30.3
16.0
54.2
63.8
79.5
37.3
37.2
44.2
46.9
45.5
Table 28: Full evaluation results of LLAMA-2 continual pre-training on OpenWebMath with PROX-D+C. Note that about IB tokens are trained per 1000 steps.
Train Steps
MMLU STEM
SAT MATH
GSMSK
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
14.1
3.8
39.5
51.6
63.6
30.9
12.5
32.9
34.4
31.5
lk
18.8
6.8
40.1
54.4
66.1
29.7
22.9
35.6
53.1
36.4
2k
23.1
8.6
45.7
56.5
72.7
30.7
25.1
35.6
46.9
38.3
3k
23.4
11.8
47.9
59.1
74.6
30.4
28.2
38.3
59.4
41.5
4k
25.2
14.2
49.0
57.8
72.7
32.8
33.1
40.7
40.6
40.7
5k
24.4
13.6
48.0
58.7
72.1
28.9
33,0
40.6
50.0
41.0
6k
29.6
12.8
46.1
63.4
75.6
33.7
31.6
42.8
53.1
43.2
7k
29.9
13.6
50.5
61.5
75.2
36.4
34.5
41.7
53.1
44.0
8k
30.2
15.8
50.8
63.7
77.1
37.7
36.3
43.4
43.8
44.3
9k
34.0
15.4
52.1
62.4
79.3
35.9
40.2
44.0
56.2
46.6
10k
30.6
16.8
50.2
63.7
79.3
37.3
40.1
43.8
53.1
46.1
39

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 29; Full evaluation results of CODELLAMA-7B continual pre-training on OpenWebMath with raw data. Note that
about 1B tokens are trained per 250 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
11.8
5.0
44.2
50.7
62.6
30.6
14.3
20.4
21,9
29.1
250
16.7
8.2
45.2
52.2
65.3
33.9
16.0
28.8
43.8
34.5
500
18.3
7.8
43.1
53.9
69.0
29.3
15.3
22.5
37.5
33.0
750
20.2
8.0
45.2
54.2
71.9
29.9
17.1
31.2
37.5
35,0
1000
24.7
9.8
40.6
58.6
72.7
29.3
20.7
31.9
34.4
35,9
1250
24.3
10.4
44.0
57.5
74.8
29.2
21.4
36.1
50.0
38.6
1500
26.2
13.2
48.4
58.8
75.4
29.4
28.1
34.9
50.0
40.5
1750
25.5
11.8
49.1
58.7
76.6
32.4
26.7
37.3
43,8
40.2
2000
28.0
13.6
46.3
61.7
80.0
33.8
29.4
37.2
50.0
42.2
2250
27.7
13.6
48.9
62.2
80.3
32.5
28.9
39.1
59.4
43.6
2500
31.1
14.8
51.4
62.1
81.2
33.6
30.4
40.5
43.8
43.2
Table 30: Full evaluation results of CODELLAMA continual pre-training on OpenWebMath with PROX-D, Note that about IB tokens are trained per 250 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
11.8
5.0
44.2
50.7
62.6
30.6
14.3
20.4
21.9
29.1
250
21.1
9.2
48.7
56.1
71.3
33.4
22.2
34.1
50.0
38.5
500
23.7
11.6
49.8
57.4
74.7
32.9
28.5
35.8
59.4
41.5
750
25.1
15.4
48.1
58.9
78.8
36.8
29.4
37.6
53.1
42.6
1000
28.4
14.2
50.9
61.2
79.8
36.7
27.7
37.6
50.0
42.9
1250
33.0
15.2
49.3
62.9
81.1
33.4
32.8
41.0
46.9
44.0
1500
36.0
15.0
54.2
65.0
81.0
39.3
34.1
42.0
62.5
47.7
1750
34.7
14.6
53.1
63.6
83.3
40.6
35.9
43.4
62.5
48.0
2000
35.7
17.6
53.3
65.4
83.5
42.4
37.1
42.4
56.2
48.2
2250
37.2
18.8
54.5
65.4
83.2
41.9
41.0
44.9
71.9
51.0
2500
38.1
17.0
54.2
67.0
83.1
40.9
39.8
43.7
50.0
48.2
40

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 31: Full evaluation results of CODELLAMA continual pre-training on Open with PROX- -DOX- Note that
about 1B tokens are trained per 250 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
11.8
5.0
44.2
50.7
62.6
30.6
14.3
20.4
21.9
29.1
250
18.1
10.2
46.0
54.5
71.9
33.0
21.3
34.4
50.0
37.7
500
22.4
10.0
50.3
59.7
76.4
31.3
26.1
36.0
59.4
41.3
750
26.8
11.4
51.2
61.0
78.5
34.9
26.4
38.0
53.1
42.4
1000
29.0
14.4
54.1
62.8
80.1
36.9
34.2
40.4
62.5
46.0
1250
31.4
15.0
51.7
63.8
81.1
37.2
32.5
41.4
75.0
47.7
1500
31.5
17.4
53.4
64.4
80.7
39.6
35.4
41.6
71.9
48.4
1750
33.7
15.2
50.6
64.3
81.5
39.2
36.1
40.5
53.1
46.0
2000
36.2
16.0
54.7
65.1
83.1
39,9
39.1
43.4
71.9
49.9
2250
37.1
16.6
55.3
65.6
82.4
41.3
36.5
42.7
75.0
50.3
2500
35.6
17.6
55.8
67.9
82.7
41.3
38.9
42.6
62.5
49.4
Table 32: Full evaluation results of MISTRAL-7B continual pre-training Open OpenWebMathh with raw data. Note that about IB tokens are trained 1000 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
0
40.6
11.4
65.4
68.5
87.0
52.9
32.3
50.0
56.2
51.6
1k
31.6
12.0
56.5
66.0
80.1
43,9
27.1
45.1
56.2
46.5
2k
32.4
10.8
54.7
63.5
82.6
40.8
31.6
45.7
59,4
46.8
3k
33.6
14.8
60.4
64.7
84.5
43.5
33.1
47.2
68.8
50.1
4k
35.1
14.8
58.7
65.2
84.4
41.2
38.5
47.3
62.5
49.7
5k
33.4
16.0
59.3
65.0
83.8
46.7
34.6
49.1
62.5
50.0
6k
38.7
16.6
61.5
68.1
86.1
47.4
35.3
48.5
37.5
48.9
7k
39.6
17.2
60.5
68.2
86.2
44.4
38.5
49.3
53.1
50.8
8k
44.0
16.4
64.5
69.8
88.7
45.5
41.3
50.6
59.4
53.4
9k
43.9
19.4
63.7
69.7
87.6
44.9
42.9
51.0
62.5
54.0
10k
44.4
19.2
65.2
69.6
88.4
46.6
43.1
50.8
65.6
54.8

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 33: Full evaluation results of MISTRAL-7B continual pre-training on OpenWebMath with PROX-D. Note that about 1B tokens are trained per 1000 steps.
Train Steps
MMLU STEM
SAT MATH
GSM8K
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
40.6
11.4
65.4
68.5
87.0
52.9
32.3
50.0
56.2
51.6
lk
36.8
14.6
57.2
66.1
83.1
45.7
32.6
47.7
59.4
49.2
2k
38.5
17.0
57.9
69.0
86.3
44.7
33.6
49.2
56.2
50.3
3k
40.0
19.0
59.3
68.7
87.0
46.8
41.0
48.0
68.8
53.2
4k
38.5
20.4
59.3
66.2
85.1
42.6
42.8
49.5
68.8
52.6
5k
42.5
20.2
63.0
70.5
86.6
47.2
43.4
49.8
62.5
54.0
6k
46.8
17.8
62.5
72.7
88.2
51.2
47.7
51.3
56.2
54.9
7k
47.5
22.4
64.1
71.8
89.1
51.4
47.9
52.4
65.6
56.9
8k
44.6
23.8
63.2
70.8
87.7
47.6
49.1
54.1
65.6
56.3
9k
46.6
24.6
61.6
72.3
86.4
46.9
49.8
53.2
65.6
56.3
10k
46.7
22.6
63.5
72.4
88.9
48.3
48.2
54.1
62.5
56.4
Table 34: Full evaluation results Mistral-7B continual pre-training on OpenWebMath with PROX-D+C. Note that about IB tokens are trained per 1000 steps.
Train Steps
MMLU STEM
SAT MATH
GSMSK
MATH
SVAMP
ASDiv
MAWPS
TAB
MQA
AVG
40.6
11.4
65.4
68.5
87.0
52.9
32.3
50.0
56.2
51.6
lk
30.9
16.0
60.1
64.5
85.3
40.8
33.9
48.0
59.4
48.8
2k
40.3
17.6
63.0
66.3
86.2
48.0
33.9
48.7
53.1
50.8
3k
42.4
17.8
59.6
69.1
85.7
50.1
38.5
49.9
59.4
52.5
4k
43.8
20.4
63.7
69.3
88.2
46.2
46.3
50.9
65.6
54.9
5k
42.5
18.4
59.3
69.6
87.9
44.3
46.1
51.9
65.6
54.0
6k
47.7
21.8
62.7
71.7
89.2
47.9
48.4
54.0
68.8
56.9
7k
46.8
21.6
62.9
72.1
88.4
50.1
46.4
52.5
68.8
56.6
8k
48.4
21.6
65.0
72.7
89.2
51.1
49.4
52.9
65.6
57.3
9k
48.5
24.8
64.4
72.6
88.3
50.7
48.1
53.4
62.5
57.0
10k
51.0
22.4
64.9
72.9
89.2
49.8
53.0
54.2
75.0
59.2
42

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
F. Analysis
F.1 Token Length Distribution
Table 35: Average length of token per document for different refining methods.
Methods
General Domain
Math Domain
N/A
1217.5
1815.8
Rule
1329.4
1955.6
PROX (ours)
2004.8
1734.9
As previously discussed in $4, our analysis reveals notable document length distribution shift in the data refined by PROX, specifically significant increase in the average token length (from 1217.5 to 2004.8 tokens per document). When further compared to the rulc-based method (we compare to Fine rules), we only observe marginal increase in token length within the general domain (from 1217.5 5 to 1329.4 tokens).
Interestingly, in the math domain, we observe an opposite trend. The raw data shows an average token length of 1815.8, which our method reduces 1734.9. while the rule-based method increases 1955.6. And the training performance in Table 5 follows the order: PROX > original >> rule-based method for TINYLLAMA-I. 1.1..11111 B. This again implies that mathematical documents used for pre-training exhibit significant differences in distribution and characteristics compared to those in the general domain.
F22 Case Studies
We provide several cases to qualitatively illustrate the refinement effect of PROX, as shown in Tables 36-37. For the general domain, using RedPajama-Vo as an example, we observe that PROX can drop low-information documents, remove meaningless content such as navigation bars, and replace URL links (see Table 36). In the mathematics domain, PROX demonstrates the ability to eliminate documents with minimal relevance to mathematical reasoning and remove less important elements like functional buttons (see Table 37). These refinements enhance the quality and relevance of the processed data across different domains.
F.3. Error Analysis
As shown in Table 38, the failure ratio across both refining stages (document-level and chunk-level) and domains (General and Math) is remarkably low (< 0.5%).. This demonstrates that ProX's 's refining tasks well-suited for small models. Specifically, for the General domain, failure ratios are 0.04% for document-level and 0.36% for chunk-level refining, with an average of 3.7 function calls per program in the chunk-level stage. For the Math domain, these ratios are 0.06% and 0.11%, respectively, with an average complexity of 2.7 function calls at the chunk-level stage.
Despite the low failure rates, we observed two prevalent failure cases ProX's programs:
1. Repeated output or empty output: This occurs when program inadvertently generates duplicate outputs or fails to produce any meaningful results. Such failures are typically linked to improper loop conditions or insufficient constraints in processing logic.
2. Non-existent target removal: In cases, ProX's 's programs attempt to remove string line that does not exist in the input data. This leads to incomplete execution or errors in the program output, particularly in datasets with irregular formats or unexpected variations.
As shown in Table 39, present failure cases illustrate instances of repeated output and non-existent target strings.
43

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 36: Cases from RedPajama- V2 after applying PROX. Text in red indicates content be removed replaced. "..." denotes omitted content due to limited space.
Case
Michael Lewis
ContributorMichael Lewis
Michael Lewis is possibly the most entertaining nonfiction writer alive. If that's not true it's at least close to true. Liar's Poker, Moneyball, The Blind Side, his NYT article about Jonathan Lebed (Google it): what's love?
504: How Got Into College
Act Two: My Ames is True
Writer Michael Lewis tells the story of man named Emir Kamenica, whose path college started with flecing the war Bosnia and becoming refugee the United States. Then a stroke luck; a student teacher read an essay he' d plagiarized from a book he' d stolen from a library back in Bosnia, and was so impressed that she got him out of bad high school and into a much better one.
Act Three
Michael Lewis' story continues, and he figures out why Emir Kamenica insists remembering, and telling, the story of his life the way he does when he finds out that some of the facts may be wrong.
Output by PROX: drop_doc ()
Case 2
Home & Staff Staff' search Dr Tim Overton Dr Tim Overton BSc PhD)
School of Chemical EngineeringSenior Lecturer
Telephone (+44) (0) 121 414 5306Emailt w.overton @ www.baa..c.. ac.uu
AddressSchool of Chemical Engineering University of Birmingham
B15 2TT
Dr Tim Overton is biochemist and molecular microbiologist who interested in applying molecular biology and single- cell techniques to understand and develop bioprocesses. He is active in microbial flow cytometry rescarch and collaborates widely with bioprocess engineers, molecular microbiologists, cell biologists and environmental microbiologists to develop new methods of answering fundamental questions on single-cell level.
His research also focuses on using bacteria to make useful products such as protein drugs and small molecules, and the bacterial responses to stress encountered in such processes. Current and recent research funding has come from the BBSRC, TSB and EU FP7. He is the director of the MSc Biochemical Engincering. Pages: 34
Google scholar: http:///sshhbtrroooooooooooottte
Output by PROX:
keep doc
remove_lines (Line-start=0, line_end=5) normalize target_str-"") normalize (source.str- 4", target.str="")
44

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 37: Cases from Open WebMath after applying PROX. Text in red indicates content to be removed or replaced. ...." denotes omitted content due to limited space.
Case
## unhybridized pi bonds sp, sp", app,sp" sp", dsp", d' sp
Tatiana 4B
Posts: 30
Joined: Sep 28, 2018 12:28 am
##ffh unhybridized pi bonds
### Re: unhybridized pi bonds
I am not too sure in my knowledge about this, but think that both have hybridized orbitals. Since hy hybridization defined as the phenomenon of intermixing of the orbitals such as sp, sigma and pi bonds are just different types of the way the atomic orbitals hybridize with each other. Sigma bonds result
covalent bonds formed depending of when the overlap of orbitals of two atoms takes place along the line joining two orbitals, while pi bonds are when two atoms overlap due to the sideways overlap their 'p" orbitals.
Hannah Yates 1K
Posts: 59
Joined: Fri Sep 28, 2018 12:27 am
#### Re: unhybridized pi bonds
I am also not too sure answer, but I am pretty sure that a sigma bond has just hybridized orbitals, but the reason bond can form is because of extra (not hybridized) p orbital. This allows for double and triple bond to form.
Output by PROX: drop doc ()
Case
Solution
Trigonometric Identities
Account Register
Share
Books Shortlist ConceptTrigonommtric Identities
Question
Prove the following trigonometric identities:
(i) sin
I-coa 5 cosecf + cot
Solution
You need to to view the solution Is there an enror in this question or solution?
Reference Material
Solution for concept: Trigonometric Identities. For the course CBSE
S
Output by PROX:
keep_doc ()
lines (line-started line.end=7)
remove ines (line start=18, Line.end=245
45

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
Table 38: Failure ratio and average complexity (function calls) for ProX refining stages across domains.
Domain
Failure Ratio (doc-level)
Failure Ratio (chunk-level)
Complexity (AVG. function calls)
General Domain
0.04%
0.36%
3.7
Math Domain
0.06%
0. %
2.7
Table 39: Failure cases from RedPajama- V2 during applying PROX. " denotes omitted content due to limited space.
The notation like [004] used indicate the line number.
Case I: Repeated output Empty output)
[004] P: 114 1. The problem statement, all variables and given/known data Mercury is poured into U-tube in Figure 1.......aaas don't t understand why you would know set the two volumes equal to each other? How do you know the volumes are the same?
[0C7] Related Discussions Mechanical Engineering 6 Introductory Physies Homework 0 General Engineering Introductory Physics Homework 2 Introductory Physics Homework 2
Output by PROX:
remove lines (start=1, end=l)
renove..lines (startt= (start=6, remove__iines (start=7, end-7)
end=6)
remove_lines (start-?, end-7)
remove_lings (start-7, end-7)
temove.lines (start=7, end
Case Non-existent target string
[195] 18. Sathyamorrthi, C. R., Mhekomize, C., Mapharing, M., & Selinkie, P. (2018). The Impact Corporate Governance on Working Capital Management Efficiency: Evidence from the Listed Companies in the Consumer Services Scctor in Botswana. International Journal of Economics and Finance, 10, 135. http:///oo.org/11.553/////// 10t.iieff..0112211133
[196] 19. Vu, T. M. T., Tran, C. Q. Doan, D. T., & Le, N. (2020). Determinants of Capi- Structure: The Case Vietnam. Journal of Asian Finance, Economies, And Business, 7(9), 159-168. http::/doii.rg// org/10.
https//dooorrgg/1...310/o/ofee.p 131166/aafebb222222 202.feeb.000.oor. wo...no9.11...
Output by PROX:
# Analysis:
this
'source str'
be found in the original 2020. no2. 53",
text
normalize (source.str== sooore.stt=hhttts//ool. http://///oo...r/.....o1. target.str="")
46

---

Programmming Every Example: Lifting Pre-training Data Quality like Experts at Scale
F.4. Computing Overhead Analysis
According to Kaplan al. (2020), both training and inference computational FLOPs for Transformer-basee Language Models (denoted as Cirin and Cinference) can be approximated as the product of model parameters (non-embedding parameter) N and the number of tokens D. This can be expressed as:
Crrin se 6 NNminn: train:
(9)
iiffrrence R 2 N (D Ppreanl + D Déecude):
(10)
In PROX, we through two data refining stages before final training, which incurs additional inference-time computational FLOPs. Suppose the refining model parameter for each stage is denoted as Nietiee and the raw data size in tokens is Draw
For the first document-level stage, the computational cost can be approximated
Cdne Rd 2- Nrefine (Draw + Doutpu) Nrefine Draw: (suppose Doupuur
< Druw)
(11)
resulting new pool of data sized Danc- Similarly, for the second chunk-level stage, the computational cost is:
Cehunk 2 2- N₂ (Daxe + Doutpui) 2 2- N,Daw
(suppose Doupuu < D
(12)
which produces the final refined data size of Dprex.
Thus, the total computational overhead for PROX can be calculated as the sum of the two stages:
CPROX Cdoe + Cetunk 2e 2 roccu raw 2 - rehunk.refing doc-
(13)
In general, we use refining models with the same sizes, so the final inference overhead can be estimated as
CPRO se 2 Nyefine D Drow + Daoe).
(14)
Additionally, we omit the FLOPs for fine-tuning since they are negligible compared to the large-scale pre-training and inference FLOPs.
47


Reviews:
[{"review_id": "pdUjBesnON", "reviewer": "Reviewer_pU7q", "summary": "Data curation for LLMs typically relies on rule-based filtering to discard documents or refine them. However, these rules are inflexible and cannot adapt to the unique characteristics of each sample, but it would be laborious for a practitioner to determine how to refine/discard at the sample level. This paper proposes ProX, a procedure where a small LM is trained to produce a set of functions that discard and refine each sample. These functions are executed to refine the dataset, and then a LLM is trained on this curated dataset. This approach allows for per-sample flexibility by using the small LM, resulting in better performance than rule-based filtering approaches. Moreover, ProX's curation process is fairly efficient due to the size of the small LM and the programmatic use of the LM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work! We are truly delighted to see your appreciation of ProX's novelty, efficiency, and effectiveness in improving data quality. Regarding your question:\n\n$\\textrm{\\color{blue}Question 1}$\n```\nCan the exact rules for C4, Gopher, and FineWeb be made available in the appendix?\n```\n\nThank you for the suggestion. For these rule-based filtering methods, we primarily refer to the implementations provided in the FineWeb codebase. We are happy to summarize them below and will include the full details in the appendix in our revised version.\n\n- C4: We re-implement the C4 filtering rules, including:\n  1. Filtering lines with overly long words or too few words\n  2. Removing citations, placeholder text (e.g., lorem ipsum), JavaScript, bracket content, and web policy mentions\n  3. Discarding documents with too few sentences\n- Gopher: We adopt Gopher’s rules, including:\n  1. Minimum and maximum word count per document\n  2. Symbol-to-word ratio\n  3. Average word length per document\n  4. Heuristics such as bullet point filtering\n- FineWeb: We use FineWeb’s official implementation, which includes:\n  1. Punctuation ratio checks\n  2. Duplicate line removal\n  3. Filtering based on newline character frequency\n  4. Additional combined filters integrating the above rules\n\nWe hope this clarifies our reference to these rule-based methods, and we’ll ensure their presentation in the appendix for reproduction.\n\n$\\textrm{\\color{blue}Question 2}$\n```\nCould experiments/ analyses be conducted to isolate ProX's performance improvement due to per-sample application versus its particular rules.\n```\n\nRegarding the performance of ProX in isolation, we would like to clarify two points:\n1.  ProX inherently and functionally covers other rule-based filtering methods, as it leverages a language model to learn and apply data selection criteria, effectively capturing patterns encoded by handcrafted rules.\n2.  During our preliminary study, we explored training the refining model using data filtered by FineWeb rules as negative samples. Compared to using the original raw data(a subset of Red-Pajama-V2), ProX achieved approximately **+1.0%** improvement in downstream performance, indicating its ability to identify lower-quality documents.\n\nFurthermore, we observed that more aggressive filtering tends to yield better performance. The use of quantifiable scores such as Edu Score and Format Score offers us the flexibility to adjust filtering thresholds and control the trade-off between data quality and quantity.\n\n$\\textrm{\\color{blue}Question 3}$\n```\nCould the paper provide an example of how a practitioner/ domain expert would add more complex functions to ProX and which parts of the procedure they would need to adjust?\n```\nRegarding the addition of specific rules or filtering strategies, we believe it is important to distinguish between document-level and chunk-level refining:\n- At the document level, if a domain expert aims to remove biased content (e.g., safety issues or toxic data), they can simply annotate relevant keywords and mark the corresponding documents with a drop_doc label. This allows for efficient and targeted removal based on expert knowledge.\n- At the chunk level, domain experts may need to:\n  1. Identify specific types of noise with clear patterns — such as frequently repeated dates in forum pages, pervasive advertisement slogans, or unnatural line breaks that disrupt the reading flow.\n  2. Abstract the noise removal strategy into a reusable function — for example, leveraging regular expression (regex) syntax for flexible and customizable pattern-based filtering.\n  3. Use model or human annotation to create seed data, which can then be used to train a refining model tailored to the noise patterns of interest.\n\nWe hope this explanation clarifies our approach.", "claims_and_evidence": "- Claims around ProX's performance and efficiency are supported by clear empirical evidence. \n- The paper seems to claim that ProX's key advantage over prior heuristic-based approaches is its flexibility to refine each sample individually, in contrast to low-coverage, hard-coded rules. However, when comparing ProX with other filtering rules like FineWeb, Gopher, or C4, it's unclear whether ProX's improvements are due solely to its per-sample, LLM-based programming approach, or also because of the nature of the rules it uses. For example, ProX uses soft rules like: 1) discarding documents with low educational content or poor formatting scores 2) within chunks, removing elements such as navigation bars, URLs, and footers. In contrast, FineWeb/Gopher/C4 use different rules; for instance, C4 removes documents where the fraction of lines ending in punctuation is low. To better understand where ProX's performance gains come from, it would be insightful to run a baseline where you \"convert\" the traditional heuristic rules from FineWeb/Gopher/C4 into flexible, LM-driven programs that can adapt per sample. This would help isolate whether the improvement is due to better rules or the adaptive, per-sample application of those rules.", "methods_and_evaluation_criteria": "Proposed method and evaluation makes sense.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Experimental design and analyses appear sound.", "relation_to_broader_scientific_literature": "This paper strikes a middle ground between two existing types of approaches for data curation: 1) manual filtering rules that can cheaply be run across many samples, but are inflexible and have low coverage, and 2) expensive, per-sample LLM-based approaches, such as getting each sample to be scored/edited/synthesized by a powerful LLM. The paper shows that we can combine strengths from both of these paradigms to produce data curation methods that are relatively efficient yet of higher quality than heuristic approaches.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n- Remarkable results showing that the this approach can be used on a variety of model sizes and datasets, outperforming hard-coded rules as well as model-based data selection algorithms. \n- ProX is an efficient curation approach (figure 7)---the total FLOPS used by ProX (including training) is less than standard training (especially for larger models). This is in contrast to other curation strategies, which are oftentimes equally or more expensive than the training itself.\n\nWeaknesses:\n- The applicability of ProX could be enhanced if the paper included more concrete information on how one (i.e., a domain expert) could specify new functions---I wonder if there are limitations of using Llama-70B to annotate the seed data for domain-specific functions. It would also be nice if approaches like Rephrasing the Web/WRAP (Maini et. al.) could be expressed in this framework and combined with the functions in this paper.", "other_comments_or_suggestions": "Typos:\n- In table 1, \"orignal\" -> \"original\"", "questions_for_authors": "1. Can the exact rules for C4, Gopher, and FineWeb be made available in the appendix?\n2. Could experiments/analyses be conducted to isolate ProX's performance improvement due to per-sample application versus its particular rules? \n3. Could the paper provide an example of how a practitioner/domain expert would add more complex functions to ProX, and which parts of the procedure they would need to adjust?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k0Tr7xjoyT", "reviewer": "Reviewer_4XW9", "summary": "The paper introduces the ProX framework which trains a model that generates short program instructions to clean pre-training documents.\nThe authors prompt large-language models to score the quality and format of documents and to remove various website artifacts (header, footers, URLs, and navigation elements). The generated programs unify both document filtering and chunk rewriting. \n\nThe paper includes experiments which show that the proposed method boosts the performance of language models when applied to the pre-training data. The method performs well across three model scales (up to 1B) and different pre-training datasets, and outperforms various baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing the practicality of our method and its strong empirical results across model scales and datasets. For your questions and suggestions:\n\n$\\textrm{\\color{blue}Question 1}$ \n```\nThe comparison in Table7 in the appendix would be important to show in Table2, and the \"kept ratio\" would also be important in Table 2.\n```\nThank you for your careful reading and for recognizing that ProX achieves a higher kept ratio than FineWeb-Edu. We will update the presentation accordingly in our revised version.\nRegarding the kept ratio of rule-based filtering methods on **RedPajama-V2**, we summarize the results as follows. Notice that **aggressively ensembling these filtering methods does not show very obvious improvement (+0.2%) but only keeps 50% of the data**.\n\n| Filtering Method      | Kept Ratio | Avg. Performance Gain |\n|-----------------------|------------|-----------------------|\n| C4                    | 73.60%     | 0.50%                 |\n| Gopher                | 70.50%     | 0.20%                 |\n| FineWeb               | 84.50%     | 0.70%                 |\n| C4 + Gopher + FineWeb | 50.00%     | 0.20%                 |\n| **ProX-D**               | 30.00%     | **1.20%**                 |\n| **ProX-D+C**              | 25.60%     | **2.50%**                 |\n\nOn FineWeb, we observe the following:\n\n| Method      | Kept Ratio | Avg. Performance |\n|-------------|------------|------------------|\n| FineWeb-Edu | 8.60%      | 45.2             |\n| **ProX**        | **28.00%**     | 45.2             |\n\nDespite keeping more than **3×** the data, ProX achieves **comparable downstream performance** to FineWeb-Edu, highlighting its effectiveness in retaining high-quality content.\n\n$\\textrm{\\color{blue}Question 2}$ \n```\nHere all baselines only perform document filtering, so showing separate rows for either document filtering, chunk rewriting, or both would be important to understand the source of the gain.\n```\nIn Table 3, we use only prox's doc refining performance to compare with other baseline data selection methods (please see our experiment setting from line 294 to line 296) This is mainly to show that using a very small LM on quality refinement task is indeed very simple and effective compared to other data selection baselines.\n\n\n$\\textrm{\\color{blue}Question 3}$ \n```\nThe chunk-level rewriting could be compared to other approaches of rewriting text. A good baseline would training the same ProX base model to generate the cleaned document (with headers, footers, URLs, navigation removed) directly.\n```\nThank you for the insightful suggestion. While rewriting is a valid approach to improve pre-training data quality, it incurs high inference costs due to full-document generation and relies heavily on large models. Recent work like WRAP and Nemotron-CC uses models over 7B (e.g., Mistral NeMo 12B). In contrast, ProX improves data quality with much lower compute by using a small 0.3B model and concise outputs. We believe efficient rewriting would require a strong base model and leave this as future work.\n\n$\\textrm{\\color{blue}Suggestion}$ \n```\nInclude deduplication as a standard part of the data curation pipeline and its relation to ProX.\n```\nThank you for the suggestion! We have discussed in the related work section that deduplication plays an important role in data preprocessing—it reduces redundancy, improves training stability, and enhances efficiency. ProX is orthogonal to deduplication, as it focuses on the quality of individual samples rather than redundancy. In practice, ProX can be applied after deduplication to further improve data quality while saving compute. We will include this discussion in the revised related work section.\n\n$\\textrm{\\color{blue}Question 4}$ \n```\nCan you explain what seed data the fasttext baseline is trained on?\n```\nTo ensure a fair comparison, we re-trained the fastText classifier on the exact same training data used for our ProX doc-level refining models. All documents labeled with `drop_doc()` are treated as negative samples, while those labeled with `keep_doc()` are treated as high-quality samples. We trained the language model on FastText filtered data from scratch using all the same configuration as the other experiments in Table 2.", "claims_and_evidence": "The central claim of superior pre-training performance is well supported by many experiments, even though the paper should compare to stronger baselines (see below). The authors also show that at larger model scale the gains in performance can justify the increased inference cost of running their ProX model on the pre-training corpus.\n\nAnother contentious issue is the comparison to methods which directly rewrite documents using LMs. The authors argue that these methods are less scalable, but it is not clear whether a strong 0.7B model might not be sufficient to rewrite data directly and the experiments do not include this relevant baseline.", "methods_and_evaluation_criteria": "The proposed method is good solution to the practical problem of selecting and rewriting pre-training data. The evaluation of the method follows standard practices from previous papers.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The choice of model sizes, data scales and evaluation tasks is appropriate for an academic paper on pre-training data.\n\nHowever, I believe that document filtering and chunk-level rewriting are distinct contributions and would benefit from better individual experiments:\n\n1. Table 2 shows how well ProX curates data from a raw corpus. However, it misses the current best curation baselines, FineWeb-Edu and DCLM-baseline. FineWeb-Edu is especially important, since document-level ProX is re-uses the FineWeb-Edu prompt (Appendix A), but using a different threshold and augmented with a format prompt.\nThe comparison in Table 7 in the appendix would be important to show in Table 2, and the \"kept ratio\" would also be important for the other methods in Table 2.\n2. Table 3 shows how well ProX performs in curating a pre-processed corpus (C4). Here all baselines only perform document filtering, so showing separate rows for either document filtering, chunk rewriting, or both would be important to understand the source of the gain. Similarly, Figure 6 shows how much ProX improves the performance of strong existing corpora (in my view, the most compelling evidence for ProX). Here it would also be very useful to understand whether the performance edge is mostly due to the chunk rewriting or additional filtering.\n3. The chunk-level rewriting could be compared to other approaches of rewriting text. A good baseline would training the same ProX base model to generate the cleaned document (with headers, footers, URLs, navigation removed) directly.", "relation_to_broader_scientific_literature": "Both prompt-based selection and chunk rewriting are established techniques for curating training data.\nIn fact, the selection strategy is very similar to FineWeb-Edu (Penedo et al., 2024a).\nThe major novelity is to use structured commands (string replacement and line removal) to modify the pre-training data, which has previously been done by using LMs to rewrite the entire document. While the authors that their method is more scalable (which I am not sure about) and may be more reliable (I am tempted to agree), there is no evidence to compare between these sophisticated methods. The authors also argue that they unify chunk rewriting and document filtering in a single programming framework, but the practical value of this is not clear, especially as multiple models are still used.\n\nBesides novelty, the released models are undoubedly useful resources to the community.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "It would be useful to discuss deduplication as a standard part of the data curation pipeline and its relation to ProX.", "questions_for_authors": "Can you explain what seed data the fasttext baseline is trained on?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxOo7mHdMv", "reviewer": "Reviewer_T2Yk", "summary": "This work proposes a new method to clean data for LM pre-training. Their method is based on a small model, which can use/combine/create functions/programs to clear and transform documents (getting rid of noise and unnecessary things). Empirically, they observe a performance improvement in downstream tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our extensive experiments, method soundness and novelty! We are happy to answer all your questions.\n\n$\\textrm{\\color{blue}Question 1}$\n```\nWhat is the main reason why documents get lengthier after cleaning? I would expect the opposite (maybe shorter documents are getting deleted?)\n```\nYes, as you correctly pointed out, the primary reason for the longer average document length in ProX-refined data is that general text corpora—**even after extensive rule-based filtering**—still contain many **short and noisy** documents. These low-quality documents are often discarded during the document-level refining stage, leading to a curated dataset with naturally longer documents.\n\nAt the chunk-level refining stage, we observe a 5–10% reduction in total tokens depending on the corpus, and document lengths correspondingly decrease. However, they **still remain consistently longer than those in the original corpus**, indicating that ProX effectively removes noise while retaining substantial and meaningful content.\n\n$\\textrm{\\color{blue}Question 2}$\n```\nMaybe you're plotting the frequency and not density?\n```\nThank you for your careful reading! After checking it, we do not perform normalization on the Y-axis, thus showing the frequency instead of density. We will correct this typo in the updated version.", "claims_and_evidence": "The claims made are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The method makes sense and is novel: using a small LM to clean/modify data, as far as I know, is a novel approach. The authors evaluate their method quite extensively.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments and analyses are sound.", "relation_to_broader_scientific_literature": "The relation seems clear from their related work section and, from what I get, their method offers novelty in how they use a LM to clean/modify lower quality documents.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "- In Figure 7, what is the main reason why documents get lengthier after cleaning? I would expect the opposite (maybe shorter documents are getting deleted?).\n- In the first plot of Figure 7, shouldn't the two areas under the curves be equal, one? One of them seems much greater than the other... maybe you're plotting the frequency and not density?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Programming Every Example (ProX), a novel framework for refining pre-training data for large language models (LLMs). ProX leverages a small language model to generate and execute fine-grained operations, such as string normalization, for each individual example at scale. The experiments demonstrate that models trained on ProX-refined data consistently outperform other baselines across various benchmarks, showcasing its effectiveness across different model sizes and pre-training corpora.

**Strengths and Weaknesses Highlighted by Reviewers:**
**Strengths:**
  - **Remarkable results** showing ProX's effectiveness across different model sizes and datasets.
  - **Efficiency** in data curation, with lower computational costs compared to other strategies.
  - **Flexibility** in refining individual samples, leading to better performance than rule-based approaches.

**Weaknesses:**
  - **Applicability** could be enhanced with more concrete information on specifying new functions.
  - **Comparison** with traditional heuristic rules needs further exploration to isolate performance improvements.
 - **Prior work discussion** needs to be extended to include other relevant curation methods

**Rebuttal and Discussion with Reviewers:**
The authors provided detailed responses to reviewers' questions, clarifying the rules used for filtering and the performance gains of ProX. They conducted additional experiments using FineWeb-style prompts to generate seed data for ProX training, showing that the choice of underlying rules matters. The authors also addressed concerns about the efficiency and robustness of ProX, highlighting its advantages over full-document generation methods.

**Recommendation:**
All things considered, this paper would be a valuable contribution to ICML due to its innovative approach to data refinement and its demonstrated effectiveness across multiple benchmarks. The authors have shown that ProX can significantly improve the quality of pre-training data while maintaining efficiency. However, we expect the authors to improve the writing, particularly in clarifying the relationship to previous works and the practical implementation of new functions, as suggested by the reviewers, especially 4XW9.
Decision: Accept (poster)