ICML 2025

Programming Every Example:
Lifting Pre-training Data Quality Like Experts at Scale
Fan Zhou * 1 2 Zengzhi Wang * 1 2 Qian Liu 3 Junlong Li 1 Pengfei Liu ‡ 1 2 4
Abstract
Large language model pre-training has tradition-
ally relied on human experts to craft heuristics for
improving the corpora quality, resulting in numer-
ous rules developed to date. However, these fixed
rules lack the flexibility to address the unique
characteristics of individual examples, yet craft-
ing sample-wise rules is impractical for human
experts. In this paper, we show that even small
language models, with only 0.3B parameters, can
exhibit substantial data refining capabilities. We
propose Programming Every Example (PROX),
a novel framework that treats data refinement as
a programming task, and enables the model to
refine corpora by generating and executing fine-
grained operations, such as string normalization,
for each individual example at scale. Experiments
show that models trained on PROX-refined data
consistently outperform other baselines across 10
benchmarks, demonstrating effectiveness across
model sizes (up to 1.7B) and pre-training cor-
pora (C4, RedPajama-V2, FineWeb, FineWeb-
Edu, and DCLM). PROX also shows great po-
tential in continual pre-training: on math domain,
PROX boosts 7B models by up to 20% within
10B tokens—results typically achieved with much
larger scale training (e.g., 200B tokens). We be-
lieve PROX offers a way to curate high-quality
pre-training data, and finally contributes to effi-
cient LLM development.
1. Introduction
Large Language Models (LLMs) have made significant
strides in capabilities (Meta, 2024; Achiam et al., 2023;
Anthropic, 2024; Reid et al., 2024), excelling in tasks such
*Equal
contribution
1Shanghai
Jiao
Tong
University
2Generative AI Research Lab (GAIR) 3Sea AI Lab 4Shanghai
Artificial Intelligence Laboratory. Correspondence to: Pengfei Liu
<pengfei@sjtu.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancou...