Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale

Authors: Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, Pengfei Liu
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Pre-training, Data Refinement, Data Engineering

Abstract:
Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these rules lack the flexibility to address the unique characteristics of individual example effectively.
Meanwhile, applying tailored rules to every example is impractical for human experts.
In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial
data refining capabilities comparable to those of human experts.
We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale.
Experimental results show that models pre-trained on ProX-curated data outperform either original data or data curated via selection methods by more than 2% across 10 downstream benchmarks.
Its effectiveness spans various model sizes (0.3B~1.7B) and pre-training corpora (C4, RedPajama-V2, and FineWeb).
Furthermore, ProX shows great potential in domain-specific continual pre-training: models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving accuracy by 7.6% on Mistral-7B, 14.6% on Llama-2-7B, and 20.3% on CodeLlama-7B within 10B tokens, comparable to Llemma-7B trained on 200B tokens. ProX significantly reduces training FLOPs, offering an efficient path for LLM pre-training.

Decision: reject
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730564660015

thread_type:
official_review

<thread_object>
creator: reviewer_l4dm

<content>
summary:
The authors introduce a framework called PROX for refining data quality in large language model (LLM) pre-training. This framework claims to address limitations in traditional, heuristic-based data-cleaning methods, which often lack flexibility and are labor-intensive.

strengths:
The advantage of this paper is its novelty in transforming data cleaning into a programmable task, which shifts away from traditional, rigid heuristic-based methods. By treating data refinement as a series of programmatically generated actions, the framework enables language models to apply tailored operations to each example, such as string normalization and noise filtering, with high precision.

weaknesses:
1. The first limitation is that the amount of pre-training data used in the authors' experiments is insufficient, falling far short of what would be needed to fully support their claims. It’s highly possible that, with more extensive training, the advantages of the proposed method might diminish or even disappear.
2. The second limitation is that using a model—especially a smaller one—for data selection introduces issues like hallucinations, bias, and the omission of less common information during the pre-training phase. These issues are difficult to resolve and can become deeply embedded in the foundation model. In fact, pre-training should ideally be a robust modeling of the real world, and relying on a smaller model for data refinement may skew this representation, undermining the model’s ability to capture a true and comprehensive understanding of diverse real-world contexts.

questions:
Each time the pre-trained model is updated, the entire pre-training dataset needs to be reprocessed, which is highly GPU-intensive. Do the authors have any alternative solutions to address this computationally expensive process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665086555

thread_type:
official_review

<thread_object>
creator: reviewer_1aqb

<content>
summary:
The paper "Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale" introduces ProX, a framework leveraging small language models to refine large-scale pre-training data through program generation. ProX aims to enhance the quality of training data by executing fine-grained operations on individual examples. The authors demonstrate significant improvements in pre-training efficiency and model performance across various benchmarks and model sizes. Notably, ProX can achieve remarkable gains in domain-specific continual pre-training tasks, as the authors show with OpenWebMath.

strengths:
* Novel Chunk-Level Rewriting Technique: A novel and fine-grained approach that moves beyond binary keep/dismiss judgments, allowing deterministic edits to individual chunks of text. This method is validated through robust ablation studies, proving its efficacy in improving pre-training data quality.
* Efficient Use of Compute: By leveraging small language models (0.3B parameters) for data refinement, ProX achieves substantial performance improvements with lower computational costs. This efficiency is especially noteworthy in domain-specific tasks like OpenWebMath, where ProX yields remarkable accuracy gains.
* Comprehensive Ablations: The authors tested ProX across multiple model sizes (0.3B to 1.7B) and pre-training corpora (C4, RedPajama-V2, FineWeb), with significant performance enhancements demonstrated consistently. The results show that even smaller refining models can produce high-quality pre-training data.
* Important Practical Implications: The framework offers a scalable solution to pre-training data refinement, which can be particularly valuable in scenarios where human expert intervention is impractical. ProX’s deterministic chunk-level edits contribute to a more nuanced and flexible data curation process.

weaknesses:
* Base Dataset Selection and Novelty: The exclusive reliance on older datasets like C4 and RedPajama limits the relevance of the findings. While the results on OpenWebMath are impressive, it would be more compelling to see ProX applied to modern datasets such as FineWeb-edu or DCLM-base, which have undergone extensive refinement.
* Document-Level Refinement: The document-level component of ProX appears to be a straightforward application of FineWeb-edu’s filtering techniques, reducing its novelty. The primary innovation lies in the chunk-level refinement, which could be more explicitly distinguished from the document-level methods.
* Evaluation Methodology: The emphasis on zero-shot performance may not fully capture the advantages of ProX, especially since the refining models are not fine-tuned for instruction-following tasks. Few-shot performance with higher n might provide more insightful comparisons.
* Exclusion of Base Model Training FLOPs: The paper excludes the computational cost (approximately 5.3e19 FLOPs) for training the base model used to create the refining model. This is significant, as it constitutes nearly half the total compute, raising concerns about the true efficiency gains. The authors should have explored using an existing pre-trained model to substantiate ProX’s cost-effectiveness.
* Context Window Considerations: Given that ProX needs to refine long documents, the context window of the refining model is a critical factor. The paper does not provide sufficient details on how the model handles long contexts, which could impact its practical utility.

questions:
1. Generalizability to Highly Refined Datasets: How would ProX perform when applied to datasets that are already highly refined, such as FineWeb-edu or DCLM-base? Are there diminishing returns when starting with higher-quality corpora?
2. Evaluation Choices: Why did the authors prioritize zero-shot performance for their evaluation? Wouldn't few-shot performance offer a more nuanced understanding of ProX’s impact, especially for instruction-following tasks?
3. Compute Cost of Refining Model Training: Why did the authors not also use an existing pre-trained model for the refining task? This could have significantly reduced the compute cost. Additionally, what measures were taken to address the long context requirements of creating refinement programs for lengthy documents?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719102797

thread_type:
official_review

<thread_object>
creator: reviewer_4aqx

<content>
summary:
The authors propose ProX, a method for data filtering and synthetic data generation for improving the quality of large pretraining corpora. ProX proposes to fine-tune a small language model to produce a “program”, which is a sequence of one or more filtering or normalization operations applied at the document and line level. These programs are synthesized and executed for each document in the corpus, which results in a smaller higher quality subset. The authors demonstrate that ProX yields significant improvements in pretraining and domain-specific continual pretraining in both accuracy and training efficiency across numerous baselines.

strengths:
- The paper introduces ProX, which is a novel method to improve training data quality using small language models to synthesize and execute short data refinement programs, rather than using static rules or large model data synthesis.
- Framing data filtering and synthetic data generation as a programming task that can be performed by small models is a creative contribution that has not been explored in previous work
- Prox-D and Prox-D+C outperform all heuristic and model based data selection baselines and improve pretraining efficiency. The empirical validation is comprehensive, testing the method across multiple model sizes (350M to 1.7B), different pre-training corpora (RedPajama-V2, C4, FineWeb), and domain-specific applications (OpenWebMath).
- The method shows significant gains in accuracy and efficiency for domain specific continual pretraining, with up to a 20x reduction in compute. The effectiveness of synthesizing example-specific filtering programs shows especially here, as the method can synthesize domain-specific filtering rules.
- Although there is additional computational overhead from running inference of the refining model to generate the programs, this rapidly decreases as a proportion of the total pretraining FLOPs as the pretrained model gets larger. The method shows a 67% FLOPs reduction in pretraining to a given accuracy for a model as small as 1.7B.
- The paper is well structured and clearly written, with nice looking and clear figures to explain the method and results.

weaknesses:
- The paper is missing a qualitative analysis of the programs generated by ProX. It could benefit from:
  - Examples of wrong programs generated by ProX and an analysis of the common failure modes
  - Statistics on the complexity of the programs (e.g. distribution of function calls per document)
  - An analysis of how program quality varies across different domains
  - A discussion of potential safety issues since programs may modify data in unintended or harmful ways

- The paper could benefit from comparison to simpler baselines, like binary classifiers (e.g. fastText) for quality filtering.

- The paper could explore expanding the space of programs, for example by inclusion of additional operations like text transformations, or composing multiple operations.

questions:
Did you compare Prox-D against an n-gram based filtering method like fastText? DataComp-LM (Li et. al.) find that fastText-based quality filtering was the best performing method they tried, outperforming the Gopher and C4 filtering rules, which are also used as baselines in this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730882951306

thread_type:
official_review

<thread_object>
creator: reviewer_tcqo

<content>
summary:
This paper presents Programming Every Example (PROX), a framework that redefines language model pre-training by treating data refinement as a programming task. PROX allows even small language models (as few as 0.3B parameters) to perform fine-grained data operations, achieving performance comparable to human experts. Models trained with PROX-curated data outperform those trained on original or traditionally curated data by more than 2% across ten benchmarks. It is effective with different model sizes and corpora, demonstrating significant potential in domain-specific tasks and reducing computational costs.

strengths:
**A highly complete paper, and I appreciate it along with your hard work.**

1. The paper is clearly articulated with intuitively designed charts that help readers easily understand the problems being addressed, the research motivation, and the methods employed. 

2. It is detailed in scope, including appendices with textual explanations of document/block-level programming, algorithm flowcharts, prompts used, pretraining details, baseline, downstream task introductions, and case analyses, all of which are comprehensive. 

3. The experiments are rich in content, especially the appendices, and features beautifully crafted charts. 

4. The field this paper focuses on is crucial to the LLM community -- "how to improve the quality of pretraining data."

weaknesses:
1. **This paper raises several concerns for me:**

    ``The methodology can be summarized as follows: it utilizes LLAMA's annotated pairs (doc, program) to fine-tune a small model Prox with approximately 0.3B parameters, serving as a proxy. Based on a vast corpus of pre-trained documents, Prox generates Python function calls to conduct document-level encoding (discard or retain) and chunk-level encoding (discard, retain, normalize).``

- a. Firstly, while the goal of the paper is to balance data processing efficiency and enhance data quality, the introduction of Prox as a proxy to call Python functions might add extra computational cost, potentially undermining the practical of this approach. A comparative analysis of time overhead with other methods would strengthen the argument for the effectiveness of this approach.

- b. Furthermore, if direct application of Python functions achieves the document-/chunk-level heuristic optimizations as proposed, how does this alternative compare to Prox in terms of performance? 
Because after all, the processes of discarding or retaining documents and chunks, and normalizing chunks (such as top menus, navigation bars, buttons, HTML elements, links, and footers) can be addressed using existing Python rules (as referenced with C4, Gopher, and RedPajama). 

- c. Additionally, how do the document- and chunk-level programming rules differ from existing heuristic rules, particularly in relation to the above mentioned processes (at paragraph b) and education scores, considering related research such as [1] and [2] have already proposed similar concepts?

2. The experimental results in the paper need further development. In Table 2, only a 750M LLAMA model is used for pretraining on 26B data, while the minimal data volume and model size in standard practice are at least a 1.3B model and 30B data. I believe Prox could demonstrate more significant improvements in in-context learning (ICL) capabilities with larger data scales and model sizes.

3. Some experimental comparisons require clarification. For example, in Table 5, the comparison on domain-specific data tasks should not be between CPT and the BASE model. Instead, it should be between randomly sampled data and Prox-refined data of equivalent volume size, with two BASE models trained from scratch.

4. The current paper only examines mathematics as a specific domain. Expanding the scope to include more vertical fields such as finance, education, or medicine would be beneficial.

> [1] Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. 

>[2] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high-quality data for training language models. In International Conference on Machine Learning (ICML), 2024.

questions:
Please Check Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732013855639

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated Manuscript and Response to all reviewers:

comment:
We sincerely thank all the reviewers for their insightful comments and constructive feedback.

We are delighted that the reviewers recognized the significance of our results and findings, as well as appreciated the extensive experiments we conducted (Reviewer `TCQo`, `4aQX`, `1aQb`), the detailed insights and clear presentation of our work (Reviewer `TCQo`, `4aQX`, `1aQb`), the non-trivial technical contributions of ProX to the community (Reviewer `TCQo`, `1aQb`, `L4Dm`), and the notable performance and efficiency improvements ProX achieved in pre-training and continual pre-training (Reviewer `TCQo`, `4aQX`, `1aQb`). This positive feedback is incredibly encouraging for us.

We have carefully considered the concerns and questions raised by each reviewer and have provided detailed responses to each one separately. Furthermore, based on the valuable feedback, we have made revisions to the paper, including the following updates (all updates are highlighted in purple for clarity):

- Included total FLOPs analysis in $\textrm{\color{blue}Table 3}$ to demonstrate ProX's acceptable extra FLOPs compared with QuRating and MATES. (Reviewer `TCQo`)
- Included a FastText classifier based filtering as one of our baselines in $\textrm{\color{blue}Table 2}$, $\textrm{\color{blue}Figure 4}$, and $\textrm{\color{blue}Table 11}$. While it serves as a strong baseline, it still lags behind the ProX methods. (Reviewer `4aQX`)
- Added experimental results in $\textrm{\color{blue}Figure 6}$ by applying ProX on an additional, highly-refined and top-tier quality pre-training corpus, Fineweb-Edu, to further showcase the effectiveness of ProX. (Reviewer `1aQb`)
- Added few-shot evaluation results in $\textrm{\color{blue}Table 10}$ ($\textrm{\color{blue}Appendix D.1}$) to confirm that models trained on ProX-curated data exhibit even stronger performance on downstream language tasks. (Reviewer `TCQo` and Reviewer `1aQb`) 
- Conducted error analysis in $\textrm{\color{blue}Appendix F.3}$ and $\textrm{\color{blue}Table 37}$, which highlights the robustness of refining models while also identifying areas for future improvement. (Reviewer `4aQX`)

Once again, we would like to thank all the reviewers for their time and efforts in helping us improve the paper! If you need any further explanations, please do not hesitate to let us know.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310736153

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks and Closing Summary

comment:
We sincerely thank you for recognizing the importance of the problem addressed by ProX. As Reviewer `TCQo` mentioned, "the field this paper focuses on is crucial to the LLM community." Several reviewers appreciated the extensive experiments, detailed methodology, and significant results presented in ProX:

- Reviewer `TCQo`: "The experiments are rich in content, especially the appendices."
- Reviewer `4aQX`: "The method shows significant gains in accuracy and efficiency for domain-specific continual pretraining, with up to a 20x reduction in compute." and "The empirical validation is comprehensive."
- Reviewer `1aQb`: "Significant performance enhancements demonstrated consistently. The results show that even smaller refining models can produce high-quality pre-training data."

We deeply appreciate your thoughtful comments and constructive feedback as the discussion period concludes. Thank you for your time and effort!

---

While we respectfully disagree with Reviewer `TCQo`'s perspective that "ProX may lack practical feasibility and value in real-world production" (we believe this may stem from some misalignment, which we have addressed in detail), this also highlights a critical factor: the need to emphasize how fundamentally different our approach is from static heuristic rules. Specifically, our method leverages models to automatically and dynamically generate customized programs for each document, setting it apart from the simpler baselines that `TCQo` is concerned about. Moreover, Reviewer `1aQb` explicitly stated in their review that **"The framework offers a scalable solution to pre-training data refinement, which can be particularly valuable in scenarios where human expert intervention is impractical."**

We believe we have thoroughly clarified and addressed Reviewer `TCQo`'s concerns in our responses.

---

Your feedback has also provided invaluable insights for improving our work:
- Thanks to Reviewer `TCQo`, We recognize the necessity of conducting a more explicit analysis of computational overhead. Beyond the extensive analysis already provided in $\textrm{\color{blue}Section 4.2}$ of the paper, we have prominently included FLOPs comparisons in $\textrm{\color{blue}Table 3}$ for a head-to-head comparison with existing data selection methods.
- We acknowledge the need for more comprehensive comparisons. In response, we introduced a stronger rule-based FastText filter baseline, and ProX still outperforms this baseline. (Reviewer `4aQX`)
- We applied ProX to further optimize a state-of-the-art, highly refined dataset, achieving significant improvements even at the 0.7B and 1.7B scales. This reinforces our belief in ProX's effectiveness for enhancing data quality. (Reviewer `1aQb`)
- The discussion on the data case study was another valuable suggestion. We believe this further enhances our presentation by illustrating how ProX works, how it compares to other methods, and the potential avenues for future exploration. (Reviewer `1aQb` and `L4Dm`)

---

To conclude, these suggestions will significantly strengthen our paper, and we are excited to incorporate them into the revised version. **Thank you again for your detailed and constructive feedback!**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the vital issue of enhancing pretraining data quality in the large language model (LLM) community. The authors introduce Prox, a small model with approximately 0.3 billion parameters, which fine-tunes using LLAMA's annotated pairs (document, program). Prox generates Python function calls for document-level and chunk-level encoding to determine which data to retain or discard.

Reviewers have raised concerns about the feasibility of applying Prox in real-world scenarios due to its computational costs. To strengthen their argument, the authors should provide concrete evidence demonstrating Prox's effectiveness compared to other LLM corpus filters, such as DCLM, which is known for its superior performance. This could involve comparing models trained with datasets filtered by Prox against those using the same training corpus as outlined in the DCLM paper.