Title: Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
Venue: NeurIPS 2025
Accepted: no
Keywords: LLM

large language models

pretraining

data filtering

data pruning
Abstract: As large language models (LLMs) are pretrained on massive web corpora, careful selection of data becomes essential to ensure effective and efficient learning. While perplexity (PPL)-based filtering has shown strong performance, it suffers from drawbacks: substantial time costs and inherent unreliability of the model when handling noisy or out-of-distribution samples. In this work, we propose a simple yet powerful alternative: a \textbf{prior-based data filtering} method that estimates token priors using corpus-level term frequency statistics, inspired by linguistic insights on word roles and lexical density. Our approach filters documents based on the mean and standard deviation of token priors, serving as a fast proxy to PPL while requiring no model inference. Despite its simplicity, the prior-based filter achieves the highest average performance across 21 downstream benchmarks, while reducing time cost by over \textbf{1000×} compared to PPL-based filtering. We further demonstrate its applicability to symbolic languages such as code and math, and its dynamic adaptability to multilingual corpora without supervision. The code is available online (\url{https://anonymous.4open.science/r/prior_filter-D88D}).


Peer reviews:
comments: SUMMARY:
This paper proposes a prior-based data filtering method as a fast and powerful alternative to perplexity (PPL)-based filtering. To address the slow speed and unreliability of the PPL approach, this new method is inspired by linguistic frequency analysis and requires no model inference. It works by estimating token "priors" from term frequency and then filtering documents based on the mean and standard deviation of these priors. Experiments show this method is over 1000x faster than PPL-based filtering while achieving superior average performance across 21 downstream benchmarks, and is also effective for symbolic and multilingual corpora

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposed method shows a large efficiency improvement over PPL-based methods.
2. The method is easy to implement.

Weakness:
1. The main weakness is the scale of the experiment and evaluation. Since the authors claim that they select data for pre-training, a 3B-tokens training is too small for a valid evaluation of pre-trained models. Besides, the scores on some core benchmarks (like MMLU) are just slightly better than a random guess, making the soundness of the experimental results questionable. 
2. The method identifies outliers by measuring deviation from the corpus median of prior-based statistics. While effective for removing noise, this approach risks discarding legitimate, high-quality documents that are simply statistical outliers. For instance, a highly technical paper with rare terminology or a poem with unusual syntax could be incorrectly filtered

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel data selection method based on the mean and std of token priors of a document. The method presents itself as a more computationally efficient and reliable method than its perplexity-based alternative filtering. It analyses the method from various aspects, such as anecdotal evidence of filtered samples as well as showing promising performance in an ablation study with a 1.5B and 137M parameter LLM.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- This work presents a novel, simple and compute-efficient, but effective alternative to perplexity filtering\n- The work is well written and the experimental design is structured and methodical.\n- The work analyzes the prior based filter performance from a variety of interesting aspects, such as anecdotal evidence for filtered documents, behavior in multilingual context or overlap with perplexity based filtering.\n\n**Weaknesses**\n- The related work is missing key works in data curation (filtering), such as the-state-of-the-art datasets DCLM and Fineweb-Edu and how the contribution of this work is set within that context together with a critical discussion. Potentially, even an experiment with a multilingual dataset such as HPLT or FineWeb-2 would have been interesting to investigate the performance aspect of the language learnability of the prior-based filtering approach.\n\nQUESTIONS:\n- How would the authors set their work in the context of recent data curation strategies, beyond perplexity filtering, such as Fineweb-Edu and DCLM?\n- In Figure 4) could the authors please clarify the rationale why they consider a 50% overlap as a strong correlation?\n- Could the authors please clarify whether they used cloze-style MMLU evaluation? Otherwise, I would be surprised to see MMLU included as a benchmark at this scale (1B model trained on 6B tokens), as I would expect it to perform below random. \n- Should appendix section C’s title be: “Details on Benchmarks”?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces a method to filter data sources for language model training using term-frequency based priors. They contrast it with previous perplexity-based methods, and demonstrate that their method is equally effective while being significantly cheaper and faster. They also show that the method can be applied in multilingual settings, as well as specialized languages like code.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Very simple and elegant solution, utilizing linguistics.
2. The analysis is well-organized and thorough.
3. The method is not only faster and cheaper compared to PPL, it’s also useful in cases when a reference model might be too small to learn the prior well. The prior-selected documents also overlap heavily with the PPL-selected ones.
4. The prior-based method does not require careful curation of reference dataset.

Weaknesses:

1. This is not a weakness of only this method per se, but all automated filtering methods - they only include what is most common. For certain cases, or to ensure that even minority data sources are represented in the data, you may still require heuristics-based filtering. It will be helpful to discuss this in the limitations.

QUESTIONS:
1. When using Dolma, did you include The Stack ? There’s work that shows that having some “noise” in your pretraining data is helpful overall. In this case, is it then better to apply the prior-based method to each data source individually?

LIMITATIONS:
Mentioned above.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a simple yet effective filtering method, called prior-based filtering, which identifies outlier documents based on corpus-level token frequency statistics—specifically the mean and variance of token priors.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n(+) Extremely fast and scalable compared to PPL-based filtering methods\n\n(+) Model-free and fully unsupervised, requiring no labeled data or reference corpus\n\n(+) Conceptually simple yet grounded in linguistic intuition; the paper is clearly written and easy to follow\n\n**Weaknesses:**\n\n(−) No comparison against other efficient, lightweight filtering baselines (e.g., heuristic filters or fastText classifiers)\n\n(−) Lacks in-depth analysis of sensitivity to design choices such as block size, filtering thresholds, and tokenizer granularity\n\nQUESTIONS:\n1. Have you compared prior-based filtering against other lightweight heuristic filters—such as sentence length thresholds, or script-based rules? Given the shared goal of efficiency, it would be helpful to understand whether prior-based filtering meaningfully outperforms these simpler baselines.\n\n2. Your method offers a fully unsupervised and model-free alternative to PPL-based filtering, which is a key strength. That said, many practical pipelines use lightweight supervised classifiers like fastText or simple quality classification. While fastText requires labeled data and may not be directly comparable, have you considered benchmarking against such classifier-based filters as a complementary baseline? It would help clarify the unique strengths of prior-based filtering relative to real-world, efficient alternatives.\n\n3. In multilingual corpora, your method detects the transition from unlearnable (noise) to learnable language clusters via mean priot. But in truly low-resource settings—where valuable language data never reaches that threshold—does this method risk filtering it out entirely? How might this be mitigated for underrepresented languages?\n\n4. Your method relies on token frequency statistics, which are tokenizer-dependent. Have you evaluated how sensitive the filtering results and downstream performance are to the choice of tokenizer (e.g., GPT-2 vs. multilingual or domain-specific tokenizers)?\n\n5. Your method operates at the document level, using 512-token blocks to compute the mean and standard deviation of token priors. Have you explored how sensitive the filtering behavior and downstream performance are to the choice of block size? For instance, would larger or smaller blocks change the robustness of the statistics or the ability to detect noisy samples?\n\nLIMITATIONS:\nGood enough", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-11"}