Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
Authors: Yeongbin Seo
Gayoung Kim
Jaehyung Kim
Jinyoung Yeo
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
large language models
pretraining
data filtering
data pruning
Primary area: foundation or frontier models, including LLMs
TL;DR: Token frequency stats can replace perplexity for LLM data filtering—1000× faster, equally effective.

Abstract: As large language models (LLMs) are pretrained on massive web corpora, careful selection of data becomes essential to ensure effective and efficient learning. While perplexity (PPL)-based filtering has demonstrated strong performance, it suffers from drawbacks: substantial time costs and inherent unreliability of the model when handling noisy or out-of-distribution samples. In this work, we propose a simple yet powerful alternative: a prior-based data filtering method that estimates token priors using corpus-level term frequency statistics, inspired by linguistic insights on word roles and lexical density. Our approach filters documents based on the mean and standard deviation of token priors, serving as a fast proxy to PPL while requiring no model inference. Despite its simplicity, the prior-based filter achieves the highest average performance across 20 downstream benchmarks, while reducing time cost by over 1000× compared to PPL-based filtering. We further demonstrate its applicability to symbolic languages such as code and math, and its dynamic adaptability to multilingual corpora without supervision.

Published as a conference paper at ICLR 2026
PRIOR-BASED NOISY TEXT DATA FILTERING: FAST AND STRONG ALTERNATIVE FOR PERPLEXITY
Yeongbin Seo
Gayoung Kim
Jaehyung Kim
Jinyoung Yeo
Department of Artificial Intelligence
Yonsei University (suhcrates, kykim, jaehyungk, jinyeo] jinyea]@yonseil @yonsei. .cc.kr
ABSTRACT
As large language models (LLMs) are pretrained on massive web corpora, careful selection of data becomes essential to ensure effective and efficient learning. While perplexity (PPL)-based filtering has demonstrated strong performance, suffers from drawbacks: substantial time costs and inherent unreliability of the model when handling noisy or out-of-distribution samples. In this work, we propose a simple yet powerful alternative: prior-based data filtering method that estimates token priors using corpus-level level term frequency statistics, inspired by linguistic insights word roles and lexical density. Our approach filters documents based on the mean and standard deviation of token priors, serving as a fast proxy to PPL while requiring model inference. Despite its simplicity, the prior-based filter achieves the highest average performance across 20 downstream benchmarks, while reducing time cost by over 1000x compared to PPL-based filtering. We further demonstrate its applicability to symbolic languages such code and math, and its dynamic adaptability multilingual corpora without supervision. The code is available online (https: / //gittuuub com/ ybseo- cecooooacpprioooo ffriorrfflter)
INTRODUCTION
Large Language Models (LLMs) have achieved impressive performance by training on massive datasets, with web text serving primary data source. As web content continues to grow indefinitely. it offers unlimited data for pretraining. However, two major challenges necessitate careful filtering steps: (1) Web data is so large that we need to choose efficiently to save computational resources, and (2) It contains a lot of noise, which can harm the model not properly filtered.
To address this need, various data selection methods have been proposed. Early approaches relied on heuristic rules (Raffel al., 2023; Bane et al., 2022), but more recent trends have shifted toward model-based techniques (Xie et al., 2023b; Marion et al., 2023). These methods typically involve training a reference model on a target dataset and using it to identify desirable data. The model may perform binary classification (Xie et al., 2023a) compute similarity with the reference dataset (Xie et al., 2023b). Among these, using the perplexity (PPL) score from reference model as a criterion of filtering is currently known to offer the best performance while maintaining a relatively simple implementation (Ankner et al., 2024). We provide a more detailed review of related work $A.
However, PPL-based approaches come with the following inherent limitations. (1) Time cost: These methods require training reference model, followed by inference of PPL over the whole corpus. Given that web-scale data can easily exceed trillions of documents and continues grow, performing inference over the entire corpus becomes prohibitively expensive. (2) Reliability: LLMs often fail to accurately assess samples from distributions that is not seen while training, such noisy data. As result, generative perplexity may sometimes assign high scores to noisy or low-quality text (Holtzman al., 2020; Welleck al., 2019). This issue might become more pronounced when using smaller models to reduce inference costs, further undermining reliability.
"Co-corresponding authors

---

Published as a conference paper at ICLR 2026
To address this limitation of the PPL-based approach, introduce prior-based data filtering method grounded in linguistic insights. Instead of computing the full conditional probability of each token in the data estimating the prior term p(a:)) with statistical metric such as term-frequency. It is extremely simple and significantly faster (almost 0.1 % time consumption compared PPL--based), while it achieves even better performance on downstream task benchmarks.
oc p(z x))=(; (z; is token of data d), this method focuses solely on
Interestingly, this method is inspired by traditional techniques used in deciphering ancient languages. The 8th-century linguist Al-Kindi first proposed that, in order to decipher an encrypted language, analyzing the frequency of its words provides clue (Al-Kadit, 1992). If some word appears with the highest frequency across multiple documents, it is likely to correspond to function word, such as "is" "a" in English. This indicates that fre-frreyencc itself a one-dimenll representation for the role of a word: high frequency maps to function words while relatively low frequency maps to content words (e-g.. "US", "president"). Combining with another linguistic observation that well-formed sentences within a language tend to exhibit consistent level of lexical density (i.e., ratio between function and content words) (Johansson, 2008), we can determine outlier document simply by computing the mean and variance of its term frequencies: which we term prior-based data filter.
The prior- filter exhibits intriguing and practical properties, which we demonstrate empirically, (1) The linguistic principles underlying the term-freee ten-freuuency hold not only for English but also for other natural languages (e.g., Chinese and French), even for symbolic languages (e.g., code, mathematics). (2) Only small amount of Chinese text data mixed into English corpus may be noise and models can not learn patterns from it; however, as its amount increases, it becomes learnable by models. The prior-based filter is capable of automatically capturing this transition of learnability.
We demonstrate that models pretrained using the prior-based filter outperform models using the PPL-based filter, across 20 diverse downstream task benchmarks. Moreover, since token priors can be estimated from a relatively small corpus, the prior-based filter is approximately 1000 times faster, requiring only 0.25 hours compared to 216 GPU hours for PPL-based filtering on 6B-token corpus.
Our contributions are as follows:
We propose the prior-based filter as an approximate alternative to the PPL--based filter.
We analyze the useful properties of the prior-based filter, including its efficiency and generalizability.
Through extensive downstream benchmarks, we demonstrate that the prior- based filter is not only faster, but also outperforms the current state-of-the-arr PPL-based filtering.
2 PRIOR IS A ONE-DIMENSIONAL REPRESENTATION FOR THE ROLE OF TOKEN
In this section, we first introduce PPL-based approach, which is the previous SOTA for data filtering. Then we define how to estimate the prior, a key component of PPL. We then analyze the linguistic properties and significance of the prior, to show its potential as an effective criterion for data filtering.
2.1 PPL-BASED APPROACH AND ESTIMATION OF PRIOR
The PPL-based filtering method is known as the most effective approach for filtering noise data from web text corpus for pretraining LLMs (Ankner et al., 2024; Marion et al., 2023). For the filtering, first, a small reference model (an autoregressive transformer architecture on the corpus D. The model then computes the PPL for each data point d = (I1, 22,... .NN) ), where I; is the token at the 7th position of a document, and d E D. Then, d with PPL values farthest from the median are discarded. Here, the PPL is defined as follows:
137M parameters) is trained
N
PPL(d) = II po
Po(x;; IxZs) is the conditional probability of token 2; given its preceding context I'i under the model 6, that can be decomposed into likelihood and prior as follows.
po (x; (ies)) x po( 2x4) po (m:)
2

---

Published as a conference paper at ICLR 2026
In this Bayesian formulation, the likelihood term po(zee z;)) captures the dependency between the token Fy and its preceding context Z<ú, indicating how well the token aligns with the surrounding text. In contrast, the prior term po(z;) represents the marginal probability of the token 2;, independent of its context.
Estimation of prior Due to the independent property of prior, it is no longer necessary for transformer model to learn the joint probability in order to estimate the prior. Therefore, in this work, we assume the prior pe (a) of token approximated by simple statistics (i.e., term-frequency) in
corpus D, estimated as follows: Pp)ioe(z) = Ev'ev (D()") Here, fD( is the number of occurrences of token in corpus D, V is the vocabulary set.
fo(z)
2.2 FREQUENCY ANALYSIS IN LINGUISTICS
To justify the use of token prior filtering criterion, draw on linguistic insights that reveal its strong connection to lexical and syntactic structure. Linguistics offers two key insights related to term frequency, and by combining them, can derive its potential utility as a data filtering criterion.
(1) Term frequency is a 1-dimensional representation of a word's role: The 8th-century linguist Al-Kindi first proposed an idea that is still widely used today (Al-Kadit, 1992): to decipher ancient or encrypted languages, analyzing the frequeney of its words gives clue. If some word appears with the highest frequency across multiple documents, it is likely correspond (e.g., "is" or "a" in English) that serves grammatical roles. In contrast, content words which carry semantic meaning (e.g.. "US", "president") tend to appear with relatively lower frequency. Therefore, frequency itself can serve as a basis for distinguishing between function words and content words. In other words, term frequency (i.e., prior) can be seen as one-dimensional representation a word's functional role. We analyze that this property partially stems from the next property.
function word
(2) Well-formed sentences typically exhibit a consistent range of lexical density: As lexical density is defined as the proportion of content words against function words, it is known that well- formed sentences in a language typically maintain a certain range of lexical density (Johansson, 2008). From this, we can infer that broken and ill-formed sentences will deviate significantly from this range to be outliers.
By combining these two properties, First, we use the token prior as a one-dimensional representation to estimate whether each token functions more like content or function word. Then, by assessing the overall composition of function and content words, we can determine whether the document an outlier.
derive principle for identifying ill-formed documents.
PRIOR-BASED DATA FILTERING
In this section, we present an explanation and analysis of the prior-based data filtering method. (1) We first analyze the token-level term frequencies, demonstrating that linguistic insights are applicable at the token level. (2) We then apply this principle to build our filtering method. (3) Lastly, we validate its feasibility by analyzing data samples filtered by our approach.
In SC, we further provide a theoretical and empirical explanation that the prior-based filter generalizes across diverse types of languages and tokenizers.
3.1 ANALYSIS ON THE TOKEN PRIOR
We first analyze the token-level term frequencies by calculating the token priors (with formulation in 12.1) on the Dolma dataset (Soldaini et al., 2024). As we sort them by the logarithm (Figure 1), we observe that the token priors distinctly fall into three clusters based on their height and slope, supporting the thesis that the token prior serves as a I-dimensional representation for the token't role.
The three clusters in Figure are follows. (1) High-prior zone: steep slope of high-prior tokens. We can observe that this zone mainly consists of function words (e.g., "the", "a", "is", "you"".. (2) Middle-prior zone: As the priors in this zone have similar range, they form a wide and gentle slope. This zone seems to mainly contain tokens for content words (e.g.. "Phone", "shortcut", "tackles", "doorstep"), ) (3) Low-prior zone: The frequency is extremely low, and the slope becomes steep again. This region is primarily composed of accidental noise tokens (e.g.. "_=". "ÄБÄЛλΔÄЛÄЙ"
3

---

Published as a conference paper at ICLR 2026
High (furction words)
Midklee priur (content words)
Crevtiin
and
423
Jaciss
Phone
_utroge
.0corstep
pd
gration
-10
DM
irlernce
Authent
-15
./b _loughter
Chester
.90
aharmacles
3G
Low
(roise)
Dont
.9is
Jave
-30
rawiltim codroo
Smartstecks
300
Rank of Takians
Figure 1: The line graph shows the logarithm of token priors (based on the GPT-2 tokenizer) computed from the Dolma dataset, sorted descending order. The boxed regions highlight tokens from the top, middle, and bottom segments of the rank.
"GaHuafia"") including tokens from other language types that appear only a few times in the data (e.g., Chinese in English corpus).
3.2 FORMULATION OF PRIOR BASED DATA FILTERING
We established two premises in $2.2: (1) A token's prior serves representation of its functional role, distinguishing function words from content words. (2) In a given language, a well-formed document typically maintains average level of lexical density. By assessing the overall composition of function and content words, we can determine whether the document is an outlier.
However, since the prior is a continuous value rather than a discrete class label, cannot directly compute the lexical ratio to assess the composition. Instead, we propose two alternative indicators to approximate the composition: the mean and standard deviation of token priors within a document.
Since well-formed documents are clustered around a certain range of lexical density, the mean of token priors within such documents should also cluster around a certain value. (2) Prior standard deviation: Given that well-formed documents tend to exhibit a stable lexical density, the variance (or standard deviation) of token priors within a document should also cluster around a specific value. We denote these metrics as P4 and σa respectively, formulating as follows. Specifically, we define the prior mean with a logarithmic transformation, as aligns with the prior term in the PPL formulation; this is discussed in more detail in $3.4.1:
(1) Prior mean:
Pa = Ez, [log Pprior (2:)), оa = stdg,ea [x:)) d D
As assume that both μ사 and da of well-formed document clustered around certain central value, we define this central value as the median over the corpus D: Mp medianag p (Pa), Ma medianae p (Fa). The distance from the median is then used measure of outlierness. (d) |pa M & (d) = Jaa - M, To perform filtering, we discard the samples with the large 5. The discarded portion is defined as the filtered set Fo.
We analyze that the two criteria capture different aspects of the data. While óy captures the compo- sition of tokens document, reflecting whether the document predominantly consists of high or low prior tokens, 8, reflects the distributional structure among tokens, indicating how uniformly or diversely the token priors are spread. This difference is also observed in the outlier samples.
In $B, we provide a theoretical derivation of the noise-detection behavior induced by μa and a and show that approximates that PPI--based filter.
3.3 OBSERVATION ON DISTRIBUTION AND OUTLIER SAMPLES OF μla AND da
We check whether the values of μl and σa clustered around central points, as hypothesized. For this, we randomly sample 600K examples from the Dolma dataset and compute μa and оa for each d. As shown in Figure 2, both values exhibit broad distributions centered around their respective
4

---

Published as a conference paper at ICLR 2026
medians, with relatively small deviations. Notably, beyond a certain threshold, observe sharp increases in deviation, forming clear outlier regions (highlighted by red dashed circles). Upon inspecting these outlier samples, we find that they primarily consist of noisy documents lacking meaningful information (boxes of Figure 2).
Figure 2:
The line graph displays the values of pa and a computed from token priors in the Dolma dataset, sorted in descending order. Boxes are outlier samples from both distributions.
(a) pA with outlier samples
5
Vin
op doohue Jedll e get Oud tetee Ligh way
-14
big wholl all doe's dand saa 2-3 dlial Jooos will dites Juigl ie wast dais Jog
spsbell gai and Sall
02005003
DO0000
O200
-28
(b) with outlier samples
losl Singer Duff Jost Stnger / loet Singer los Singer of isel sinper 7 losts near &l Jost sin owr wf Jeel tinger of Joel Singer // loet ngerroal Singer / lsel Singer toel Singe's by loel Eingerinel Singer f/ Joe nger Singer Bul lnel Singer
1205 [15a] Cher Ignaiyee M. then Bay 8 5t.
Technol Candini T.T.S sthem 2011
2001. A (832) 110 G0t Marquardt Dit (8kg] D. (75b
Hassol M. zo11 Q. A. 385. Warg 2007. 2003. Brycu.
2011 Knuthof K. Bankovan 2001 yon de Vever R22 Grewn Crem cam Verromal
R.S.A 4 Sce Sume Sodesaws.5 5 zhu, Bicker
Eir ger Jasl Sirgee
Strger Strge
Sreper
singe
Singe Singor
Botchscin12 [Four of Pentades) Fruga. The Economical Finoncall only co necrattive Milionaare mind liniding firm Teacitionall Eomvent onal faxings
Inspinng work morms vn. Medicen So-sa Shodds
SINH AMMATUL SATIAA AFAO AHRADD PRIVA KESHARI
BARANINAL
RKM KESHARE DARANSAI CHANDARSI
RESKRMIAMI BI KESANWANI
KA SINGH PRASAD
Acxrus. fremible Choape Aneria Gready Hoard
Stockpil Scrimp, Misarly, Desefisted Stings Scrooge Tighnwad Penny pincher Crespotone'yt
KANT RASHMI UPRET DINESH CHANDRA
Characteristics of outliers from each metric We observe that the outliers for fáa and aa exhibit different characteristics. In the case Ad. the outliers tend to consist of tokens with either extremely high or extremely low prior values. For example, on the extreme-high side of the μua (left boxes of Figure 2a), documents mainly consist of line breaks ( ("In') space characters (* "), which is one of the tokens with the highest prior. On the extreme-low side (right boxes of Figure 2a), documents are often filled with non-English language or special characters.
Conversely, in the case of da many outlier documents contain content-word tokens with middle-prior (boxes of Figure 2h). However, these words arranged in unstructured ways, often appearing as list of nouns without sentence structure. These differences arise because the μ4 reflects the average composition of tokens in a document, whereas the aa captures the distributional pattern of those tokens. This suggests that both values should be used together for more effective data selection.
3.4
PROPERTIES OF PRIOR-BASED FILTER
3.4.1
PRIOR-BASER FILTER APPROXIMATES PPL-BASED FILTER
The prior-based filter serves as an approximation to the PPL-based filter. We support this claim through both formulation analysis and statistical comparison of filtered data overlap.
log (d) Σ toggo(Z<)
Σ log (x)))
Tiikelskood
Tprior
First, the logarithmic form of PPL reveals that both the Pa and da express two components of the PPL. (1) xprior: The formulation of μa in Equation 3 is exactly equivalent to the Tprior- (2) Tiskelihood: as Tiskelihood captures the regularity of relationships among tokens within document, da similarly reflects the regularity in distribution of token priors. This suggests that the two measures are weakly aligned. Taken together, combining da and aa can serve reasonable proxy for perplexity.
5

---

Published as a conference paper at ICLR 2026
Prior can be even better metric than PPL While da captures approximation of the likelihood term, it is significantly more saturated than the actual likelihood, which can be considered limitation. However, conversely, the inherent instability of the Tttkeiooood (described follows) poses limitation for the PPI.-based approach. (1) When the model is small, it struggles to accurately learn the likelihood (Wei al., 2022). (2) The model does not learn how to estimate likelihood for data from previously unseen distributions (mostly noisy data), which is not a problem for estimating only the prior. For this reason, previous studies have also reported that PPL often mistake repetitive or pattern-basel noise as valid text (Holtzman et al., 2020). Empirically, the model trained with the prior-based filter shows better downstream performance than the one trained with the PPL-based filter ($4).
Figure 3: Extreme outlier samples selected based on three criteria, ensuring that each sample comes from a distinct criterion: PPL., 1dd, and dd- v /indicates filtered out.
(5)
x
PPL X
Prior
Prioe std,
optimum bost encuring fiehting dasien didirg
E5 2nd vag 45 Heufiior aig wfges APG awerrp feory edhi stolii zu Revvs Rig lale wala 7e a wrke int twird 5 fag exforre l'waefz a fay muree mnuitt go yefs ean
Itaily life Ordinary, Common Regular Acarays. Normal Unus.
pocket. encures tough est weight bleyds pocket moterilTHY
siby arlow in. tastfigg Reey
Leathe Turtle Tortako Mundane,
COLLECTIBE IBLE Worden bexl handmade Storage Natural Roon
Plodding Lazy Sedentary Sugpps Lazybones Couch potati Guiland Drans
Woodoouder local oven 17 inta butter charge Bake use
Monatany Slab.
Matcho Baking dingnese ree "egulor pressure open beautiful Sugar
web nist dit affow inpetwwerfs off yress the yonr infer
Stovenalyne (Pagt of Pantaces) Leaming Studying
freeht out weight stop help providies what ensures world Our locat wire up
Erucation Student Pipl. chesroom School College. University Campus Darn tory Semeettr Collegjate Scholar-
rich calories AND
In ŞB, we provide a theoretical explanation of how the prior-based based filter approximates the noise- detection behavior of the PPL-based filter, well the boundary conditions where discrepancy between the two emerges. We also explain why median- based thresholding is robust.
Observation on filtered samples This characteristic of PPL is also observed in outlier samples. We investigate the most extreme outlier samples from each metric (PPL, Ma. oa), excluding their overlaps (Figure 3). As described in $3.3, outliers of μ4 tend to be filled with extremely low or high prior tokens (Figure 3b), while those of da often consist of content words but lack function words or valid sentence structure (Figure 3c). In outliers of PPL (Figure 3a), content and function words appear to be well-balanced, giving the surface impression of well-formed sentences, but upon eloser reading, many of them turn out to be semantically meaningless. This may reveal both a strength and a weakness of the PPL metrie: it effectively captures subtle irregularities within well-formed documents, but may fail to detect noise arising from entirely out-o--distribution samples.
Statistical comparison To demonstrate that prior-based filtering approximates the PPL-based filter, we measure the overlap ratio of data filtered by each metric. We first randomly sample 600K examples from the Dolma dataset. Then, for each value (Ma, da PPL.,, we extract the data points whose percentile rank falls within the top or bottom 5% (Figure 4). These are denoted as the filtered sets Fu. Fo, and Fppt. respectively. For each filtered set, we compute the overlap ratio with Fpp. defined as |FDFF
0.5
Portion of intersaccion
0.3
0.1
prior stds
random
0.01
0.05
0.10
0.20
0.30
Portion of filtering e/2)
Fggl
Figure 4: Overlap between outliers based on μı and a with those based on PPL, when filtering the top and bottom a % of samples (X-axis: e).
The results show strong correlation: When filtering by the e = 0010, nearly 50% of F2 and Frpt overlap. We also find Fj. aligns more closely with Fepl than F,. We provide additional evidence in $E.2
3.4.2 Pa REFLECTS LEARNABILITY OF MINOR LANGUAGE IN MULTI-LINGUAL SETTING
The prior mean value has the property of dynamically reflecting the learnability of data cluster (i.e., language type), especially when multiple clusters with distinct characteristics are mixed. For example, consider a corpus primarily composed of English data with small portion of Chinese data included. While the Chinese samples may contain meaningful content, if their quantity is too small, the model will fail to learn the pattern of language. In this case, Chinese data is no more than noise. However, once the volume of Chinese data increases sufficiently, the model becomes able to interpret the language, making it learnable and meaningful data.

---

Published as a conference paper at ICLR 2026
The prior-based filter captures this dynamic behavior without any special tuning. As shown in Figure 3, prior mean values tend to classify non-English samples as noise when they are sparsely mixed into English data. However, when the proportion of such data exceeds a certain threshold, the filter begins to treat them as valid language rather than noise.
To demonstrate this, add Chinese dataset (Wiki-ch) to an English corpus (Dolma), with the Chinese data scaled a% of the English corpus size. We then measure the percentage of added Chinese samples that fall into the
Proportion of oudiers (10%)
outlier set (percentile rank falls within the top or bottom 10%). As shown in Figure 5, when the size of the Chinese data only 1% relative to the English data, nearly all of is classified as noise. However, once its proportion exceeds 20%, the rate of being classified as outliers drops to level comparable to random filtering (10%, indicated by the red dashed line).
0.6
0.4
3.
22
90
100
This characteristic offers major advantage over methods that require manually specifying a reference dataset (e.g.. DSIR (Xie et al., 2023b)). In DSIR, human must decide whether to select English or Chinese data and then pro- vide suitable reference dataset accordingly. In contrast, the prior-based filter automatically determines whether language should be filtered out based its learnability.
Size of data compared to (3)
Figure 5: Proportion of Chinese data classified as outliers (Y-axis), after mix- ing Chinese and English data at a ratio of a : 100 (a as X-axis). Outliers are the top and bottom 5% of Ma.
3.4.3 FAST, SCALABLE FILTERING USING SUBSAMPLED PRIORS
14:
One of the key advantages of the prior-based filter model-based methods lies in its efficiency, Given the mas- sive volume of new web data, which rapidly grows daily, training and inferring PPL value with reference model can significantly amplify the time cost of filtering. In con- trast, the prior-based filter only requires computing term frequencies and then calculating the mean and standard deviation of the priors.
Outleer overap
Minubos
0.3
0.88
Mixute
130
23.0
502
Semple
Remarkably, the already minimal computation time of the prior-based filter can be further reduced. For 6B- token corpus, the entire process takes about 35 minutes on 40 CPUs (Intel Xeon Silver 4210R @ 2.40GHz), which consists of two stages: assessing the token prior, and com- puting Paa and da. Among these, the most time-consuming step is the token prior assessing phase, which alone takes around 30 minutes.
Figure 6: When token prior is computed with 8% subset of Dolma (X-axis is b), the proportion of outliers overlapping with those from b 100 is the left Y-axis. The right Y-axis shows the com- putation time (in minutes) required calculate the token prior at each b.
This assessment time can be significantly reduced, as term-frequency estimates remain highly consistent even when calculated from a small subset of the data. To verify this, we sample 6% of 6B-token dataset to compute the token prior and then measure how much the resulting outlier set (top/bottom 10%) overlaps with the outlier set derived from the full corpus (b 100). As shown in Figure 6, even with just b = 1%, the extracted outliers are nearly identical to those from full corpus; requiring only about 70 seconds, or roughly one minute.
Table Dolma v1.6 com- position and its proportions based on token count.
4 EXPERIMENT ON DOWNSTREAM TASK
In this section, we evaluate the downstream task performance of models pretrained with different data filtering methods. Most training settings and hyperparameters follow those of "Perplexed by perplexity: Perplexity-base data pruning with small reference model (Ankner et al., 2024)".. We first conduct experiments on a natural language
Stnne
Dxcuncnt
With Pages
13.95
Hanks
mada
2.9%
(i5201
odicaiceal page
2.5%
Project Canrabery
encyylopeske

---

Published as a conference paper at ICLR 2026
(specified to English) web corpus, Dolma (Soldaini et al., 2024). This allows us assess the effects on general language capabilities of a model (e.g., knowledge, language understanding. and symbolic understanding). To demonstrate that our prior-based method is applicable even to symbolic languages such as code and math, we also perform experiments on the Pile-github² dataset.
4.1 EXPERIMENT ON NATURAL LANGUAGE CORPUS AND GENERAL ABILITY
Corpus setup Following Ankner et al. (2024), we mainly use Dolma (Soldaini et al., 2024) as pretraining corpus for a testbed of filtering methods. Dolma a large-scale, diverse web-text corpus, designed for training and evaluating LLMs. It contains noisy web data sources that support general language use ability, such as world knowledge, commonsense reasoning. and symbolic problem solving. This corpus is composed of multiple web-scale datasets, including Common Crawl, Reddit, Wikipedia, and Wikibooks', The Stack (Kocetkov et al., 2022), C4 (Raffel et al., 2023), PeS2o (Soldaini & Lo, 2023). Project Gutenberg (Angelescu, 2013) (see Table 1). Among these, Common Crawl accounts for the major portion (74.5%) of the corpus. This makes it particularly suitable environment for evaluating filtering methods, it contains high proportion of noisy web content that must be thoroughly filtered, while a small but valuable subset (e.g.. books and educational data) must be preserved. For testing under resource constraints, we select v1.6 smaller subset with 6.3B tokens. We divide this into blocks (d) of 512 tokens, and select a subset of N tokens for pretraining.
Baseline setup When selecting a subset from Dolma, we follow the procedure defined by each method: (1) no-filter: Randomly selects N without applying any filtering method. (2) PPL-based: Following the approach of Ankner et al. (2024) and $2.1, we first train a reference model (137M) on the random 3B tokens subset of dataset. We then compute the PPL score for each sample in the dataset. To obtain a final subset of size N, we discard samples with the highest and lowest PPL. scores. (3) DSIR: Adopting the well-known method DSIR (Xie et al., 2023b), we estimate n-gram frequency from the reference dataset (we choose Bookcorpus and Wiki-en) and compute importance weights. DSIR is proposed an advancement of FastText, a classifier-based method using manual data curation. (4) prior-based based (ours): As described in $3.2, we first estimate token priors using a 10% subset of the full corpus. Based
these priors, we compute pu and aa (d D). We then discard samples with the highest 5n and 6 values in the constraint of Ful = |Fa] until the volume of final subset FF, U Fz| reaches N. For all baselines, N is set to 50% (3B), which is observed in Ankner et al. (2024) to yield the best performance.
We use the GPT-2 architecture for pretraining, with large (1.5B) and small (137M) size models, using 8 GPUs (RTX A5000). Following Ankner et al. (2024), we set a max token length of 512, global batch size 256, and learning rate 2e-4, and train for 40K global steps (about 6B token duration). According to Ankner et al. (2024), the relative performance trends observed at 40K steps are maintained in later training steps.
As our study requires a significant amount of resources for pre-training across multiple baselines, we carefully adopted settings and observations from recent representative works to ensure experimental rigor under limited resources. We explain how our baseline lineup comprehensively covers the major streams of previous works SA.2. We also provide justification of our experimental scale (e.g.. training token duration, model size) in $F.2.
Benchmark and evaluation setup The types and settings of downstream tasks follow those used in the Ankner et al. (2024), based on the MosaicML evaluation gauntlet (MosaicML, 2023). Gauntlet includes tasks designed to assess five core capabilities: world knowledge, common sense reasoning, language understanding, symbolic problem solving, and reading comprchension. We normalize the accuracy of the individual task as a, = am -a. where am is the accuracy of the model and a, is the expected accuracy of random guessing. We report the average normalized accuracy for each task, task category, and the average across all categories. Since some tasks are not proper for 1.5B models, we exclude benchmarks with average on of baselines under 0.001. This results in a total of 20 benchmarks (details in $G)
1-a,
As described in Table 2, the results show that the model trained with prisr-based filtering achieves the highest average performance, with extremely small time cost. Key observations are as follows: (1) DSIR outperforms no-filter, and PPL-based outperforms DSIR, which aligns with
Results
https//commmonrrawlorr// https//wwww.rcddi..com https://dumps.wikimgg wikimedia.or/
8

---

Published as a conference paper at ICLR 2026
findings from previous research (Ankner al., 2024; Xie et al., 2023b). (2) Prior-based filter approximates PPL-based filter in principle, but yields better downstream performance. We analyze that this is because PPL score depends on the model's likelihood, which can be unstable. On the other hand, the prior is based on simple word frequencies, so it gives a more stable and reliable signal. (3) Though the prior-based model outperforms the PPI-based model in downstream performance, the prior-based filtering requires significantly less processing time. PPL-based filtering takes 216 GPU hours to select a 3B token subset (20 x 8 GPU hours of training the reference model, 8 GPU hours of PPL inference), while prior-based filtering takes only 15 minutes (6 minutes of assessing token prior, 6 minutes of calculating Ma and 0g in D) under 0. % of the time spent for PPL. This demonstrates the superior scalability and efficiency of our approach.
(4) In symbolic problem solving, PPL- based filtering performs the worst, whereas prior-based filtering per- forms competitively with other base- lines. This suggests that PPL fails to capture small and meaningful seg- ments of different types of data, while prior-based filtering is more robust in preserving them. This is due to the property of Ha that reflects the learnability of multiple language types ($3.4.2). (5) While no-filter performs poorly across most abilities, it shows the highest score in symbolic problem solving. This might be because small but meaningful portions of data (e.g.. math or programming tially filtered out in other methods, but retained in the no-filter. For a prior-based filter, this issue be handled by augmenting the small subset of the corpus for the targeted data type (i.e., datasets
Table Performance and time cost (for filtering) of the baselines pre-trained on Dolma across 20 benchmarks. The average normalized accuracy is the average of all categories. Avg: Avg normalized accuracy, W: World knowledge, C: Commonsense reasoning, L: Language understanding, S: Symbolic problem solving, R: Reading comprehension
Type
Time
W
I.
1.5B model
vanilla
mil-hased classiferr
5.52
0.44
6.14
3,6
a-grum model--haod (ouns)
7.00
6.84
12.62
9.98
11.91
7.34
7:91
Prior based
0.25 hrs
9.30
9.53
11.27
10.31
11.13
3.79
137M madell
vanilla
rule based classtteer
4.96
4.96
1.81
1.47
12.83
3.70
Pastleut
3.6 has
5.39
5.12
4.29
1.74
12.31
3.49
DSSR
4 has
5.60
5.68
4.93
1.97
11.60
3,80
PPL-hesed
model-hased
216 GPU
Prior-based
(ours)
par-
focused on coding or mathematics). This adjustment is straightforward and incurs minimal effort. (6) Across other skill categories, the prior-based method consistently outperforms other baselines or performs comparably to the best-performing one, resulting in the highest overall performance. (7) This trend remains consistent even for different-sized models.
Further analysis As data filtering removes noisy data, which is often found in outliers, it inherently risks discarding valuable minority data (e.g., highly technical paper with rare terminology, corpus including minority languages) as trade-off. In $D, we analyze the robustness of the prior-based filter with respect to this trade-off.
In SC, we further provide a theoretical and empirical explanation that the prior- based filter generalizes across diverse types of languages and tokenizers. We also provide analysis on the sensitivity to factors such as block size and threshold in ŠE.
4.1.1 CONSISTENCY IN LARGE-SCAAL SETTING
We conduct experiments larger models and datasets. For the efficiency of massive experiments, we adopt the strongest baselines from previous experiments (PPL-based and 3B (Qwen2. 5- 3B) and 1.5B models with training up to 12B token durations (2 x repetitions). The results consistently demonstrate the superiority of the prior-based filter (Figure 7).
We evaluate
4.2 EXPERIMENT ON SYMBOLIC LANGUAGE CORPUS
We retain most of the settings from experiments of $4.1, including baselines and training configura- tions, but change the pretraining corpus to Pile-github. From the subset of 6B tokens, we extract subset of 3B tokens with each filtering method. We exclude DSIR due to the difficulty of determining an appropriate reference dataset for Pile-github. This is also a critical limitation of the DSIR.
Pile-github mainly consists of code scripts, additionally containing a little mathematical data and natural language data. As it contains little information related to general language skills, such as world knowledge, we limit the evaluation only to symbolic problem-solving benchmarks in gauntlet.

---

Published as a conference paper at ICLR 2026
18
14
Prior-based PPL-based
14
10
Avg ACC
Avg AcC
10
0.75
1.5
3
12
0.75
1.5
3
12
Token Duration (B)
Token Duration (B)
(a) 3B model
(b) 1.5B model
Figure 7: Average normalized accuracy across 20 benchmarks.
Results The observed results are follows: (1) Consistent with the previous experiments, the prior-based method achieves the best performance with signifi- cantly less time than the PPL- based approach. (2) These find- ings suggest that our methods hold not only for natural lan- guages (e.g., English, Chinese) but also for artificial symbolic languages ge.g., code, math). This means that well-formed data in certain language type can be identified via based statistics, regardless of language type. (3) Math-relate benchmarks (BIG- bench elementary math QA, GSM8K, SVAMP) exhibit near-random performance across all baselines, likely because the Pile-github dataset consists predominantly of code scripts.
Table 3: Performance of the baselines pre-trained on Pile-github across 6 symbolic problem-solving benchmarks. CS: BIG-bench algorithms, Dyck: BIG-bench dyck languages, Op: BIG-bench operators, Ma: BIG-bench elementary math QA, GSM: GSM8k, SVA: SVAMP
Time
CS
Dyck
Ma
GSM
SVA
ne-filler
9.51
35.75
12.30
5.71
1.15
0.15
2,00
PPL-hased
224 GPU hours
11.21
37.42
20.60
7.14
2.09
0.00
0.00
(ouns)
1.17
137M model
no-filler
10.15
37.87
16.30
5.23
1.52
0.00
0.00
PPE-based
Prior-based (ours)
0.26 hours
12.19
10.22
16.00
7.14
3,08
0.00
6.66
CONCLUSION AND LIMITATION
We proposed prior-based text data filtering method grounded in linguistic insight. The prior- based filter serves as an approximation of PPL-based methods, while achieving superior downstream performance and being over 1000x faster. Furthermore, shows strong generalizability by performing effectively even on symbolic languages. This enables efficient filtering of rapidly growing web text data and provides foundation for faster continual pretraining of LLMs.
However, since this method leverages linguistic properties, unlike other approaches such as PPL-based filtering or DSIR, it is less suited for extension other modalities such as image data.
ACKNOWLEDGEMENT
This research was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSITT) under Al Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data(2022-0-0007) RS- 2022-11220077, 50%). ITRC(Inoommation Technology Research Center) support program (ogram(1ITP-2025: RS-2024-00437102, 30%), Artificial Intelligence Graduate School Program (Yonsei University) 2020-11201361, 20%).. Jinyoung Yeo and Jaehyung Kim are the co-corresponding authors.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2333.095400 2023.
Ibrahim Kadit. Origins of cryptology: The arab contributions. Cryptologia, 16(2).7--22) 1992.
Radu Angelescu. GutenbergPy. sion 0.3.5, accessed August 2023.
com/ http:////.....a..cooo////oaaaaa/
2013. Ver-
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-bon based data pruning with small reference models, 2024. URL https: / ///axiv..oo org/ abs/ / 2405. 20541.
Fred Bane, Celia Soler Uguet, Wiktor Stribižew, and Anna Zaretskaya. A comparison data filtering methods for neural machine translation. In Janice Campbell, Stephen Larocca, Jay Marciano, Konstantin Savenkov, and Alex Yanishevsky (eds.), Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pp. 313-325, Orlando, USA, September 2022. Association for Machine Translation in the Americas. URL https: //aclanthoi ogy- wt..rrggg...... 2022. amt upg.aaup..22.... 22/1.
Adrien Barbaresi. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 122-131, 2021.
Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic chatnoir: Search engine for the clueweb and the common crawl. In European conference on information retrieval, pp. 820-824. Springer, 2018.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models, 2023. URL. https: /aaxiiivvv... www.o/////aooo abs//bss/// /204 11158.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Fhirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuni Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107. 033744, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.055. 2018.
Harlan Cleveland. Information as a resource. Futurist, 16(6):34-3, 1982.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https:ttt//ee /ttp://aaavvvv/aa///a//eae/aaaaa org/ wttp:bs//gg///ooo//e//eegggggaaa 12220.11111661
William Croft. Typology and universals. Cambridge university press, 2002.
Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Climb: Clustering-basee iterative data mixture bootstrapping for language model pre-training. arXiv e-prints, pp. arXiv-2504, 2025.
Jesse Dodge, Maarten Sap, Ana Marasovié, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Dncumenting large wehtext corpora: A case study on the colossal clean crawled corpus, 2021. URL https::tp///cc...t. aaaiiv v.w.vvvva..o www./bs./////ooooooooo....o... 2.04. 4.88888
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020. URL https: / axxiv wow.oor/aaa......... 19044 09751.
Jue Hou, Anisia Katinskaia, Anh-Duc Vu, and Roman Yangarber. Effects of sub-word segmentation on performance of transformer language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 7413-7425, 2023.
Aditi Jha, Sam Havens, Jeremy Dohmann, Alex Trott, and Jacoh Portes. Limit: Less is more for instruction tuning across evaluation paradigms. arXiv preprint arXiv:2311.13133 2023.
11

---

Published as a conference paper at ICLR 2026
Victoria Johansson. Lexical diversity and lexical density in speech and writing: A developmental perspective. Working papers/Land University, Department of Linguistics and Phonetics, 53:61-79, 2008.
Mandar Joshi, Eunsol Choi, Damiel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distandly supervised challenge dataset for reading comprehension, 2017. URL https: / aariii org owg/abs/////ooooooe////e//ee 1aabss/1// 1705. 03551.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classifica- tion. arXiv preprint ar arviv: /607,0179, 2016.
Denis Kocetkov, Raymond Li. Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: tb of permissively licensed source code, 2022. URL https: aarxiv..iv...... .rgg /
2211. 15533.
Julia Kreutzer. Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch. Nasanbayar Ulzüi Orshikh, Allahsera Tapo, Nishant Subramani. Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios. Isabel Papadimitriou. Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidhek Mirzakhalov, Tapiwanashc Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Diamini, Nisansa de Silva, Sakine Çabuk Ball, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics. 10:50-72, 2022. ISSN 2307-387X. doi: 10.1162/tacl 0044477. URL http: //dx. /rg. 10. 1162/116//2 wtw///eeeeeeee a_ 00447.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012.
Jeffrey Li, Alex Fang, Georgios Smymiss Maor Ivgi. Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Kch, Kushal Arora, et al. Datacomp-Im, In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200-11282, 2024a.
Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, and Houwen Peng. Scalingfilter: Assessing data quality through inverse utilization of scaling laws. In Proceedings of the 2024 Conference on Empirical Methods in Natural Longuage Processing, pp. 3209- 3222, 2024b.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, al. A pretrainer's guide training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long 2024.
Papers), pp.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadace, and Sara Hooker. When less more: Investigating data pruning for pretraining Ilms at scale, 2023. URL https: http:///////vvvv/ wwtp://wwwwgg//////////aa/s///// abs /2399 04564.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arX0292410.05229 2024.
MosaicML. LLM Evaluation Scores. 2023. URL httttps://www.nosstte com/ 11m evaluation 2023a.
Binh-Nguyen Nguyen and Yang He. Swift cross-dataset pruning: Enhancing fine-tuning efficiency in natural /2501 02432.
language understanding, 2025. URL https: http:/::///aaaaaaaaaeeeaaa
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL, https: / ///x//axvvv//////////a////aaaaaa v...iivvvv org/ abs/ 6.660000.0
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080 2094, Online, June 2021. Association for Computational Linguisties. doi: 10. 10.18865311/000 maacl-main 168. URL. https: / /acl hot ogy. org/2021 naacl -mai 168.
12

---

Published as a conference paper at ICLR 2026
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning a data diet: Finding important examples early in training, 2023. URL aattps://axiia .rrg org/abs/////ooooooooo...o.....os 2107. 07075.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01100 2023.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811 1008811--049,,- 2024.
Jack W Rac, Sebastian Borgeaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1 11446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Ii, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, (1(140):1-6) 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
URL https:/// /arxiv.iiv...////a//...////aaava wwwoorggaaaooo /aabs//1 1910.11110 10683.
Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge, 2019. URL https: /arxii v. oorg/////ssss......o aaaaab///18000 C7042.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pp. 90 90-99,,22 2011.
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, E Chi, James Caverlee. Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient Ilms, arXiv preprint 240v:2400.09666 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021.
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset. https://githuts al allenai/pes2e 2023.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus three trillion tokens for language model pretraining research, 2024. URL https: / azxiv org/ abs / 2402. 00159.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoch, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, and Alethea Power and. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL https: // /aaxivv..... www.v.org/aa..oaaoo......twwotwe at///////2 22006.2022222 046155.
Karin Stromswold. The nature and acquisition of lexical and functional categories. ms., Rutgers University, New Brunswick, New Jersey, 1994.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768. 2022.
Scan Welleck, Ila Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, 2019. URL
/1908. 04319.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1819-1863 2024.
13

---

Published as a conference paper at ICLR 2026
Sang Michael Xie, Hiuu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V. Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. 2023a. URL attpp::/arxxvv.rgg/ abs 2305.10429.
Sang Michael Xie. Shibani Santurkar, Tengyu Ma, and Perey Liang. Data selection language models importance resampling, 2023b. URL https: : http//aw.v.oggg........o aat.////////////a////// 2302. 03169.
Rowan Zellers, Ari Holtzman, Yonatan Bisk. Farhadi, Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centrie benchmark for evaluating foundation models, 2023. URL https: //wxivv orggabs//300 06364.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
In this section, we first review two streams (c.g.. rule-based and model-based) of previous works on web text data filtering for the pretraining of L.LMs. Next, we explain that the baselines employed in our experimental setup cover all categories of existing baselines. Finally, we provide a more in in-depth explanation of previous methods that share conceptual similarities with our proposed method.
A.1 MAJOR STREAMS OF PRIOR WORK
Rule-based Raw web-scraped data often contains a substantial amount of low-quality content, including documents with only space or such noise can be detected with only very simple rules, the earliest filtering methods were rule-based. Raffel et al. (2020) introduced the following rules, which have subsequently been adopted a similar form by most later works (Li et al., 2024a; Penedo et al., 2023; Chen et al., 2021: Rac al., 2021; Bane et al., 2022): (1) Retain lines only with a terminal punctuation mark. (2) Discard pages with fewer than sentences. (3) Keep lines with at least 5 words. (4) Remove pages with words from the bad-word list. (5) Removed lines containing the word "Javascript" and "lorem ipsum".
spam (Kreutzer et al., 2022). Because
Because of its simplicity, such heuristic methods often fail apply fine-graines filtering and risk discarding semantically valuable content inadvertently (Dodge et al., 2021). Nevertheless, because rule-based methods are extremely lightweight and require minimal computational resources, they commonly applied at the web crawling stage (Bevendorff et al., 2018; Barbaresi, 2021). Consequently, most web datasets already have such methods applied by default (Penedo al., 2023; Soldaini et al., 2024).
Model-based More sophisticated approaches have been proposed that leverage the capabilities of deep neural networks (c.g., classifiers), achieving superior performance compared to heuristic filtering. Such approaches can be categorized into two groups based on whether they require a manually curated reference dataset.
(1) Without reference: A representative method is computing the perplexity of an LM on the text and filtering out outliers according this measure. Another approach employs linear classification method based on the embeddings of an LLM (Li et al., 2024a). AskLLM (Sachdeva et al., 2024) operates by presenting data points to the L.MM and utilizing its reasoning capability judge whether they constitute noisy data. EL2N (Paul et al., 2023) ranks samples based on the 1.2 distance between model's prediction and the ground truth, thereby identifying data points that are more important for learning. Similarly, memorization-ben based methods (Biderman et al., 2023) assess how well a model memorizes token sequences within a document. PageRank score a method of filtering documents based on how likely they are to be hyperlinked to other documents (Li et al., 2024a). Semantic Deduplication (semDedup) embeds each point in a dataset, clusters data points together, and removes data points within clusters that are too similar (Abbas et al., 2023).
(2) With reference: using either n- n-grams or FastText embeddings samples in the raw dataset to these reference embeddings. DSIR is proposed to advance Fast (Joulin et al., 2016); it trains the model to classify curated dataset either high quality or low quality.
well-known method is DSIR (Xie al., 2023b), which constructs embeddings curated dataset, and evaluates the similarity of
Although model-based methods achieve high performance, they require training time and resources. In particular, when using LLMs for reasoning or embedding extraction, the process is more time- consuming, making it difficult handle large datasets for pretraining. Moreover, when human curation is involved, performance becomes unstable and dependent on heuristic choices.
A.2 EXPLANATION ON OUR BASELINE SETTINGS
As shown in Table 2. the baselines used in our main experiments include the vanilla Dolma dataset, FastText, DSIR, and base filtering. We explain how these methods comprchensively cover the major streams of baselines described above.
15

---

Published as a conference paper at ICLR 2026
(1) vanilla Dolma (noted as "no-filter" in the paper) Dolma (Soldaini al., 2024) corpus itself already incorporates multiple filtering stages incluuinn heuristic filtering, quality filtering, and even model-based filtering.
Heuristic rule: adopted from rules used in C4, Gopher.
Language filter (classifier-based) Using fastText to select English data.
Content filter (classifier-based) Using fastText trained on the Jigsaw Toxic Comment dataset to filter out toxic documents
Deduplication (rule-based), duplication regarding an exact match of URL, document, or paragraph.
Our method achieves substantial performance gains on top of this. Moreover, these filters are usually computationally lightweight and often applied already during the web-crawling stage (Li et al., 2024a; Soldaini et al., 2024). Therefore, they are compatible with our method by default.
(2) DSIR DSIR operates on n-gram embedding vectors trained on curated dataset. This work (Xie et al., 2023b) also compares classifier- embedding option (FastText). So this baseline covers both n n-gram and classifier-based methods.
(3) PPL-based The (Li et al., 2024a) paper compares wide range of methods, and demonstrates that PPI.-based approach outperforms well-known model-based methods (AskIILM, classifier on BGE features, PageRank, SemDedup) and heuristic filters, or is comparable to FastText and top-k average logits. Therefore, we selected the PPL-based method as the baseline representing other classifier-based methods.
(4) Fext To directly compare against classifier- based approaches, we additionally provide the comparison with FastText (trained under the same dataset as DSIR). Its performance similar but slightly lower than DSIR, which consistent with the findings reported in (Xie et al., 2023b). The limitations of curation-based methods are also discussed in $A.3.3.
Conclusion In summary, our experiments cover various types of baselines, including heuristic rule- based, n-gram-based, and classifier-based methods. In addition, provide extra classifier-based baseline experiment for the sake of clarity.
A.3 METHODS WITH CONCEPTUAL RELATION
A.3.1 DSIR
DSIR (Xie et al., 2023b) assumes that a well-curated reference dataset consisting of high-quality, well-formed text is available (Wikipedia and Bookcorpus is used in the original work). The method is to evaluate the similarity of sample d in the raw dataset to this reference corpus, and uses it as the filtering criterion.
According to Xie et al. (2023b), the process for estimating this similarity proceeds as follows. Given corpus D, each document input is "Alice is eating". it forms the list [Alice, is, eating, Alice is, is eat ang]. These n-grams are then mapped hash indices, which are subsequently grouped into m hash buckets (with m = 10000). The resulting hash frequencies form an m-dimensional categorical distribution vector 7 Rm referred to as the feature distribution P. Separate feature distributions Praw and Pref are computed for the reference dataset and the raw dataset, respectively (each denoted as q and p in the original paper).
D is sliced into sequence of n-grams. For example, if the text
From the feature distributions, we can derive feature extractors P(d)as follows:
P(d) II 4b44b
(5)
dlj] indicates jeh element the sample d. With this, we can calculate the importance weight for each data: w(d) Prer(d) The final selection is made by retaining those with the highest w(d). (d)
16

---

Published as a conference paper at ICLR 2026
Comparison with Our Method. If we set the n-gram size buckets m equal the vocabulary size, the DSIR feature distribution P essentially becomes the token prior used in our work. Moreover, the computation of our μl (the mean log prior of tokens in document) is conceptually similar to DSIR's 's feature extraction process.
and let the number of hash
However, our approach differs in several important ways: (1) Unlike DSIR, which requires both the feature distribution of the raw and the reference dataset, our method relies solely on the raw dataset. This reduces the dependency and effort for a high-quality refined reference. In practice, obtaining truly noise-free dataset is difficult, as corpora like Wikipedia or BookCorpus (used in DSIR) also have noise. Furthermore, for diverse domains (e.g.. GitHub, Chinese corpora), DSIR demands a separate Nomain-speciff reference corpus, which introduces additional overhead and subjectivity in selecting appropriate reference data.
(2) DSIR typically uses bigrams (n = 2). while our method is based on unigrams (n = 1). As a result, function words in DSIR are often tied to neighboring content words and rarely appear independently in the feature distribution, like in the example [Alice, is, eating, Alice is, is eating]. Consequently, DSIR's distribution tends to reflect the frequency of content words while neglecting the function words. This indicates a difference in the filtering principle from our approach.
A.3.2 SCDP
SCDP (Swift Cross-Data Pruning) (Nguyen & He, 2025) is method that selects data based on the multivariate median of TF-IDF (term frequency and inverse document frequency) representations. This method selects data that most similar the dominant topic frequently covered in the corpus.
To describe the method, first, a feature vector t; = T', O 1 IDF; is computed for each d E D. And documents that are closest the median (multivariate median) are selected.
Compared to our approach, SCDP differs in a fundamental way: whereas we compute token priors based on TF DF, SCDP uses TF IDF, which is the inverse way of reflecting DF. Because tokens with high document frequency receive lower IDF scores, the function words are down- weighted or often entirely suppressed. As result, SCDP's representation captures the frequency of content words only. This is in contrast to our method, which treats both function and content words as integral components of a document.
Such an approach leads to the following characteristics: (1) By eliminating the influence of function words, the method focuses on the composition of content words (i.e., topic), rather than on grammati- cal regularity. (2) Since selection is based on the median value, it favors documents that are closely related to one most frequent topic in the corpus.
This approach has limitation in that the topic of the document does not necessarily correlate with its noise level. More specifically: (1) A corpus typically contains a diverse range of topics, some of which may be represented by only a small number of samples. If selection is based on topic similarity, informative but underrepresented data may be filtered out, even it is not noisy. (2) Conversely, documents that align closely with the median topic can contain noise, while still being selected. For example, as exhibited in Figure 2b, certain web data consists of norm lists or repetitive content that may appear topically relevant but lack meaningful or well-structured information.
Due to these reasons, we argue that our approach is more optimal for identifying ill-formed, noise- heavy documents. This is because our method evaluates data based on whether the sentence is structurally well-formed, regardless of its topic.
A.3.3 FASTTEXT
FastText (Joulin et al., 2016) is a model-based filtering method. To train the model, web-crawled data are assigned as "low-quality," while manually curated target data are assigned as "high-quality". And the classifier model trained to infer the probability of a datapoint belonging to a high-quality set. (Xie et al., 2023b) explains that DSIR is improved version of FastText, outperforming in performance. Like DSIR, key limitation of FastText is that filtering quality is constrained by the accuracy of manual curation, which also increases rigidity and complexity due to the required human
17

---

Published as a conference paper at ICLR 2026
labor. Moreover, because web-crawled data must be additionally tokenized, the computational time cost increases.
We provide additional analysis the FastText model in the same setting DCLM (Li al., 2024a), assigning web-crawled data (RefinedWeb (Penedo et al., 2023), which is based on Common Crawl) the "low-quality" set, while ELI5 and OH-2.5 as the "high-quality" set.
Case analysis We analyzed the outlier cases from the FastText classifier and observed several unexpected patterns. Among the samples classified as having a 0% probability of belonging to the high-quality set, we observed a substantial number of well-formed texts 1etl-ppatteelallyy those resembling news articles (Case 1). Conversely, many samples classified as having a 100% probability of being high-quality were clearly noisy or nonsensical scripts (Case 2). We double-checked our implementation, but confirmed that these results were solely due the model's inference behavior.
Case Data assigned a 0% probability of being high-quality (i.e., 100% belonging to RefinedWeh) ranking the absolute bottom of the distribution.
Paula's Choice Donating $50, 000 to the COVID-19 Solidarity Re- sponse Funa for World Health Organization.
Pyer Moss Pyer Moss has set aside $10, 000 get supplies for med- ical workers while also converting their NYC office into donation center to store the supplies. Using lecal factories, Pyer Moss is creating 1, 000 mask covers to send directly to front line workers. With the help of Jen Rubio,
Case2. Data assigned 100% probability of being high-quality - ranking at the absolute top distribution (Ist rank).
'0] |001::SNIII001111111111001111110
? 2100110000N1|20000]]100000011110
ddMMyy HH:mm:ss (100:::0011"11111111001111111133 [nn]
? 1 ( JunAM \/Agg\M
Aall "MM
d HH:mm:ss"
Sep
We hypothesize the following reasons for this unpreferred behavior:
(1) RefinedWell contains substantial amount of well formed data. To avoid the cost of human labeling, DCLM chose to label RefinedWeb (Penedo et al., 2023) the low-quality set, while labeling OH-2.5 and ELI5 as the high-quality set. During inference, the model predicts the probability (0 100%) that given text belongs to the high-quality set. However, RefinedWeb also contains considerable proportion well-formed documents. As result, many samples that FastText classifies with 0% probability of belonging the high-quality set are in fact well-formed and informative texts (Case 1), often resembling those found in RefinedWeb..
One possible explanation is that news article-style texts prevalent in web-crawled sources (e.g.. RefinedWeb), but largely absent from curated datasets like OH-2.5 and ELI5, which mainly contain question-answertt format. As result, the Fast model may have implicitly learned classify the news article (or other non con-QA-style) format as belonging to low-quality set, leading to systematic misclassification.
(2) Limited discrimination capacity. Another possible explanation is the prevalence of code and math-related content in OH-2.5. Since such sources are relatively less common in RefinedWeb, the model may have overfit to these symbolic patterns during training. However, due to the limited capacity of the small FastText model, it is unable capture deeper coherence within symbolic
18

---

Published as a conference paper at ICLR 2026
language. As result, it may incorrectly classify meaningless noise that superficially resembles symbolic content (as in Case 2) as belonging to the high-quality set with 100% probability.
These results underscore the weaknesses of model-based methods, supporting the robustness of prior-based approach.
B THEORETICAL DERIVATION AND BOUNDARY CONDITION ANALYSIS
In this section, show that (1) under the linguistic observation presented in $2.2, the prior- based metric shares noise- diiiseiditinuushhing behavior with PPL, regardless of the scale of the model or corpus. In particular, (2) demonstrate that the distance from the central point is distinguisher of outliers. (3) We also explain the boundary conditions under which this property does not hold.
B.1 THEORETICAL GUARANTEF OF NOISE DETECTION BEHAVIOR
B.1.1 THE COMPONENTS OF PPI. AND THE APPROXIMATION VIA PRIOR PRIOR-BASED METRIC
As is well known, PPL consists of the following components: the likelihood and the prior term.
log PPI(d) & Σ, pore( (z;) + Σ; pogz(
And in $3.2, we introduced two prior-based based metrics:
(1) Pd < Σ, log p(x;)
(2) σ. = std]p(a;)] x E.(((r)) - eal? ea E[p(z,)]
Prior term of PPI. can be seen, the μaa identical to the prior term of PPI.. Therefore, the basor-based filter partially approximates PPL.
Likelihood term of PPL. The and likelihood share similarity in that they reflect relationships among tokens rather than averaging the independent properties of individual tokens. However, is difficult to formally prove universal connection.
However, under the following assumptions and scenarios, they exhibit similar behavior in filtering noisy data.
Assumptions:
We assume very simplified settings of linguistic observation in $2.2.
(1) Suppose tokens consist of only two categories, and C, where F represents function words and represents content words. (2) Within any well-formed document, these two groups must follow fixed ratio r" (e.g., F:C=4:1).
(3) Actually, data d has fixed length (c.g., 5), and has observed ratio And has only two possible values: y ("well-formed) or á (noise).
Example of r* FFFCC, FFCFF
Example of f FFFFF, FFCCF
(4) If the data of r* takes the big proportion in the corpus, both the empirical average of ratio and the trained model p converge near to r*.
(1)-() are simplified versions of the linguistic observation in $2.2, and (4) represents the minimal
condition required for our method to operate. (Experiment $3.4.2 shows the minimal proportion is around 10
Property of conditional likelihood In this simplified setting, conditional likelihood is determined solely by r (e.g., p(else C)) & p(FFFF C* 1).
Therefore, the distribution of E, log p( (x;) places data with r* near to 1, while placing ş near the left tail. And since x* dominates the corpus (assumption 4), the overall distribution of I, becomes right-skewed.
19

---

Published as a conference paper at ICLR 2026
Therefore, in the distribution of la, data with ratio can be identified through the left tails that deviate far from the representative value (c.g.. median) of the distribution.
Property of dd For the dd. the computation is straightforward. Given fixed occurrence prob- abilities (e.g., (CC) = 1/5 and pFF) 4/5), the aa are distributed around specific value (e.g.. F[ 0.26) where the ratio of data is r" And the samples that deviate far from this point can be distinguished as the one with f.
Therefore, both the likelihood term and the aa exhibit similar behavior in identifying A.
Property of pdd Under the FC assumption, we further explain the role of the prior term of PPL (Ha) in distinguishing é. Suppose that F and C have fixed occurrence probabilities (e.g., p(F) = 4/5, p(C) 1/5) and appear within each document according to those probabilities. Then, we can observe that the average of their occurrence probabilities tends to concentrate around specific value (e.g., E(ua) 0.68).
If the observed ratio deviates from this central value, it will move further away from the central value, allowing us to identify é accordingly.
In conclusion, under assumptions 1-4, we can guarantee two properties of prior-based metrics. (1) the prior- based metrics (Ha, align with all components PPL in identifying So, we can say the prior- brser-based filter "weakly aligns" with PPL-based filter. (2) the prior-based metric of normal data with valid structures (r") converges around single central point, allowing outliers (f) to be distinguished by their distance from this center.
B.1.2 DISCREPANCY FROM THEORETICAL GUARANTEE: BOUNDARY CONDITION ANALYSIS
However, such an approximation required simplified assumptions. Consequently, when these as- sumptions are violated in a real-world data, discrepancy arises. The most significant violation case we can identify is the following: even when document follows the ratio r", it may still contain no meaningful information. This corresponds to cases such as:
(1) Text with an appropriate balance (r*) of function and content words, yet are semantically nonsensical. This case is illustrated with Figure 3(a). (2) The data are not noise, nor semantically nonsensical; rather, they are either too difficult or too trivial to be useful for the model. (In this scenario, the main LLM should compute PPL directly, so this case is not applicable to our PPL baseline either.)
In such cases, PPL will play a clearer role, and the prior-based filter will not be able to handle this, and we do not claim this. Prior filter is designed to identify only the noisy data (f), as we emphasized in the title and 11.
However, claim the following: (1) The prior at least provide clearer signal in identifying A, as model-predicted likelihood can often be unreliable. (2) In the corpus with a very high proportion of noise (c.g., web crawled corpus), identifying f may be more impactful than deliberate reordering among r" data. (3) Still, compatible with Pal-ba filter.
B.2 ROBUSTNESS JUSTIFICATION FOR MEDIAN BASED THRESHOLDING
By drawing on the above theoretical arguments, we can justify the robustness of our methodological choices of median threshold. For example, this allows us to answer the question of whether the median-bait based threshold remains effective when the corpus exhibits a mixed multimodal distribution.
B.2.1 MIXED MULTIMODAL DISTRIBUTIONS
(1) Prior-based filter only measures the grammatical structure via the function-content word ratio. And texts in one specific language tend to share the same grammatical structure, regardless of their style domain (e.g., news, forum). Thus, frior-filter will generalize to each other.
Dolma is such an example. Although Dolma consists various types of data (e.g., news, books, symbolic scripts), our method has been shown to perform well across these domains.
20

---

Published as a conference paper at ICLR 2026
(2) Std metric (oa)) measures the distance between token priors, thus the distance itself may not vary much among other types of language.
Let's bring the "FC" assumption from $B.1 to here. Let's assume there heterogencous language groups A and B, and each has its function-content words pair, and each group follows the assumption of $B.1. (F, C) E A, (R. T) B.
According to the linguistic axiom, we can assume that distinct linguistic system (A, B) shares a similar function-conter ratio (e.g.. 1:4). Therefore, the central value of std metric for both languages the same (e.g.. A = Elo = 0.26). Thus, da for A and B share the same central value, even though they heterogeneous.
(3) However, in practice, if the amount of B is significantly smaller than A, then there can be an imbal- ance of the funetion-content ratio. For example, p(F) 0.4, p(C) 0.1, p( (0004,p(7) 0.0001. In this case, the central value of the distribution of o and o. will diverge. We suggest two points:
(3-1) We tested an extreme case of this in $3.4.2 with the Chinese- Chinese-English set, to answer the question "When two entirely different languages are mixed together. what extent can a single median encompass both languages?" Our experiments indicate that if a minority group is mixed only over 10%, most of the minority language still considered an inlier when using single median.
(3-2) If this is still insufficient, we can compute the token prior by mixing in a corpus that shares attributes with the target corpus. For example, we mix in more data that resembles corpus and then recalculate the token prior, might obtain token priors such as p(F) 0.4, p(C) 0.1, p( Rt) = 0.4, and p(C)= 0.1. this case, we can bring corpora A and to the same median value again, even though the actual proportion of B is very small.
This approach corresponds to the method described in (2. of our paper. For example, when we compute the token prior using setting where GitHub data are added to Dolma, the resulting token prior enables better identification of GitHub-like data (e.g., symbolic scripts) within Dolma.
C GENERALIZATION TO OTHER LANGUAGES AND TOKENIZERS
In this section, we discuss how our method generalizes to languages beyond English and to tokenizers other than the GPT-2 tokenizer.
As we discussed in $B, our method is supposed to work with any language system following two features (1) the language is composed of function words and content words, (2) function and content words have distinct distribution of frequency. This character applies to most natural languages (e.g.. Chinese, Japanese, Turkish).
Most languages are composed of function-content words It is well observed in linguistics (Stromswold, 1994; Croft, 2002) that matter how compound language is, it can be separated into function morphemes and content morphemes.
For instance, the Turkish compounding word evlerimizden ('from homes") decomposes into ev (content morpheme) + ler + imiz + den (grammatical morphemes).
These function and content words follow the frequency rule (i.e., function words tend have higher frequency). For example, in Turkish, the function word "ler" appears across many different nouns (e.g., evler, yehirler, ögrenciler resulting in higher term frequency.
The same phenomenon occurs even when dealing with English. For example, a compound word such childish is typically segmented into child and -ish. The suffix -ish appears across many different content words, and thus behaves similarly to a high-frequency function morpheme.
The tokenizers split words into morphemes based on frequency Most tokenizers (e.g., BPE, WordPiece, SentencePiece) have tendency to merge characters with high co-occurrence probability into single token (e.g., "i" + "n" "in"). This is because most tokenizers, either directly or approximately, follow the Minimum Description Length (MDL) (Hou et al., 2023) objective, which aims minimize both the vocabulary size and the length of representation. For example, the token
21

---

Published as a conference paper at ICLR 2026
"ler" frequently co-occurs with various words, assigning it as independent token is more efficient (i.e., covering a wider range of expression with limited vocab) than assigning "sehirler" and "evler" separate tokens.
Combining together, most natural languages can be segmented into function and content compo- nents, and most tokenizers effectively reflect this distinction, even in morphologically compounding languages.
C.I. EMPIRICAL EVIDENCE
Other languages The Chinese-Englins experiment ($3.4.2) provides evidence for generalization to other languages, such as Chinese.
Table 4: Same setting Figure 5. Pro- portion of Turkish data classified as outliers after mixing Turkish and En- glish data at ratio of 100. Outliers correspond to the top and bottom 5% μla
Suppose that the prior-based filter works only for English but not for Chinese, then our filter will classify all Chinese data noise. Then, in Figure at the point where Chinese and English are mixed at 10:100 ratio (at any ratio), nearly 100% of the Chinese data should be classified However, the actual results show that most of the Chinese data were classified as inliers, only if this language group takes more than 10% of the corpus. Therefore, the prior- based filter works for Chinese as well.
Size of Turkish data (a)
Outliers (10%)
outliers.
0.5036
5
0.2778
10
0,2448
20
0.1984
100
0.1337
We also conducted the same experiment on Turkish (Table 4). And the results were consistent with those the Chinese-Englist experiment. We provide an explanation below.
Other tokenizers We provide comparison with two additional tokenizers: LLaMA-3-8B (UTF-8- based, vocab size 128K), and T5-small (SentencePiece-basee tokenizer (mainly used in our paper, 50K) has a vocab size of 50K. All of these tokenizers tend to assign function morphemes as individual tokens in accordance with the MDL. principle. However, we measure the performance differences among different granularities.
vocab size 32K), in Table GPT-2
The result is reported ETable 4 (also Table of our paper). All tokenizers showed the advan- tage of filtering, while T5 exhibited slightly lower performance gain. We propose two possible in- terpretations: (1) There may exist a threshold of optimal granularity, which lies between 32K and 50K. (2) Using the tokenizer paired with the model may offer stability.
Table 5: Performance of large (1.5B) models Dolma under different tokenizers. Avg: Avg normalized accuracy, W: World knowledge, C: Commonsense reasoning, L: Language under- standing, S: Symbolic problem solving, R: Read- ing comprehension
Tokenizer
Avg
W
C
S
R
GPT2
9.20
9.53
11.27
10.31
11.13
3.79
Nevertheless, as demonstrated, priors can be com- puted even with tokenizers mismatched the model. If an optimal tokenizer exists, it can be freely adopted at any time.
LLaMA-3
9.39
9.54
11.16
10.78
11.66
3.64
TS-small
8.11
8.59
7.43
7.95
12.32
4.22
C.2 SYMBOLIC LANGUAGE
Symbolic languages (e.g., codes) are more worrisome case, they are more different from English, and it has not been verified whether linguistic observation of natural language applies to them. Therefore, we conducted experiments this extreme case ($4.2), and empirically proved our method still generalizes to symbolic language as well. We can expect that it will generalize to other natural languages more similar English.
D PRESERVING MINORITY DATA
As data filtering removes noisy data, which is often found in outliers, it inherently risks discarding valuable minority data (e.g., highly technical paper with rare terminology, corpus including
22

---

Published as a conference paper at ICLR 2026
minority languages) trade-off. Minimizing this trade-off is the general goal of research filtering. In this section, we discuss our prior-based filtering method from the view of this trade-off.
This trade-off can be measured through Dolma In our experiments on Dolma, we also observe such a trade-off. As shown in Table 1, Dolma consists primarily of English- based web data, with small portion of programming language data (The Stack), and understanding this language is separately measured through "symbolic problem-solving" in Table 2.
Prior-based filter shows better trade-off' Table shows that the no-filter setting achieves the highest symbolic understanding, while the PPL-based filter shows very low symbolic understanding. even with the improved overall score. This clearly illustrates the trade-off: filtering is applied, minority languages are pruned away. In contrast, the prior- based filter shows much higher symbolic performance compared to the PPL-based filter, while also maintaining higher overall score. This indicates a significantly better trade-off of our method. We provide additional evidence in $D.2.
Nevertheless, propose additional approaches further address this trade-off.
D.1 ADDITIONAL METHODS TO ADDRESS TRADE-OFF
(1) Using only stds: Our original method leverages both the mean and standard deviations (stds) of the prior. Using only the stds may be beneficial, since the mean reflects the average frequency of tokens, whereas the std captures the dynamics among them. This distinction can make stds-based filtering more effective in identifying well-structured documents with Now-frequency languages. In Table the filter with stds criterion achieves higher symbolic understanding than the mean+stds filter, while maintaining comparable average accuracy.
(2) Calculate prior on blended corpus: Another approach is to incorporate target-domain data when estimating token frequencies, thereby assigning higher prior probabilities to domain specific terms and preventing their exclusion during filtering. In practice, Dolma in equal proportion for prior computation, which needed only an additional 10 minutes of processing. As result, overall performance improved beyond the original method, while symbolic task performance also increased ("Dolma + Github" among Source column in Table 6).
mixed File-GitHub data with
Table 6: Performance among baselines pre-trained on Dolma.
Criteria
Source
Average normalizel accuracy
knowledge
Commonsense reasocing
Language understanding
Symbolic problem salving
Reading compre-
hession
Large (1.5B) model
prioe based
Dolma
8.50
9.12
10.25
7.45
11.38
4.28
10.57
mitiin
11.27
3.79
Dolma + Github
9.48
11:47
10.83
3.97
12:22
3.78
D.2 ASSESSING DATA LOSS OF PRIOR-BASED FILTER CONTROLLED SETTING
There be worry that highly technical paper with termi- nology or a poem with unusual syntax could be incorrectly filtered. We aim validate this scenario in controlled setting. We first con- sider a scenario where a document with general structure contains extremely rare terminology. To simulate this, we sampled 1,000 data points (each 5 512-token length) from the Dolma dataset within the central +15% range of μ and gradually injected rare terms into them. Rare terminology was generated by concatenating two tokens ranked the bottom 10% of the prior distribution (e.g., "prosecromeda", combining two tokens "prosec" and "romeda"). We inserted of these terms into each text and measured the percentage of these texts classified outliers +25% threshold). The result is presented in Table 7. Up seven insertions of rare terms amounting to 14 tokens (2.7%) within 512-token 10% of texts were filtered out.
Table 7: is the number of injected terminology, and in- liers is the rate of texts re- maining in 25% 75% bound- ary.
8
nX token length
inliers (25% 7516)
1.0
12
1.0
0.98
0.91
were required before around
23

---

Published as a conference paper at ICLR 2026
To assess whether seven occurrences are reasonable upper bound, examined the typical frequency of topic-specific terminology in real-world text. Specifically, we sampled 10,000 Wikipedia articles, segmented them into 512-token blocks, normalized all text to lowercase, and counted the occurrences of the article title within each block. On average, the title appeared 1.09 times per block, corresponding average token length of 3.44. This suggests that even documents intended explain given concept are far from being dominated by that terminology. Taken together, these findings indicate that the prior-based filter exhibits strong robustness when handling scenarios involving rare terminology.
E ADDITIONAL ANALYSIS
E.1 SENSITIVITY
Sensitivity block size We conduct the following analysis to assess whether outliers remain consistently detected across different block sizes. We randomly concatenated two 512-token blocks to make 1024-token blocks, then trimmed e/2% from each side to identify outlier samples: x 21024;- = x z5522 + 512 2 where i denotes the data index in the corpus X, and x" represents text block of size with 5112 E X-12
Table 8: Overlaps of outliers between dataset with differ- ent block size.
1024
2048
0.7935
0.6954
10
0.8145
0.7263
e 20
0,8102
0.7265
If a sample zz!12 is classified as e% outlier, we then check whether the concatenated block a,1024 is
also classified as an outlier. We repeat the same comparison for x 2048 As shown in Table 8, outliers of smaller blocks were largely retained outliers in larger blocks, indicating strong alignment. However, the overlap diminishes as n grows. This is because if a page contains both noisy and (more or equal proportion of) clean content, discarding the entire page may not always be ideal.
Sensitivity to threshold Regarding the threshold ratio of outliers, we follow the optimal threshold of the PPL-based filter. Ankner et al. (2024) extensively evaluated PPL-based filtering across different selection rates (25%, 50%, 75%) and concluded that 50% was the most effective. Nevertheless, explore an additional approach for verification ("elbow"" in Table 9). As illustrated in Figure 2, we normalize μ., σ,, rank to 0 scale, and define boundaries where the gradient crosses -1 (on both sides). resulting in an 82% threshold of the data. However, the overall performance was lower than the 50% threshold. We also present an experiment with threshold of 25%, which also showed clearly lower performance. These observations were consistent with the observation of Ankner et al. (2024).
Table 9: Performance of large (L.5B) models on Dolma, under different threshold settings.
Average normalized иccuracy
Symbolic problem solving
Reading compre- hension
World knowledge
Commonsense reasoning
Language understanding
Threshold
25%
6.21
7.03
7.06
5.99
7.36
3.57
50%
9.20
9.53
11.27
10.31
11.13
3.79
81% (elhow)
8.79
10.04
7.67
8.97
13.04
4.21
E.2 CORRELATION BETWEEN PRIOR AND PPL
We present an additional analysis illustrating the correlation between the prior and PPL. Our hypothe- sis is that this correlation would be more pronounced among outliers, as inlier data exhibit very low variance, making rankings fluctuate easily.
To examine this, we selected the top, middle, and bottom Ik samples based on j and computed their average PPL (E(PPL), see Table 10). While the results initially appear to show a highly linear correlation, this is largely driven by extreme outliers within each subset. To miti- gate this effect, recalculated the average after
Table 10: 1000 samples from the top, middle, and bottom ranks.
EUPL) EPPLL Inmoned
ETPPL MPPLT
13.
34.7 4701.9
460.
20
9.0
25.6
24

---

Published as a conference paper at ICLR 2026
trimming the top and bottom five samples from each subset ( (E(PLL*) ), and then measured the 1-norm difference from the median PPL: [[PPL*) M(PLL)..
The results reveal that subsets centered on correspond median PPL values, whereas subsets with extreme μ values (top and bottom) exhibit PPL averages that are substantially distant from the median. This indicates that prior-based rankings effectively identify segments of the data that deviate from the central distribution under PPL.
F DETAILS ON EXPERIMENTS
F.1
SCORES FOR EACH BENCHMARK
Table 11: Benchmark performance of large (1.5B) models.
Model
World knowledge
Commonsensee reasoning
Language undersiunding
ARC casy
BIG bench wikidara
TriviaQA
COPA
OBQA
PIQA
HellaSwag
LAMBADA
Winegrad
Winogrand:
no-
DSIR
9.65
4.42
0.47
1.47
0.53
16.00
2,70
13.43
13.55
-0.71
PPI -Pi-s
11.79
8.19
0.87
2.34
0.27
19.48
4.11
16.85
9.89
-1.18
Prior based (ours)
12.29
6.78
1.27
1.38
40.53
20.35
5.84
18.45
14.29
2.45
Symbolic problem solving
Reading comprehension
Model
BIGi-hench dyck lan- guages
BIGi-hench elementary math QA
BIG BIG-bench operators
algorithms
GSM8K
SVAMP
LSAT-LR
LSAT-RC SAT-English
CoQA
no-filter
57.12
7.14
6.67
3.48
6.80
DSIR
39.92
13.70
2,70
5.71
0.15
1.33
3.79
4.48
6.15
1.47
PPL-hased
25.23
0.60
3,27
7.14
0.68
3.33
3.53
4.48
5.50
2.34
Prioe-based (ours)
33.03
Table 11 reports the performance of large (1.5B) models on Dolma across different filtering methods. As discussed above, the prior-based generally outperforms other baselines or performs comparably to the best baselines.
F.2 JUSTIFICATION OF EXPERIMENT SCALE
F.2.1 REGARDING TRAINING TOKEN DURATION
To ensure experimental rigor despite our resource constraints, we followed the experimental setup of representative paper that we conceptually inherit Ankner et al. (2024). According to this study, the performance gap between baselines is established the early phases of training. Ankner et al. (2024) reports downstream task performance up to 25 or 50B token duration, and it shows that the performance gaps between baselines observed at 5B token duration remain largely consistent throughout the last stage (Figure 1 of Ankner al. (2024)). Our work leverages this observation. and this also aligns with our experiment Figure 7.
Particularly in Ankner et al. (2024), all models trained with 2x data repetition, and the 5B-token duration in Figure of Ankner et al. (2024) corresponds to 2.5B tokens data repeated twice. Our experimental setup also uses 3B tokens repeated twice (total 6B), which makes directly comparable to the 5B-token point in Figure 1 of Ankner et al. (2024).
These trends consistently observed in other works. li2024datacomp also demonstrates that the ranking of model performance is largely determined early in training and remains consistent. In addition, Figure of diao2025climb clearly shows that the initial performance gaps are established in the early phases.
F.2.2 REGARDING THE MODEL SIZE
Many representative works data filtering employ relatively small models around 1.5B parameters, assuming that such a scale is sufficient for reliable evaluation. For example, Ankner et al. (2024) uses IB and 2B models, while Diao al. (2025) employs 350M and 1B models. In addition, recent
25

---

Published as a conference paper at ICLR 2026
representative works in this topic (Li et al., 2024b; Penedo al., 2024; Longpre al., 2024; Xie et al., 2023b) all conduct experiments with models smaller than 2B. This contrasts with studies focusing on model architectures (e.g., LLaMA, DeepSeek) that compete in complex reasoning performance, which typically require pretraining of large-scale models (e.g.. 7B and 70B). A detailed rationale of why small model settings can still maintain experimental rigor in data filtering, is provided below.
F.3 INDIRECT EVALUATION ON THE COMPLEX REASONING TASK
Certain abilities, like complex reasoning skills, may exhibit emergence in the later stages of training or in larger models. This may be why the reviewers expect experiments at larger scale, We will provide an explanation and an indirect evaluation regarding this.
F.3.1 JUSTIFICATION
Our method is designed to filter noisy text data, which consists of broken sentence structures. If we can assume noise as "data that contains no interpretable information", then learning such data does not contribute the emergence of complex reasoning. This is because the emergence of complex intelligence arises from the integration of information within the model (Cleveland, 1982), but noise contains no information contribute to this process. See noise cases filtered by prior-filter in Appendix F; they will remain useless no matter how much the model learns it or how large model learns it.
If we can assume that learning more information (i.e., from corpora with less noise) leads to better complex reasoning skills, the amount learned information can be evaluated at smaller scale and with more straightforward benchmarks, which are sufficiently covered in our experiments.
F.3.2 INDIRECT EVALUATION
Recently, some research views that reasoning skill is learned directly from the dataset as part of pattern memorization, rather than emerging a mere intellectual invention (Wu et al., 2024; Mirzadeh al., 2024). From this view, if the training dataset contains more data that is similar the targeted complex task benchmark, the model's performance that target benchmark would increase accordingly. This because the model can utilize larger number of direct templates to solve the task, in manner of pattern memorization.
From this concept, we mixed the targeted benchmark dataset (e.g., MMLU-pro) into the Dolma corpus and measured how much of this benchmark data is maintained through each filtering method. This simulates a scenario where the training dataset contains data similar to the target benchmark. Therefore, if more benchmark data remain after filtering, that such data (i.e., similar to the targeted benchmark) is preserved in a larger-scale setting, leading to better benchmark performance. Although a small LM (usually under 7B), this may offer an indirect way to estimate the potential impact of filters on complex task-solving capability.
assume there higher likelihood
not directly evaluate MMLU-pro performance with
As a result, we find that the prior- based filter achieves significantly better recognition performance on MMLU-Pro data compared PPL and random selection. Based on this observation, difficult to argue that model trained with PPL-based filter would achieve a better score than a based filter MMLU-Pro.
Table 12: The remaining portion of the MMI.U-Pr dataset for each filtering method and threshold.
Threshold (<)
+0,2
+0.25
+0.3
#0.4
+0.45
based
0.222
0.292
0.436
0.740
0.987
Brior-based
0.510
0.629
0.743
0.931
0.990
Random
0.400
0.500
0.600
0.800
0.900
We analyze that this might be due to our method's better preservation of "rare but valuable data." as
discussed in Appendix B. MMLU-Pro is known to contain a lot of logical statements, which likely
constitute only a small portion of web corpora (c.g.. Dolma). As prior-filter is more sensitive to the rare group (SD), it might have higher chance of capturing such data.
26

---

Published as a conference paper at ICLR 2026
G DETAILS ON BENCHMARKS
Jha et al. (2023) also use the MosaicML evaluation gauntlet to perform evaluations in their work. As such, with explicit permission from the authors, we reproduce their text describing the tasks and task categories in the evaluation gauntlet. The following is from Section D of their paper:
The World Knowledge category includes the following datasets:
ARC easy: 2,376 easy four-choice multiple choice science questions drawn from grade 3-9
science exams. (Clark et al., 2018)
BIG-bench wikidata: 20,321 questions regarding factual information pulled from Wikipedia. (Srivastava et al., 2023)
TriviaQA: 3,000 question-answeriin dataset; clipped all answers be most 10 tokens long to improve speed. (Joshi et al., 2017)
The Commonsense Reasoning category loosely assesses a model's ability to do basic reasoning tasks that require commonsense knowledge of objects, their properties and their behavior. It includes the following datasets:
CОРА: 100 cause/effect multiple choice questions. (Roemmele et al., 2011)
OBQA (OpenBook QA): 500 four-choice multiple choice questions that rely on basic physical and scientific intuition about common objects and entities. (Mihaylov al., 2018)
PIQA: 1,838 commonsense physical intuition 2-choice multiple choice questions. (Bisk et al., 2020)
Language Understanding tasks evaluate the model's ability to understand the structure and properties of languages and include the following datasets:
HellaSwag: 10,042 multiple choice scenarios in which the model is prompted with scenario and choose the most likely conclusion to the scenario from four possible options. (Zellers et al., 2019)
LAMBADA: 6,153 passages take from books version. (Paperno et al., 2016)
the formatting adopted by OpenAl's
Winograd Schema Challenge: 273 scenarios in which the model must use semantics to correctly resolve the anaphora in sentence. (Levesque et al., 2012)
Winogrande: 1,267 scenarios in which two possible beginnings of sentence presented along with a single ending. (Sakaguchi et al., 2021)
Symbolic problem solving tasks test the model's ' ability to solve a diverse range of symbolic tasks including arithmetic, logical reasoning, algorithms and algebra. These datasets include:
BIG-bench algorithms: 1,320 multiple choice questions. (Srivastava al., 2023)
BIG-bench dyck languages: 1000 complete-the-sequenee questions. (Srivastava al.,
2023)
BIG-bench elementary math QA: 38,160 four-choice multiple choice arithmetic word problems. (Srivastava et al., 2023)
BIG-bench operators: 210 questions involving mathematical operators. (Srivastava et al., 2023)
GSM8K: 1,319 short, free-response grade school-tevel arithmetic word problems with simple numerical solutions. (Cobbe al., 2021)
300 short, free-response grade school-level arithmetic word problems with simple numerical solutions. (Patel et al., 2021)
SVAMP:
The Reading comprehension benchmarks test model's ability to answer questions based on the information in passage of text. The datasets include:
27

---

Published as a conference paper at ICLR 2026
LSAT-LR: 510 passage-basee four choice multiple choice questions. (Zhong et al., 2023)
LSAT-RC: 268 passage-basee based four choice multiple choice questions. (Zhong et al., 2023)
SAT-English: 206 passage-basee four choice multiple choice questions. (Zhong et al., 2023)
CoQA: 7,983 passage-based short free response questions. (Reddy et al., 2019)
H USAGE OF AI ASSISTANTS
In preparing this manuscript, we relied on Al-powered writing tools to refine sentence flow, fix grammatical mistakes, and improve readability. These assistants used strietly for language polishing and played no role in shaping the technical content, research design, or experimental work. All scientific concepts, findings, and conclusions presented in this paper were fully developed and written by the researchers. The involvement of Al was limited to editorial support and did not influence the originality intellectual contributions of the study.
REPRODUCIBILITY STATEMENT
Within the abstract, included URL anonymous GitHub repository where code is made available. This repository not only contains the full implementation but also offers detailed guidelines for installation, along with step-by-step instructions how perform both training and testing.
28

---

Published as a conference paper at ICLR 2026
J FILTERED CASES
In this section, provide more cases that are classified extreme outliers by the prior-based criteria.
J.1 PRIOR STDS(0a) ) BASED
Sieindleeroe Lokk docke di
D...S. C.A.C.A................... D.A.C.D...................
A.G.AAAA...RR.. F.M.L.C.................... C.M.A.C.K....
geditrae aeeeeaaaee taaieeeiii au misie aund any s.n.d.i i---- e.r.e..odda w.w.d..diiiiiiii....... eedii casauni oo oil i.a... a.d.d..ndd au.... wnd.iii mi and aedeeee And.i a.p.a.d.a. tinioss... waw.aaaaaaa savaiinn oi. 1... ai. in auriaa d.d.. iinii f.dde Lot doaie Anwo.di ci aoni 6 -2.0"
M.S.D.D.......M... N.D.A.F........ FACEPO.O.,
R.S.D.O.,
C.N.P.A.S.... P.A.A.
bottom 1
P.A. w...s.d............... W.M.S.H.........NNN...... FACOMBONS
recrCeeN, APM, wWW.N.T..P..... N.T.N.AA.PP......TTTT.
Ng...A............
1.d, nid.
n.d. Vneox. gain trd.n.
bottom
n.d. Aomok, loss
9.7.
ga5n
5:15
30.4.
16.5 18.5.
18.7. 13.3.
18.3.
10.10.
16.11.
10.12.
n.d. n.d. 1rRLD
0.82
0.62 9:02 0.00 n.d. n.d. n.d.
n.4. n.d. n.d. n.d. n.d.
11.1.
20-16.
31.3.
10.17
n.d. 0.01 0.01 0.82
0.02 0.02 n-d. m.ddinn 0.004
0.004 n.d. m.d. 0.11 8.13 0.75 0.35 n.d. nid, n.d. n.d.
12.33 -12.33.
-32.34. 12..5. 112.6.
n.d. n.d. n-d. n-4-Vic3 0.006 6.064 n.d. nid, 9.02 9.65
12.37 12.33
<32.35
1833-
33..2 13.3. 15.4.
0.02 0.81 0.83 6.800 r.d. n.d. n.d. n.d. n.d.
13.5.
133.4/1544.1. - -34.1. 33.3. 1164.
indevnit of total cations plua OK, n.d'
16.5 155--wt5.3
25.2.
-35.3
15.4. 15.5. 15.6.
15.7. 13.8.
13.3.
bottom 3
1.7. 1.8.
p.3.
p.d.
Vnl.3.
1.9.
1:18
5:2.
L.3. 1:4, 5.5. 1.6,
1.11.
bottom 4
n.2. n.a. 4.4.
0.5.
9.7.
nia. n.9.
n.18/141.1.
5.2. 1.3.
1.4.
1.5.
1.4. 1:7.
candie. was Mick, Lit, clous, soake woaldrring Dibers: Siace. fire. Firess firerigterr fire replesion.
1.4. 1.9.
1.38.
1.33.
1.12.
1.39. 1.14.
1.35. 1.15. 1:37.
3.18.
11119
3.200 11211
1.22. 1.28. 1.235
1.25.
11.00
+3.27. + 1..28.
teplesive. luller. austauerr ytroegy tyrotemiia. firerra. firscracher. Soroccat. Netooris, teeelve. otil. satellite. trjeetory. Fleeting. tarrage Valley. iti. Ceron, Consecative. streat taple tire, troming arctery. quiver, arree. Rociet. Rissila. silo, Terpade. lasrch. finng, catepilt. Slingshst:
1.2.. 1.2. 2:3- 22.4. 22.5/183.
face
5.1. ..4. 1.5. 3.6. +3.7. ..R. 1... + -..1...
8.11. 5.12. ...1. 1.12. + 1.15. 1.16. 1.17.
5.18. 9:19. - 3.20.
1.25--wt:1.. 4.2. 4.3.. d.4.+
a.5.vis.1. -..2. 5.3. 5.4,
5.6. 5.7.
5.a. - 5.9. 5.10. + A6..5. 6.2. 6.2. 6.4.
Thrsst.
Pitcter.
6.5. 6.6. 4.7,
Parvestor.
bottom 5
IMllogpter. Torqua, Proseller. Tum, letate. (hiri. Siiri Triss, Thister. Torreco, Mirliint unleach. Target. boll rarkenon, Ain Point. Hozueet. nes. turr. nebett here, tirth, Retiosisen Mizachandrss. carbetydrates calories. toasto. Rayal Jelly, Haner. terret. Haspa, Bizzing: haming. tixe.
busta
cen anrels dertadd balle wtil cnggertteetteeeeieee deport
31h contentcctioeeee di
ternesslogirt. Indectios.
Tingin.
pengle dertail
butle. J18
Barzte stpton. aladder.
con piggle dentadd balte
bottom 10
pigle baite
ren engll destady bulte 31h eetl 002letleert
rossle, statts. archi Liaee, Vrieeer mec, boc.
con pogla dwrtocl bultA 31h vell inooenptthiittte
best. tel. amy
imn, 20, cares velsiare and
turtinas 2011, der Roor.
wot.pogga..dettee deronii buile 2ik visi. exdeties saport
Pisl, Dwn,
con prgia dentasde buite fia--h, Wilnigaties voiport
anges (orel:. ribera wet voild 2814 2014, to Soudes indeessinn wost. henertutt, Irapher U. surta vinics berbolrt, c. with, 6. vajoa)
Doyaee,
bottom 100
Musaaah, dallowyy neneery sampicser, tuyo. soul arcsmitee Nuwer.
3, (ajan
bottom 500
meve bit deverc scae rt, peer dele /o0 seume Lis mime
ihe is rose 221 aasoss a samaae Sie peases Janamee iise Wastss 23.
12111220 riccuss anl peictzi arowes Meccuer ww.wws. ancans aooms
(1s anrene
10142200 altaat Mho1224 caly as AWALII stvaes 30142 si-ett eivtte 4ovet
daea Shep 797 cesase nerieie
coce cito saao
0411225 al Mix1344 301 amaain aenees atrull 33/net whiii 42mel
on cow iim mveees venre sel
72112400 210100 dalnnta sentiia Sriwer Merii. Sileeaa Struii Sowee
meme 6.
10 1ore120 sttoet cicuus anores awlesie showes wwcero apreiass
bottom 1000
bottom 10000
Figure 8: Bottom-n ranked data based prior stds (aa).
29

---

Published as a conference paper at ICLR 2026
top 1
top 2
top 3
top
top 5
(uneert secoy veoo eecoy ierrumeenn turter wnteee
turee cem
itauc Port Mo sotter More Moseieett dets coriccunn
117
Lunson mesr. ees cumen mecc wwr. untarr..rtee
(urtet' marrree
top 50
top 10
1i8e4s33 den [1M puort "retrric fange aveer (4773061) den [hrttinpeerr n0n021 Lindies fownt deww 1nsen Lihdios huts wittim de can (hopciss (1 can (Doras smorge semull 1191119))
Siagen budt sinper best strger Stnger firger
f nast sws singer Aud
orle
Streer bus Stnger in Sispereet a.A bed stage- 6f 201 conger Somper of 1991 strper daye strper ting comn
caro (Dosusss
car [Trostis selsen cureer (50501]] ihu (pesttee seen arthe
Canar1)) den (Dacteon hew dn (D Limieran sa1o08) two cimteett
of rost wes singe @ Josl
rus sirger 200k sa stege a lext Singen or tad stree voote it
csesta)]] [Baae No0eea))
nune s1s071) Elbtaye mayy stmea) den llanger
gpe strger 2001 strge songer sipper di cuu singer 'e Jus staprbel shapr dl bud sinen dr led Singer 3ett stage"
[Down cepsate)) can (theseniige cacaat (osee))) con [oerrnal millo Embhander 14550099)) shoo [iwwen: Sewer Jamroos) der [Theee bemne sto
top 500
top 100
pre metton phaneey scack Sepily
plonicy
eelast
"taoous auutt www.eeaa wilit ofly slan craccyy aorttyyaaa aha
prtoe prrtooe coiite oisoeity printtyy sill andonoc he cotetctio occonie hnatts
nort ram aenaoo oedeerr Sectome weaoooo peaee coo
mon 2o15
nests taltes shepp vnminihe oloetche cochhe otllery odioldiie but meticin stla to for coloticiee coder odidicti cioclaie coociacia Contoer scilksso stos pialisptins 2o
neriine motel
top 1000
top 10000
Figure 9: Top-n ranked data based on prior stds (aa). For the top 1-5 examples, the whitespace contains special characters other than "/s".
30

---

Published as a conference paper at ICLR 2026
J.2
PRIOR MEAN( )-BASED
V
5
14
16
10
1n
10
1n
1.
1
50
bottom
bottom
11
51
1n
5n
Un
1n
In
1n
In
1n
1n
5n
1n
bottom 4
bottom 3
1e
W
An
bottom 5
bottom 100
Service Name
Zin
Otboonn
claree
swtemetee Despettion
bottom 500
bottom 1000
20 A/
LEb4-1 Juptin
Jumeaa (2V
bottom 10000
Figure 10: Bottom-n ranked data based on prior mean (pa).
31

---

Published as a conference paper at ICLR 2026
top 2
top
top
0
teocorthwe stoden, plnng aoser
top 3
top 5
cortigt) Islate
coraste.
fanctionl:
w'all for thlaPva
1000
11111 101
(122)2200(22559
top 20
1g yP turnce qui 1P sucona stleg viste
1n
77
antier adrvendag lidt a saft svand art us suikra leea Hos vó qp ubei iosseee alge use wva aws
top 10
top 100
Your dhene piol ("alets" chape gast daaaaeer Selahee' tyeland aelkal Quster
Freia cue aonce arat waa ceen is veto un ahe name tht 383 mower tourr pricaiiee inz) timara prcono in vonse sereemen 11o cerato ai sener an namer aiae mrasaa ann can aeria
Taale 2 Liya Socsal boutt add tylhe 1rae ralloate "slad fatted clach
("alery" Suyart
poels" on waze teyu "aaly"
top 500
top 1000
decw ties socuece Viauro and Laac slpu tokarian
Loffre fante teu nanser byylhe ridprrune wale hiphtin Evirre Inspertinnt ndeen salie
ferenwerr
top 10000
Figure 11: Top-n ranked data based on prior mean (fa)).
32


Reviews:
[{"review_id": "Ix7yYPRjKB", "reviewer": "Reviewer_533u", "summary": "This paper proposes a prior-based filter that estimates a token's context free probability from corpus term frequencies (priors). For each document, it computes the mean and standard of token priors and drops outliers far from corpus medians, motivated by (i) frequency $\\approx$ function/content role and (ii) well-formed text having a stable lexical density. This method approximates PPL filtering while being $~1000\\cdot$ cheaper (no model inference) and often outperforms PPL-based and DSIR baseline across  nearly 20 downstream tasks. The code is reproducible.", "questions": "Can the authors add more baselines to compare and see whether other lightweights methods have shorter time and also good performance? For instance, baselines listed in related work?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Practical and speedy. This method requires only frequency counts and reported wall-clock is orders of magnitude lower than PPL-based pipelines.\n2. Clear, language agnostic intuition: This paper links token frequency to function/content roles; uses document-level $\\mu, \\sigma$ to capture lexical density.\n3. It covers 2 model sizes with 20 tasks. Extensive experiments.", "weaknesses": "1. Frequency-driven criteria can over-filter rare but valuable content (named entities, dialects, minority languages). This paper discusses some multi-lingual behavior in Section 3.4.2 but they use Chinese and English as an example. These two languages are very common in web. Broader language settings might be needed.\n2. Baseline scope might be limited. Comparisons focus on PPL and DSIR; other lightweight filters (classifier-based, memorization/dup measures, n-gram LM perplexity, quality/explicit toxicity heuristics) are not covered.\n3. Scaling to large models: The authors report results for 137M-1.5B GPT-2 like models. It's unclear whether the advantage persists at 7B-70B scales or with instruction-tuning datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TYZgwNJ7Yv", "reviewer": "Reviewer_HKmF", "summary": "This paper proposes a text data filtering method based on word frequency prior to replace mainstream perplexity-based filtering strategies. Drawing upon statistical patterns of word frequency and lexical density in linguistics, the method identifies and filters anomalous documents deviating from the overall distribution by calculating the mean and standard deviation of the logarithm of word frequency within documents. Experimental results demonstrate that this method achieves performance superior to or equivalent to perplexity-based filtering across 20 downstream tasks, while reducing filtering time to approximately 0.1% of the original method, exhibiting remarkable computational efficiency. Furthermore, the method shows good applicability in both symbolic languages and multilingual corpora, without requiring pre-trained models or manually annotated data.", "questions": "1. The method uses the median as the central estimate for “normal documents.” For corpora with highly skewed distributions or mixed multimodal distributions (e.g., a blend of professional forums and news), is the median still a robust estimator? Have other robust central measures been considered?\n\n2. The experiment demonstrated the effectiveness of processing mixed Chinese-English corpora. However, for variants within the same language (such as academic English versus colloquial English), word frequency distributions may differ. Can the method distinguish stylistic variations within the same language from genuine noise?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method employs word frequency statistics for filtering, eliminating the need for training or invoking reference models, thereby reducing implementation complexity and resource dependency.\n\n2. Across 20 downstream tasks encompassing world knowledge and common-sense reasoning, the model demonstrated superior average performance compared to baseline models.\n\n3. This method is linguistically grounded in word frequency and lexical density, possessing an interpretable statistical foundation.", "weaknesses": "1. The paper should thoroughly discuss the advantages of PPL in capturing semantic coherence and contextual dependencies. Emphasizing only its computational cost and noise sensitivity issues may be misleading.\n\n2. The author should consider incorporating additional lightweight or baseline metrics (such as character repetition rate and language model scores) for comparison to enhance the persuasiveness of the experimental results.\n\n3. The authors should strengthen the discussion on the scaling relationship between computational efficiency and corpus size. Since experiments were conducted only at a 6B-word scale, they should demonstrate feasibility at larger scales.\n\n4. Although the hypothesis of a priori approximate PPL is proposed, it lacks rigorous theoretical derivation or boundary condition analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OtqgSvt2Ko", "reviewer": "Reviewer_edXW", "summary": "This paper introduces a prior-based data filtering method as a simple, fast, and powerful alternative to perplexity (PPL) for cleaning massive web corpora used in LLM pretraining. Inspired by linguistic insights on lexical density, the method estimates token priors using corpus-level term frequency and filters documents based on the mean and standard deviation of these priors, requiring no model inference. Despite its simplicity, the prior-based filter achieves state-of-the-art average performance, outperforming PPL-based filtering across 20 downstream benchmarks. The proposed approach is also over 1000 times faster than PPL filtering and demonstrates robust applicability to multilingual corpora and symbolic languages like code and math.", "questions": "* Have the authors considered combining different methods to achieve better filtering performance?\n* Is there any scaling experiment on data filtering? The authors should test with larger data proportions and model sizes to fully demonstrate the effectiveness of the filtering approach.\n* Why did the authors choose word frequency instead of n-grams or more complex statistical metrics? Previous work in statistical NLP has shown that n-grams are often more effective than using word frequency alone.\n* Is there a detailed experiment on the filtering ratio to determine the optimal filtering threshold?\n* Would different tokenizers significantly affect the filtering results? Moreover, can the filtering method remain effective across different tokenizers?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This paper proposes a simple yet effective data filtering method that achieves efficient information filtering using only statistical features.\n* Experiments show that the method is not only fast but also outperforms PPL-based approaches in terms of performance.\n* The authors demonstrate the generality of the method, showing that it can be applied to text, code, and math data.", "weaknesses": "* The experiments are conducted only on a 3B-scale dataset, which is too small to be convincing. I believe the authors should use a larger dataset (e.g., over 100B) and test on more complex tasks to better demonstrate the superiority of the proposed data filtering method.\n* In terms of writing, Chapter 2 includes excessive discussion of basic NLP concepts such as word frequency. I suggest that the authors reduce such background explanations or move them to the appendix.\n* The authors only test on unigrams and do not extend the analysis to n-grams. I believe they should further discuss the filtering effectiveness for n-grams.\n* The paper lacks experiments on the choice of filtering thresholds and filtering ratios. I suggest that the authors conduct additional experiments on these two hyperparameters to determine the optimal settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dW8r6eQyt8", "reviewer": "Reviewer_8chv", "summary": "The paper proposes a prior-based data filtering approach that replaces perplexity-based filtering with a simple frequency-based proxy. By leveraging corpus-level token priors (term frequencies) and their document-level mean and variance, the method identifies noisy or ill-formed data efficiently. It achieves comparable or better downstream performance than PPL-based methods while being 1000x faster.", "questions": "-How sensitive are results to tokenizer choice or token granularity?\n- Could the method over-filter rare but meaningful content?\n- How does this interact with deduplication or semantic filtering pipelines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Efficient and elegant alternative to PPL filtering.\n- Strong empirical results across 20 benchmarks and two model sizes.\n- Grounded in interpretable linguistic insights (lexical density, word frequency).\n- Scalable and robust, it extends to code and math data.", "weaknesses": "- Results mostly on English and related datasets; generalization to other scripts unclear: The experiments are focused almost entirely on English or similar languages, so it’s hard to tell how well the method would hold up for other scripts or morphologically rich languages. Since the approach depends on token frequencies, it might behave very differently for languages with complex segmentation or compounding (like Japanese or Turkish). Even a small multilingual check would help back up the broader claims.\n- Some claims (e.g., automatic \"learnability\" detection) lack quantitative rigor: The paper argues that the method can automatically detect “learnable” data, but this idea isn’t really supported by clear evidence. There aren’t concrete metrics or controlled experiments showing that the filtered data is actually easier for a model to learn from. As it stands, the claim feels more intuitive than proven.\n- May remove low-frequency, high-value data: Because the method penalizes documents with rare token patterns, it could unintentionally throw away useful or unique data like text from small domains, rare languages, or creative writing. This might reduce the diversity of the pretraining corpus, especially in the long tail where valuable but infrequent patterns live. A short analysis of this trade-off would make the results more convincing.\n- Some sections read more as post-hoc rationalization (linguistic grounding) than rigorous analysis.\n- Limited novelty from a machine learning standpoint", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)