How to train data-efficient LLMs
Authors: Noveen Sachdeva
Benjamin Coleman
Wang-Cheng Kang
Jianmo Ni
Lichan Hong
Ed H. Chi
James Caverlee
Julian McAuley
Derek Zhiyuan Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: data
sampling
data efficiency
LLMs
data curation
data quality
Primary area: foundation or frontier models, including LLMs

Abstract: The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, \ie, techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, AskLLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose density sampling, which models the data distribution to select a diverse sample. Testing the effect of $22$ different data curation techniques on the pre-training of T5-style of models, involving hundreds of pre-training runs and post fine-tuning evaluation tasks, we find that AskLLM and density are the best methods in their respective categories. While coverage sampling techniques often recover the performance of training on the entire dataset, training on data curated via AskLLM consistently outperforms full-data training---even when we sample only $10$\% of the original dataset, while converging up to $70$\% faster.

Published as a conference paper at ICLR 2026
How TO TRAIN DATA-EFFICIEE LLMs
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni Lichan Hong, Ed H. Chi & Derek Z. Cheng
Google DeepMind
{noveen, coleman,,, wckang, jianmon, lichan, edchi, zcheng) @ggooggl com
James Caverlee
Julian McAuley
Texas A&M University
University of California, San Diego
caver cse. aamu.edu
jmcaul ee@ucsd..u.c..... edu
ABSTRACT
The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, Le., techniques that aim to opti- mize the Pareto frontier of model quality and training resource/data consumption. We seek understand the tradeoffs associated with data selection routines based on (i) expensive-to toteeee-to-oopuete data-qu qualittuaatyy estimates, and (ii) maximization of coverage and diversity- based measures in the feature space. Our first technique, A,,, leverages the zero-shot reasoning capabilities of instruction-tuned L.LMs to directly assess the quality of a training example. To target coverage, we propose DENSITY sampling, which models the data distribution to select diverse sample. Testing the effect of 22 different data curation techniques on the pre-training of TS-style of models, involving hundreds of pre-training runs and post fine-tuning evaluation tasks, we find that ASK-LLM and DENSITY are the best methods in their respective categories. While coverage sampling techniques often recover the performance of training on the entire dataset, training on data curated via ASK-LLM consistently outperformy full-data maining-eved when sample only 10% of the original dataset, while converging up to 70% faster.
INTRODUCTION
Large language model (LLM) pre-training is perhaps the most data- and compute-intensive task attempted by the machine learning community to date, with impressive capabilities primarily being accomplished by training massive transformer architectures trillions of tokens of text (OpenAI, 2023; Touvron et al., 2023b; Gemini et al., 2023; Comanici et al., 2025; Singh et al., 2025).
But even these incredibly capable L.MMs are subject empirical scaling laws, which predict sharply diminishing returns from a linear increase in model- or data-size (Hoffmann et al., 2022; Kaplan al., 2020). Power-law scaling therefore acts as a soft limit on model quality, beyond which is prohibitively expensive to drive performance by scaling up the data or model. At the same time, Sorscher al. (2022)-in the context of vision pre-training-shor that we can significantly improve the power law constants in the aforementioned scaling laws if we prioritize important training examples using some robust notion of data quality or impact.
A similar call for cnta-curation is also apparent in the context of training LLMs, where our largest models are quickly approaching their capacity and data thresholds. LIMA (Zhou et al., 2023) showed that LLaMA-65B (Touvron et al., 2023a) can be better aligned with human preferences when trained on a set 1,000 carefully selected fine-tuning prompts, compared to training on as much as 52,000 unfiltered examples. Tirumala et al. (2023) recently conducted large-scale data-efficient pre-training evaluation, showing that a 6.7B OPT model (Zhang et al., 2022) can converge up to 20% faster on data curated by technique based on stratified cluster sampling. The Phi-2 experiments also suggest that when data curation is performed at a human- numan-expert level (e.g., by textbook editors), models can outperform baselines that are up to 25x larger (Javaheripi et al., 2023).

---

Published as a conference paper at ICLR 2026
Perplexity (HQ)
4.10
4.20
86.52
speedup
Super-GLUE
-3
GLUE
82
07.53
3.00
80
3.80
speedun
180
200
4.50
500
00:50
400
200
400
530
Training Tokens (Billion)
Training Tokens (Billion)
Training Takens (Billion)
Full data
Random
ASK-LLM (XL)
Figure I: Data-efficient pre-training run of T5-Large (800M) using ASK-LLM with Flan-T5-XL as the data quality scorer. Training on 60% of the original dataset, ASK-LLM is able to train T5-Large both better and 70% faster, compared training on 100% of the dataset.
Data curation routines can be fundamentally characterized as selecting training samples for quality, coverage, or some mixture of both (Figure 2). In this work, we seek to understand how quality and coverage affect the data efficiency of LLM pre-training. Our core research question is:
"Are cheap-to-comput heuristies like maximum-covve coveraae enough to pre-train a SoTA LLM, or are there real benefits from costly samplers that carefully evaluate the quality of each example?"
This question is crucial to answer because data-curation algorithms can improve the Pareto frontier of the -moda-ttt--molllquullliy tradeoff, directly addressing the bottleneck of power-law scaling by enabling higher-quality models to be trained using less data. Data curation also unlocks new tradeoffs between training time, inference cost, data collection effort, and downstream performance. For example, if we consider the compute-constrained (singleeepoch) regime, a data-efficient L.M training routine may reach the desired performance using only X% of the data (corresponding to a <X% training speedup).
Despite considerable interest from the community for building data-efficient training meth- ods (Sorscher et al., 2022; Paul et al., 2021; Coleman et al., 2020; Jiang et al., 2019; Katharopoulos & Fleuret, 2018), large-scale analyses of data pruning strategies are rare because of the extreme computational cost- esse-espciallly in the context of LLM pre- training.. To be more specifie, extensive comparative study necessarily entails pre- pre-training (i) various sizes of LLMs, (ii) for a variety of data sampling rates, (ii) obtained through various pruning strategies. Further, downstream evaluations for LLMs also frequently involve fine-tuning, which is resource intensive in itself.
Contributions. We hypothesize that the roles of coverage and quality depend on the stage of training. size of the model, and the sampling rate. To understand the coverage/quality design choice better, we develop new data-efficiency routines that independently (and solely) target quality and coverage. Our ASK-LLM sampler prioritizes high-quality and informative training samples by asking proxy L.M. Our DENSITY sampler seeks to maximize the coverage of latent topics the input dataset through a diversified sampling procedure. To summarize, our contributions are follows:
ASK-LLM sampling. We develop AsK-LLM, data curation technique that can train better models (vs. training on the entire dataset) even after removing up to 90% of training samples, while also consistently outperforming other well-established data curation routines. Furthermore, we find ASK-LIMM also promotes data-efficiency during training (Figure 1).
Exhaustive benchmark. We implement 22 different sampling strategies for pre-training T5 L5-Le (800M) and T5-Small (60M) on 524B tokens and evaluate them on 111 downstream evaluation tasks. This leads to total of 220 pre-training and 1, 100 distinct fine-tuning runs.
New insights. By analyzing the differences between ASK-LLM and DENSITY sampling, we study the role of coverage, quality, and sampling cost in LLM pre-training. We support our conclusions with additional studies of the convergence rate, correlations between sampler outputs, and impact of sampling cost on downstream performance. Our results show that while coverage sampling often recovers the performance of training the full dataset, ASk LLL (quality filtering) can often exceed it. These experiments suggest that LLM--baed data quality raters are a worthwhile and effective way to drive performance in pre-training.
2

---

Published as a conference paper at ICLR 2026
SemDeDup
Protorypes
Quality
Ask-LLM Perplexity Q-Classifier
DSIR
Density
Coverage
Figure 2: While there is no inherent tradeoff between coverage and quality, samplers target these metrics on spectrum (up and to the left indicates more aggressive prioritization). See Section D for a detailed description.
2 RELATED WORK
Data selection is classical problem with well-established literature on coresets, sketching, importance sampling, filtering, denoising, and a host of other algorithms with similar goals. While we cannot possibly catalog the entire sampling literature, behind common data selection algorithms. We also describe how these algorithms have been applied machine learning, focusing language model training.
hope provide an overview of the principles
Coverage Sampling. The first class of methods maximize the coverage of the sample by selecting points that are evenly distributed across the entire input domain, e.g., an e-net for a Lipschitz function (Phillips, 7). Coverage sampling is motivated by the intuition that we ought to show language model the full breadth of genres, topics, and languages (Longpre et al., 2023b).
Coverage sampling is typically accomplished by embedding examples into metric space and selecting points which are mutually far from each other (Lee al., 2023). A popular implementation of coverage sampling are cluster sampling algorithms, which groups inputs based on embedding similarity and selects representatives from each group. These algorithms are popular, scalable, interpretable, and enjoy strong theoretical support (Tukan et al., 2021; Feldman et al., 2020). However, there are also recent techniques based on submodular coverage optimization (Chen et al., 2012: Indyk et al., 2014; Borsos et al., 2020), models of the data distribution (Coleman et al., 2022), discrepancy minimization (Karnin & Liberty, 2019), and deduplication through token matching / similarity hashing (Lee et al., 2022).
Many variations of cluster sampling have been applied to vision and language model training. Sorscher et al. (2022) propose the "SSL prototypes" method for vision models, which removes points that fall too close to the nearest m-means centroid. SemDeDup (Abbas et al., 2023) also removes points based this distance, but targets pairs of nearby examples, or "semantic duplicates," " and prefers points close to the centroid. The D4 sampler chains MinHash deduplication, SemDeDup, and SSL prototypes together prune both high- high-variance, sparse regions and prototypical, dense regions of LLM pre-training datasets (Tirumala et al., 2023). Coleman et al. (2020) considers k-centers submodular selection routine on the last-layer embeddings of ResNet vision models.
Quality-score Sampling. Another class of methods are based quality scores, where scoring algorithm rates every example and the sampler preferentially selects points with high scores. For example, the selection-via-pron validation loss and uncertainty scores of pre-trained model on the input (Coleman et al., 2020; Sachdeva et al., 2021). Paul et al. (2021) sample according to "EL2N score" formed by ensembling the losses 10 lightly-trained models. Ensemble prediction variance has also been used as the scoring metric (Chitta et al., 2021), have ensemble disagreement rates (Meding al., 2021). Other scores measure whether an example is likely to be forgotten (Toneva et al., 2019), memorized (Feldman & Zhang, 2020), un-leamable (Mindermann al., 2022).
(SVP) algorithm determines the importance of an input using the
In the context of pre-training LLMs, perplexity-filete filtering is one of the arguably most used quality- scoring technique, which prioritizes samples with low perplexity or conversely filters out highly
3

---

Published as a conference paper at ICLR 2026
surprising examples (Wenzek et al., 2019; Marion al., 2023; Muennighoff al., 2023). Notably, recent advancements in cheaper to run model- based training- run simulators for LLMs can be used to estimate the perplexity of training sample instead of running LLM inference (Guu et al., 2023). Another group of methods selects training data that minimizes the distance hetween the distribution of selected data and handcrafted high-quality data source (typically wikipedia and books). Typical ways are to do this in a feature space (Xie et al., 2023b) or by training contrastive-style classifer (Radford al., 2019; Anil et al., 2023; Javaheripi et al., 2023). Similar ideas have also been explored for optimizing the data mixture weights for pre-training (Xie et al., 2023a).
More recently, the community has decomposed the LLM data efficiency problem into quality- data filtering and data mixing (Li et al., 2024). There are variety of methods for each setting, including score-based re- weighting of the corpus (Kamath et al., 2025; Shukor et al., 2025) and aggregation of multiple rubric-based scores for filtering (Wettig et al., 2024). Data quality scores can also be derived from the evaluation results of small proxy L.MSs trained on data subsets (Diao et al., 2025), and there are recent attempts to combine quality and diversity measures (Liu et al., 2025).
3
METHODS
We propose two samplers, ASK-LLM and DENSITY. These samplers have significantly different costs- AsK-LLM requires an LLM inference call for each training sample, whereas DENSITY is based on a diversified sampling routine that is cheaper than even clustering the dataset. They also exhibit substantially different selection behavior: ASK-ILMM conducts a highly nuanced and contextual quality evaluation for each sample, while DENSITY asks whether we have already sampled many similar examples. By studying samplers on extreme ends of this spectrum, we hope to better understand the salient factors for LLM data curation.
Ask-LLM prompt
pretzsining-. -- datapoint.
Dows the previnus paragraph demeroated within Malal and wike contoin informative signal for pre co-trrining brgeettnguugg model? An informative datapolnt should be wol--tormttted, contsin
knowledge rocist scutst contont.
strietly
harmful
OPTIONS:
Sampling P("yes" prompt)
Figure The ASK-LLM prompt to obtain each example's 's sampling score.
3.1 ASK-LLM SAMPLING
Intuition. Our intuition is that humans can easily identify commonly occurring failure modes in state-of-the-art data quality scorers. Hence, it should be possible to correct these mistakes using the reasoning capabilities of modern instruction-tuned LLMs. To do so, ASK-LLM, we directly prompt instruction-tuned proxy LLM with the prospective training example for the sampling decision (Figure 3). We take the softmax probability of the token "yes" as the estimated data-quality score. This procedure avoids the following common failure modes of perplexity filtering (i.e., keeping the lowest perplexity examples). See Section H) for qualitative analysis.
Contextuality. Perplexity filters often select samples that lack context, e-g., containing questions without answers (Examples 11, 12, 15). ASK-LLM correctly identifies that these examples do not provide new information.
Nonsense. Perplexity filters can select examples that repeat common words / phrases (Examples 14 and 15), likely because these common word combinations have high likelihood.
Niche examples. Perplexity filters can reject niche topics that are otherwise informative, well-written, and contain useful tail knowledge of the world. Example 17 contains detailed information about Manchester art installation but is assigned a high perplexity, likely because it contains uncommon (but valid) word combinations. Examples 20-22 display similar behavior for other niche topics.
3.2 DENSITY SAMPLING
Intuition. Our intuition is that the data distribution provides strong coverage signal. High- probability regions contain "prototypical" examples-ore ones with many near-duplicates and strong representation in the dataset. Low-probability regions will contain outliers, noise, and unique/rare
4

---

Published as a conference paper at ICLR 2026
inputs. If we wish to maximize topic coverage, should boost the signal from under-represente portions of the imput domain and downsample redundant, high-density information.
The key difficulty for our DENSITY sampler is to accurately estimate example's 's local density. Like Tirumala et al. (2023) (D4), we assume access to embeddings from a pre-trained LLM. However, we depart from the traditional approach of clustering and opt to sample based kernel sums. Given a dataset D of embeddings and a kernel k(x,, y), we estimate the density using the following score.
score(y) =
Σ RED
kx (2, 9).
is a smoothing parameter called the kernel bandwidth that controls the scale of the points' effects. To reduce the complexity from O(N2) to O(N log N). we use recent breakthroughs from the algorithm community to approximate the sum (Siminelakis et al., 2019; Coleman & Shrivastava, 2020). Our method resembles that of Coleman et al. (2022), except that (i) we adopt a two-pass sampling algorithm with stronger theoretical guarantees (Theorem C.2) and (ii) we perform the density estimation in the latent space of the model, rather than using Jaccard distances on n-grams.
3.3 SAMPLING TECHNIQUES
DENSITY and ASK-LLM both scoring methods that reduce an example a floating point value that measures coverage or quality. Once we have these scores for complete dataset of training samples, we consider two ways to select examples.
In our experiments, the DENSITY sampler uses IPS to maximize the coverage of the dataset.' For our ASK-LLM filter, we adopt top-k sampling because we expect the "yes" probability be a reliable and strong measure of quality.
3.4 RELATIONSHIPS BETWEEN METHODS
DENSITY, Perplexity, and Loss. When a language model is trained to minimize perplexity, the L.M itself is data distribution model. Therefore, the perplexity and loss filtering approaches of Marion et al. (2023), Muennighoff et al. (2023), and other authors can be viewed as model-based density sampling. However, our sampler measures the density of the training dataset in latent geometric space, while perplexity measures the likelihood under the scoring model. The samplers also differ terms of decision complexity. Thanks to the capacity the LLM, a perplexity filter can make highly-nuanced decisions between two texts on the same topic. On the other hand, our DENSITY sampler constructed from a simple nonparametric density model (Rosenblatt, 1956) that does not have the capacity to distinguish examples at such granular level.
and Perplexity. Perplexity filters exhibit a strong in-distribution bias, making decisions based on the data used to train the scoring model (not the dataset we wish to sample). By using the LLM for quality evaluation rather than likelihood estimation, our sampler can escape this bias because the additional context and alternative task change the sampling distribution. This occurs when the ASK-LLM and perplexity models are the same size.
DENSITY and Clustering. The kernel sum procedure at the core of DENSITY operates embedding- similarity relationships in a similar way to SemDeDup and SSL prototypes. Indeed, near-duplicate detection can be viewed Mitzenmacher, 2006). Outlier rejection, which motivates the "nearest-to-centroii heuristic of SSL prototypes, also has intimate connections with density estimation (Schubert al., 2014).
discretized version of similarity-based density estimation (Kirsch
Intuition. Perplexity should be viewed as "difficulty" or "quality" score rather than as a coverage- maximizing score. Our ASK- -LLM sampler should be viewed as a contextualized quality score that incorporates reasoning. Our DENSITY sampler is a pure "coverage" score in the latent representation space, while SemDeDup, and SSL Prototypes all incorporate quality outlier filtering to some extent (e.g., by preferring points near / far from a centroid).
We also implemented top- K and bottom-K sampling, but these samplers do not maintain coverage and perform poorly.
Note that ASK-LLM may also incidentally improve coverage because it does not suffer from in-distribution
bias.
5

---

Published as a conference paper at ICLR 2026
4 EMPIRICAL SETUP
Models. We pre-train T3-style models (Raffel et al., 2020), which belong to the encoder-decoder family of Transformer models and offer competitive performance on many tasks (Shen et al., 2023). See Phuong & Hutter (2022) for formal introduction to various Transformer model configurations. We train T5-Small (60M) and L3-Large (800M), reusing all of the training settings from the original T5 implementation except the batch size (2048 + 1024). We train on batches of 1024 sequences of length 512 for 1M steps. All our experiments are conducted on the TPUv5e architecture.
For perplexity filtering, experiment with five different language models: T5-(Small, Base, Large, XL, X. For ASK-LLM's quality-scoring model, we consider the FLAN-T5 (Longpre al., 2023a) models of the same five sizes (Small to XXL), as well as the isstruction-tued Gemma-7B model (Team et al., 2024) (named G.7B for brevity).
Datasets. We use the C4 dataset (Raffel et al., 2019) available under the ODC-By license, which was also used for pre-training the original T5 family of models. The C4 dataset is a version of the Common Crawl-a publicly available archive of web-text-that has been pre-processed using several heuristics (Raffel et al., 2020, Section 2.2). In its entirety, C4 contains 184B tokens. We use our algorithms (see Section for a list) to sample {10, 21, 40, 800,800 % of C4.
Because low sampling ratio yields exceedingly small datasets, we choose train in the iso-compute setting, i.e., training all models for exactly 524B tokens. This results in more epochs (repetitions) at smaller sampling rates. We believe this gives each data curation method equal chance to maximize model performance, and not penalize methods that sample a small number of high-quality repeatable tokens vs. large number of non-repeatable tokens. See Section D, Figure for a demonstration of this process.
Evaluation. We use 111 downstream evaluation tasks to assess diverse performance indicators for pre-trained LLMs. On a high-level, conduct post-finetuning evaluation GLUE and SuperGLUE for natural language understanding capabilities, as well as benchmark on various knowledge, reason- ing, and Q/A evaluation sets after finetuning the FLANv2 dataset. See Section F for complete list and further details.
In addition to these individual tasks, to compare singular normalized average performance im- provement over all downstream evaluations, we devise a metric called "Effective Model Size." challenging to concisely summarize performance using single measure, primarily because our evaluation consists of 111 individual tasks, all of which respond at different rates to data and model optimizations.
Inspired by the LLM scaling-law literature (Hoffmaann et al., 2022: Muennighoff et al., 2023), the "Effective Model Size" " metric measures the effective model size by extrapolating on parametric fit of the "number of parameters vs. downstream eval" trend, averaged over various downstream tasks. See Section E for a formal definition, as well as the fitted scaling laws for the original T5-models on the downstream tasks used in this paper. To summarize, this metric gives principled answer to the question, "If using a technique leads to x performance, what size LLM achieves the same x performance if the technique is not used?"
EXPERIMENTS
Does reasoning improve efficiency? Figure 4 shows that ASK-LLM trains 800M models to an equivalent performance, as if we were to train 1.5B models on the original C4 dataset. ASK-LLM consistently outperforms perplexity filtering (and coverage-man evveragemaximiziig baselines), despite having access to scoring model of the same model capacity (XL). Similar findings hold for training efficiency (Figure 5). ASK-LLM converges faster than perplexity filters, both in average (expected final performance over all proxy model sizes) and pointwise for the best configuration (Small and XL. for training S5mll and T5-Large).
Figure 7 further demonstrates that prompting adds critical information the sampler not present in perplexity: clear behavioral difference, we conclude that reasoning and context are crucial ingredients. We
scores show no correlation with the perplexity scores. Based on this

---

Published as a conference paper at ICLR 2026
Effectivn Model Siz
150
80.00
$8592
80.2
15 13-urgo
Avg.
Effective Model Size
Aeg. (Post.FLAM)
28.43
$1.55
26.30
27.53
27.00
28.40
23.50
70.50
40
75
250
40
113
100
204
Dataset
(Biillon Tokenss)
Tokens)
Dataset (Billion Tokens)
Full data
Density
Ask-LLM (Base)
Ask-LLM (G.78)
Perplexity (XL)
Q-Clssifiier
Random
- Ask-LLM (Small)
AsK-LLM (XL)
Perplexity (Small)
Prototypes
DSIR
Figure 4: Tradeoff between data quantity (number of unique tokens in the sampled dataset) and model quality for (top) T5-Large and (bottom) TS-Small pre-training. Each point corresponds to a converged pre-training over sub-sample. To avoid clutter, not all sampling methods or evaluation metrics are shown Figure or Table I; see Section for the results of all 22 samplers and metrics.
Table 1: Comparison of sampling algorithms at a fixed sample size. For each sampling strategy, we sample the dataset to X% of the original size and pre-train T5-Large for 524B tokens. This table is cross-section of Figure but with more metrics.
Training config.
Downstream tasks
FLAN Instruction Tuning
Effective Model Size
Super GLUE
CNN/
Sampler
SQuAD
MMLU
Reasoning
QA
DM
T5-Large
86.7
40.7
21.6
T5 Large
Random
18B
713M
88.4
82.3
20.8
85.9
41.8
33.6
20.6
71.5
T5-1 arge
Density
18R
802M
88.0
80.5
20.9
86.9
42.6
35.5
19.1
70.6
TS-Large
Prototypes
18B
423M
87.7
80.5
20.4
86.6
36.7
33.0
17.6
66.0
T5-Large
Perplexity (Small)
18B
301M
87.6
80.2
20.5
85.2
36.8
33.8
17.7
60.9
15- L5-Large
DSIR
188
476M
87.3
81.7
20.7
83.4
39.8
33.3
22.2
65.0
TS-Large
Q-Classifier
18B
797M
88.7
83.6
20.8
87.7
40.5
35.0
20.2
65.4
T5-Large
Ask-LLM (G.7B)
18B
1.5B
88.2
82.5
20.8
87.8
44.2
37.1
22.7
78.2
expect prompting techniques such as chain-of-thought reasoning (Wei al., 2022) to further drive performance.
When expensive scores justified? Observing the effective model size while training T5-Large, Figure suggests that other samplers start performing well only at larger sample ratios (2 60%), with performance very close ASK-LLM. On the other hand, at smaller sampling ratios, AsK-LLM tends to significantly outperform both coverage-basee samplers, as well as cheaper alternatives for data-quality scoring like Q-Classifier and DSIR (Section D). Hence, the higher costs of LLM-based filters are most justified in two scenarios: (i) improving full-data performance, where quality filtering by removing the lowest-quality data is the main way to push the upper limit model performance; or (ii) in the low-data regime, where keeping only the highest-quality data drives the most model performance compared to other sampling strategies.
To provide concrete numbers, we report the accelerator-hour costs of training and scoring in Table 2. As described Section 4, our experiments were run in the iso-compute setting where the goal is to maximize performance at fixed number of training tokens. To estimate the cost savings, instead must look at the amount by which we could reduce the training cost while maintaining neutral performance. As example cost-benefit analysis, consider setup where AskLLM-T5-X scores are used to train T5-L. Figure suggests that, conservatively, a 44% reduction the training horizon is possible without performance loss. This corresponds cost of 56% x44 10 ~ 23.44
7

---

Published as a conference paper at ICLR 2026
Table Cost of training and scoring on the C4 dataset. We report the average of 30 training runs.
Metric (Acoderator-ttte) T5-XXL
TS-XL.
TS-Large
TS-Base
TS-Small
Scoring Cost (C4) Training Cost (C4)
10.0
24.0
9.3
TS-Large
TS-Large
T5-Large
Perpl exity (HK2)
4.4
89.5
03.0
Super-GLuE
4.2
07:5
07.0
0.0
06.
85.0
230
202
250
592
150
200
500
300
T5-Small
T5-Small
T5-Small
Perplxkky (HQ)
8.5
7.8
Super GLUE
10
79.
7.0
200
230
505
250
400
450
505
250
200
200
300
400
300
100
200
300
400
200
Training Tokens (Billion)
Training Tokens (Billioo)
Training Tokens (Billion)
Ask LLM (Avg.)
LLM (XL)
Ask LLM (Small) Perplexity (Small)
Full Data
Perplexity (Avg.)
Perplexity (XL)
Figure 5: Training efficiency comparison between ASK-LLM and Perplexity filtering, shown by comparing performance of intermediate checkpoints. The (Avg) represents performance averaged across (i) scoring model sizes, i.e., T5-[Small, Base, Large, XL, XXL]; and (ii) sampling ratios, iee. (10, 20, 40, 60, 80) %. The (Small) and (XI.) series show the T5-(Small, XI.) runs, averaged only over the sampling ratios.
accelerator-hours for AskLLM-sampi training compared to cost of 24 accelerator-hours without AskLLM, with the benefit that the AskLLM scores can reused by future runs.
We also observe that random sampling is strong baseline, aligning with recent observations in the literature. Guo et al. (2022a) found that only three methods outperformed random sampling a computer vision benchmark of 15 algorithms,, and Ayed & Hayou (2023a) prove the existence of adversarial problem instances random sampling is optimal. These results higlight the significance of ASK-LLM's gains.
Effect of scoring model capacity: Figure demonstrates clear scaling trend for
quality-scoring model: larger scoring models are increasingly beneficial as the scale of the to-be- trained L.M increases. Perplexity filters do not seem to exhibit such trends. The strongly consistent trend suggests that AsK-LLM performance will improve with stronger quality-scoring models whether obtained by fine-tuning, chain-of-thought prompting, or size. However, we still observe compelling performance even when training large models on data chosen by small ASK-LLM models. For example, ASK-LLM (Base) outperforms perplexity filtering with any scoring- model (including T5-XXL) for most sampling ratios (Section F).
Do samplers select different examples? We computed the Kendall Tau rank correlation between samplers on 500k examples (Figure 7), finding significant and interesting differences. For example, the "TS-LLarg" rows show that (i) L5-Large outputs perplexity scores similar to T5-Small early in training, but becomes progressively more nuanced on the path from 20k to 700k training steps, and (ii) perplexity, density, and AsK-LLM select for wildly different criteria, with almost no ranking correlation. This supports our hypothesis that DENSITY prioritizes coverage, representing the original training objective better than any other method besides uniform sampling ( (eection G); perplexity re- weightghts the training objective to up- weigtt data regions that are on-distribution for the proxy model; and ASk- LLMM up-weights the regions that are identified as "high-quality" by the prompt.
8

---

Published as a conference paper at ICLR 2026
Ask-LLLM Sampling
-1.05
-3.75
-0.52
-0.35
0.00
0.25
0.50
0.75
1.00
90)
ARank (T)
031
5650
Small
Base
Perplexity Filtering
La.Lare (304)
T-large (1304)
ARank (1)
Laaagg (3308)
(230))
Lanag (7908)
Demsly
Frenesppe 03 20e (al
Small
Base
Large
Gornra 75
Snd
Pin Us
Han 15.
ESN75500
15
15-00
75.00
TF 1000 DM6
upaar 130066
TP uaaaa 18006
To Larg (000s
Larg 17006
71000000
Proxy
Figure 6: Change in ranking of scor- ing models. A positive ARank indi- cates that the scorer's 's task-averaged rank increased when training T5- large T5-Small.
Figure 7: Kendall's Tau correlation amongst the scores from quality filters (first 8), perplexity filters (next 10), and coverage-based samplers (last. 3).
DISCUSSION
Amortized scoring. The ASK-LLM and perplexity scorers require considerable computation one LLM inference call for every training sample-which is concerning from both a carbon-emission and cost perspective (Strubell et al., 2019). However, we argue that the scoring costs are amortized many pre-training runs, which together cost significantly more than the ASK-LLM inference calls (Luccioni et al., 2023). In practical systems, cheaper samplers scoring models can also pre-filter examples for one-time cost, this has historically not been the case. We therefore view quality scores as a long-term investment. See Section C.I for deeper discussion about the cost of ASK-LLM scoring.
expensive scorers. While LLM pre-training is often thought of as
LLM-Based Data Refinement. Recursively training on model-generated data causes degredation in both diffusion models and LLMs, inciting concerns about whether the internet will remain viable source of training data (Shumailov et al., 2023; Alemohammad et al., 2023; Briesch et al., 2023), The primary mechanism for this degredation is a mode-collapse behavior where the self-consumption loop amplifies the in-distribution bias of the LLM. It is therefore somewhat surprising that LLMs are so effective at deciding which training data to consume.
We rigorously examine the bias characteristics of ASK-LLM in Figure 11 (Appendix G), finding much greater variance topic affinity among the T5 models when the score obtained via ASK- LLM versus perplexity. Therefore, we conclude that the ASK-LLM biases depend more strongly on the fine-tuning/ / prompting the model than on in-distribution bias (since all models the T5 family share the same training distribution bias). These results raise important questions about whether LLM-based filters can function as an intervention in the self-consumption loop, allowing LLMs to self-improve.
LLM and Prompt Sensitivity. While we are unable to present rigorous ablations of the prompt, we have observed poor calibration with numeric score outputs and Likert rubrics (unless dighly-detailed, case-by-case grading criteria are provided). This results in noisy labels for borderline data and degenerate distributions that make sampling difficult. One known method improve the calibration

---

Published as a conference paper at ICLR 2026
of LLM raters is to transform scalar-output tasks into token- probabirbbabilitty tasks (Ren et al., 2023), which may explain why we observe relatively robust performance with our Pr Pr[Yes] scores. We do examine the sensitivity of scoring decisions to the choice of LLM judge, with detailed comparisons in Appendix H. The general rule that larger model sizes can distinguish quality in more nuanced ways (e.g., advertisement copy is selected by small judges but not by large ones).
One can cheaply guess whether a new prompt or LLM might yield different downstream training performance by measuring the rank correlation between scores: high correlation (> 0.8) is sufficient condition for similar performance. However, the converse is not true two judges can achieve similar results even if they disagree on many points. We hypothesize that this occurs because the performance of a model trained on sampled data is an aggregate property of the training set. In practice, we observe that the different ASK- judges are not strongly correlated on individual score pairs ( Figure 7), even though they attain similar downstream performance.
Decoder-Only Models. The experiments in this work address the data-efficiency of encoder-decoder models in the T5 family. The encoder-decoder structure is still the subject of research and industrial use, especially for applications that can employ asymmetric encoder and decoder components to exploit efficiency tradeoffs (Zhang et al., 2025). However, the vast majority of today's 's research and practical applications focus on decoder-only models.
While do not conduct experiments with decoder-only architectures in this paper, would like to note that a variant our AsK-LLM technique was used in the development of the Gemma family of models. Specifically, the AsK-LLM framework was employed to re-weight the pretraining corpus to reduce occurrences of low-quality data (Kamath et al., 2025).
CONCLUSION
We studied the performance of sampling algorithms that select high-quality data through highly- capable proxies and maximize coverage through embedding similarity. Our experiments reveal that LLM-based quality filtering yields a Parteo optimal efficiency tradeoff between data quantity and model quality, with important implications for training cost, self- iaff-improveeent, and LLM training data curation.
ACKNOWLEDGEMENTS
We sincerely thank Xinyun Chen and Kelvin Guu for their insightful feedback on early drafts of this paper. We would also like thank Sammy Jerome, Anton Tsitsulin, Peilin Zhong, and Marc Brockschmidt for their insights and technical discussions as we refined the ASK-LLM technique. We thank Zhang Li for his significant engineering contributions the project. We also thank the Gemma 3 team for being our first production users especially Alek Andreev, who helped create the opportunity, and Cassidy Hardin, who worked tirelessly to productionize ASK-LLM. Finally, wish to thank Fernando Pereira, Vahab Mirrokni, Armand Joulin, Andrew Dai, and Ya Xu for their strong leadership and support for this project.
REFERENCES
Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig. Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL
Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.1366 13649, 2023.
Sina Alemohammad, Josue Casco-Rodrigun Lorenzo Luzi Ahmed Imtiaz Humayun, Hossein Babaci, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. Self-consuming generative models go mad. arXiv preprint arXiv:2307.018800 2023.
10

---

Published as a conference paper at ICLR 2026
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, and Zhifeng Chen et al. Palm 2 technical report, 2023.
Fadhel Ayed and Soufiane Hayou. Data pruning and neural scaling laws: fundamental limitations of score-based algorithms. Transactions on Machine Learning Research, 2023a. ISSN 2835-8856. URL httph:tt/////..
/wwppreeviiiieeeeeeee net / forum? i d= RTL4 paao..
Fadhel Ayed and Soufiane Hayou. Data pruning and neural scaling laws: fundamental limitations of score-based algorithms. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856.
URL http:::////// / wppp/eeeerevveeeeeeeeeeeeee net/ forum? i dd=i I
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems, 2020.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint arXiv:23 I. /6822, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint 13Xiv: 2013.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint 17X2v 2012.
Kashyap Chitta, José M Álvarez, Elmar Haussmann, and Clément Farabet. Training data subset search with ensemble active learning. IEEE Transactions on Intelligent Transportation Systems, 13491114711 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Christopher Clark, Kenton Lee, Ming-We Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 19X. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 18X3::1803.05433 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Rejichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Benjamin Coleman and Anshumali Shrivastava. Sub-linear race sketches for approximate kernel density estimation on streaming data. In Proceedings The Web Conference 2020, pp. 1739-1749, 2020.
Benjamin Coleman, Benito Geordie, Li Chou, RA Leo Elworth, Todd Treangen, and Anshumali Shrivastava. One-pass diversified sampling with application terabyte-scale genomic sequence streams. In International Conference on Machine Learning, pp. 4202-4218. PMLR, 2022.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In ICLR, 2020.
11

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, and Others. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https::tp//............t htt.///xiiivv////ev/eee/eeeeevvt org/ abs 2507 06261.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitive sensitive hashing scheme based on p-stable distributions. Proceedings of the twentieth annual symposium on Computational geometry, pp. 253-262, 2004.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers scale. arXiv preprint arXiv:2208.07339 2022.
Luc Devroye. The equivalence of weak, strong and complete convergence in 11 for kernel density estimates. The Annals of Statistics, pp. 896-904, 1983.
Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Nemotron-climb: Clustering-baser iterative data mixture bootstrapping for language model pre-training. In Advances in Neural Information Processing Systems, 2025.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixxurs-of-expeets. In International Conference on Machine Learning, pp. 5547-5569. PMLR, 2022.
Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pea, and projective clustering. SIAM Journal on Computing, 49(3):601-657, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprini arXiv:2101.00023 2020.
Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Saan-Bapiiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:23066 1164,, 2023.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022a.
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference Database and Expert Systems Applications, pp. 181-195. Springer, 2022b.
Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 2440-2452, 2020.
12

---

Published as a conference paper at ICLR 2026
Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint arXiv:2503.08114, 2023.
Dan Hendrycks, Collin Burs,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:30016-30030, 2022.
Piotr Indyk, Sepideh Mahabadi, Mohammad Mahdian, and Vahab Mirrokni. Composable core-sets for diversity and coverage maximization. In Proceedings of the 33rd ACM SIGMOD-SIGACT- SIGART symposium on Principles of database systems. pp. 100-108, 2014.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models, 2023.
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing the biggest losers. arXiv preprint 190.0:191..00762. 2019.
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Róbert Bukete, Alex Feng, Noveen Sachdeva, Benjamin Coleman. Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari. Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin 7elle, Fric Noland, Erwin Huizenga, Fugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucinn
Harman Singh, Harsh Mchta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting. Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Mant Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid. Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan. Reza Rokni, Rob Willoughhy, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Gir- gin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evei, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle K. Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov,
13

---

Published as a conference paper at ICLR 2026
Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clément Farabet, Elena Buchatskaya, Jean- Bant-Bttstte Alayrac, Rohan Anil, Dmitry Dima Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report. CoRR, 191726 March 2025. URL https::/p//...... /oii .org .t..g//....5500... rxiv 10...19..77..
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.08300 2020.
Zohar Karnin and Edo Liberty. Discrepancy, coresets, and sketches in machine learning. In Conference Learning Theory, pp. 1975-1993. PMLR, 2019.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp.
PMLR,
2018. 8.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with single qa system. arXiv preprint arXiv:2005.0070 2020.
Adam Kirsch and Michael Mitzenmacher. Distance-sensitive bloom filters. In 2006 Proceedings of the Eighth Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 41-50. SIAM, 2006.
Alycia Lee, Brando Miranda, and Sanmi Koyejo. Beyond scale: the diversity coefficient data quality metric demonstrates Ilms are pre-trained on formally diverse data. arXiv preprint arXiv:2306. 13840, 2023.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers). pp. 8424-8445, 2022.
Jeffrey Li, Alex Fang, Georgios Smymis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu
Yu Hsich, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu, Thao Nguyen, Igor Vasiljevic. Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dav, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: in search of the next generation of training sets for language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 97983313 114385..
Fengze Liu, Weidong Zhou, Binbin Liu, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Bingni Zhang, Xiaohuan Zhou, Taifeng Wang, et al. Quadmix: Quality-diversig balanced data selection for efficient Ilm pretraining. arXiv preprint arXiv:2504. 1652504441111 2025.
Zichang Liu, Zhaozhuo Xu, Benjamin Coleman, and Anshumali Shrivastava. One-pass distribution sketch for measuring data heterogeneity in federated learning. In Thirty- Conference on Neural Information Processing Systems, 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv: 13688, 2023a.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13166 13169, 2023b.
14

---

Published as a conference paper at ICLR 2026
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253): 1-15, 2023.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprint (ttv:2309.04564, 2023.
Kristof Meding, Luca M Schulze Buschoff, Robert Geirhos, and Felix A Wichmann. Trivial or imposibbe-ddichotomous data difficulty masks model differences (on imagenet and beyond), arXiv preprint arX1v:2110.05922, 2021.
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.1577 2021.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649, PMLR, 2022.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305.16264 2023.
Jianmo Ni, Gustavo Hernández Ábrego. Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2/08.08877 2021.
OpenAl. Gpt-4 technical report, 2023.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint 10110 2021.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Jeff M Phillips. Coresets and sketches In Handbook of discrete and computational geometry, pp- Chapman and Hall/CRC, 2017.
Mary Phuong and Marcus Hutter. Formal algorithms for transformers. arXiv preprint arXiv:2207.09258, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1 (4855 55551, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 0606.0550, 2016.
Jie Ren, Yao Zhao, Tu Vu, Peter J. Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation large language models. In Javier Antorán, Arno Blaas, Kelly Buchanan, Fan Feng, Vincent Fortuin, Sahra Ghalebikesabi, Andreas Kriegler, Ian Mason, David Rohde, Francisco R. Ruiz, Tobias Uelwer, Yubin Xie, and Rui Yang (eds.), Proceedings on "I Can't 't Believe It's Not Better: Failure Modes in the Age of Foundation Models" at NeurlPS 2023 Workshops, volume 239 of Proceedings of Machine Learning Research, pp. 49-64. PMLR, 16 Dec 2023. URL https: proc/edo inga ngs ml p.rssss////eeeee v239 / html.
15

---

Published as a conference paper at ICLR 2026
Murray Rosenblatt. Remarks some nonparametric estimates of density function. The annals of mathematical statistics, pp. 832-82- 822-837,1 1956.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. Transactions Machine Learning Research, 2023. ISSN 2835-8856. Survey Certification.
Noveen Sachdeva, Carole-Jean Wu, and Julian McAuley. Svp-cf: Selection via proxy for collaborative filtering data. arXiv preprint ar0iv:2107 04984, 2021.
Noveen Sachdeva, Zexue He, Bang-Cheng Kang, Jianmo Ni, Derek Zhiyuan Cheng, and Julian McAuley. Farzi data: Autoregressive data distillation. arXiv preprint arXiv:2310.0998. 2023.
Erich Schubert, Arthur Zimek, and Hans-Peter Kriegel. Generalized outlier detection with flexible kernel density estimates. In Proceedings of the 2014 SIAM International Conference on Data Mining, pp. 542-550. SIAM, 2014.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. Mixture-of-experts meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305. 14705, 2023.
Mustafa Shukor, Louis Bethune, Dan Bushridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures. In NeurIPS, 2025. URL https: //arxiv....///////////////aaavva httgg/bb/25500 09404.
Ila Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget.(5 2023). URI: htts:://axxii org/abs/2305.17492 2023.
Paris Siminelakis, Kexin Rong, Peter Bailis, Moses Charikar, and Philip Levis. Rehashing kernel evaluation in high dimensions. In International Conference on Machine Learning, pp. 5789-5798.
PMLR, 2019.
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, and Others. Openai gpt-5 system card, 2025. URL https: ///aaxia aaraiivvoooo org/ abs/ 601 03267.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536 2022.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04611 04511, 2022.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint 10Xiv:9906002244 2019.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint 40Xiv:2403.08295, 2024.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving 1lm pretraining via document de-duplication and diversification. arXiv preprint 2.08. 2023.
M. Toneva, A. Sordoni, R. Combes, A. Trischler, Y. Bengio, and G. Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
16

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXin:2307.09288 2023b.
Murad Tukan, Cenk Baykal, Dan Feldman, and Daniela Rus. On coresets for support vector machines. Theoretical Computer Science, 890: 11-111, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 2018.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Lilian Weng. Large transformer model inference optimization. Lil' Jan 2023. URL https: wpt.aiiiiiiiiiiii./..o..
//lilianweng github.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Conet: Extracting high quality monolingual datasets from web crawl data. arXiv preprini 19I 2019.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: selecting high-quality data for training language models. In Proceedings of the 41st International Conference on Machine Learning, ICML.24. J.Lg. 2024.
Dominik Wied and Rafael Weißbach. Consistency the kernel density estimator: a survey. Statistical Papers, 53(1):1-21, 2012.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305. 10429, 2023a.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2332.03/66 2023b.
Sang Michael Xie. Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36, 2024.
Biao Zhang, Fedor Moiseev, Joshua Ainslie, Paul Suganthan, Min Ma. Surya Bhupatiraju, Fede Lebron, Orhan Firat, Armand Joulin, and Zhe Dong- Encoder-decoder gemma: Improving the quality-efficiency trade-off via adaptation. arXiv preprint arXiv:2504.06225, 2025.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arxiv:2205 2205.0 01068, 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. arXiv preprint arXiv:2305. 1206, 2023.
17

---

Published as a conference paper at ICLR 2026
Appendices
A Impact Statement
19
Limitations
19
Algorithms
19
C.1 AsK-LLM Sampling
19
C.2 DENSITY Sampling
20
Data-curation Techniques
21
D.I Random Sampling
21
D.2 DENSITY Sampling
22
D.3 SemDeDup
................................
22
D.4 SSL Prototypes
22
D.5 Perplexity Filtering
................................
22
D.6 Text-quality Classifier (Q-Classifier)
22
D.7 Data Selection with Importance Resampling (DSIR)
23
D.8 AsK-LLM Sampling
23
Effective Model Size
23
F Downstream Evaluation Tasks
24
E.1
Perplexity
24
F22
HQ Perplexity
24
F3
GLUE
24
F.4
SuperGLUE
24
F5
CNN/DM
24
F.6
SQuAD
................................
24
F.7
FLAN Instruction Tuning
24
G Additional Results
25
G.1 (Figure 10) Quality-score Distribution for Different Samplers
25
G.2 (Figure 11) Analysis for Different Samplers' Affinity to Different Topics
25
G.3 (Figures 12 to 20) Data-quantity vs. Model-quality for Different Samplers
26
G.4 (Figures 21 to 29) Quality of Fresh vs. Repeated Tokens for Different Samplers
32
G.5 (Figures 30 to 36) Data-efficiency of Different Samplers
37
Qualitative Results
41
H.1 High-quality Samples Identified by ASK-LLM
41
18

---

Published as a conference paper at ICLR 2026
H.2
Low-quality Samples Identified by AsK-LLM.
45
H.3
Increasing-quality Samples Identified by ASK-LLM
48
H.4 Decreasing-qualiyy Samples Identified by ASK-LLM
50
A
IMPACT STATEMENT
While increased LLM accessibility has well-documented risks, we expect data-efficient pre-training to be a social good that reduces (amortized) carbon emissions and pre- pre-training cost while improving quality.
B LIMITATIONS
While the paper pushes the frontier of LLM-training from both quality and efficiency fronts by better curating pre-training datasets, we would also like to note the limitations of this paper that both better informs the reader hopefully guides future work. First, due to the sheer cost of training LLMs once, let alone doing data-efficiency research, (encoder-decoder and only on the C4 dataset. The transferability of our results to more popular, decoder-only models and larger datasets is still yet to be explored, and an interesting direction for future work. Next, due to T5-models' inability to code (primarily a tokenization issue), we don't t include any coding evaluations in this paper. Curating high-quality coding data an interesting and active direction of research (Gunasekar et al., 2023). Further, all of our evaluations are limited to post-finetuning (as is the prevalent setting with T5 models), hence the effect of our data-curation techniques on zero/few-shot prompting is also not clear.
only train one kind of transformer models
C ALGORITHMS
C.I ASK-LLM SAMPLING
Algorithm AsK-LLM Sampling
Input: Dataset D = {I1, [2, aND s.t. Iı E X' is the training sample in plain-text, sample size k.
scoring model M : X2; X IR
Output: Sampled data
1: Initialize list of scores S [].
2: for n 1 N do
3: prompt,, make eee-maaaeappppppee prompt(x.)
Make ASK-LLM prompts as in Figure D Use M to score x.
4: Append MM""dd(""""""" prompt,,) to S
5: return elements from D with top-k scores in S, without replacement.
Discussion the cost of AsK-LLM scoring. Even though AsK-LLM sampling results in im- pressive performance and training efficiency improvements compared to training on the full-dataset (Section G), the data quality scoring cost might seem prohibitive. On the other hand, on top of the improved results, we argue the following to be compelling points in justifying ASK-LLM's one-time-amortized data scoring cost:
ASK-LLM only requires forward passes on the entire dataset. This is much cheaper than (i) training the model itself which requires both forward and backward passes on multiple repetitions of the entire dataset, (ii) gradient- gradient-based data-curation techniques (Sachdeva & McMully, 2023; Sachdeva al., 2023) that also require backward passes, etc.
An additional benefit of the AsK-LLM framework is the ability to leverage memory-efficient, quantized LLM inference setups (Dettmers et al., 2022). This is strictly not possible, e.g., for pre-training LLMs. Notably, quantization isn' 't the only AsK-LLM-frienae recent (and future) advances in efficient inference techniques for LLMs (Weng, 2023) directly reduce the amortization cost of the ASK-LLM framework.
technique. All the
19

---

Published as a conference paper at ICLR 2026
Another benefit of AsK-LLM is the ability naively parallelize quality scoring. To be more specific, we can simply scale-up the amount of small & independent inference resources, and run inference calls for various training samples parallely. Note that inference hardware has much smaller requirements compared to, e-g. pre-training or fine-tuning requirements. This is primarily true because of no batch size requirement for inference vs. large batch size requirement while training. This enables scaling-up hardware to happen via a large number of small-compute setups (e.g.. 4 interconnected GPUs per node) versus increasing the number of large-compute setups (e.g.. 1000s of interconnected GPUs per node).
ASK-LLM also uses strictly less compute compared to teacher-student knowledge distillation based training setups (Agarwal et al., 2023). This is true simply because knowledge distillation require (i) bigger teacher model's ' softmax predictions (ii) for each token in our training data. On the other hand, ASK-LLM requires just the score of the token "yes" given the prompt.
C.2 DENSITY SAMPLING
Our density sampler is adapted from that of Coleman et al. (2022), with few critical departures:
We two-pass procedure that allows for more rigorous theoretical guarantees (and different sampling behavior).
We conduct the density estimation in the model's latent space rather than using Jaccard similarity over n-grams,
Jaccard similarities are sufficient to construct a reasonable sampling distribution for genomics applications, which are significantly more structured than natural language. However, this is not the case with text we found that sampling based on Jaccard density is no better than random. For this reason, must different kernels (p-stable rather than MinHash) and different input representations (embedding rather than n-grams).
Improvements:
However, more interesting departure from Coleman et al. (2022) is two-pass sampling procedure, which changes the behavior of the algorithm and allows for more rigorous theoretical guarantees. The original method was only able demonstrate convergence of cluster populations in the sampled dataset. While this leads to (weak) convergence for some measures of diversity, it also requires strong assumptions about the cluster structure.
Theory: We use recent result that demonstrates consistent sketch-based estimation of the kernel sum (Theorem 3.3 of Liu al. (2023)), which we paraphrase below.
Lemma C.1. Let P(x) denote a probability density function. Let P(x) denote dataset. Let
iid
k(I, y) be a positive definite LSH kernel, and let S be the DENSITY score. Then S(x) a consistent estimator for the kernel sum.
S(x)
+ i.p. N
1
Σ 4)
Z.ED
with convergence rate Ol (Tg R/R).
If we perform inverse propensity sampling using the score in Theorem C.1, we obtain a sampling procedure that outputs mniffornlyddistibbun sample.
Theorem C.2. Let (xx) be the distribution formed by (i) drawing N samples i.i.d. from distribution P. e.g. D = {II....N ~ P. and (ii) keeping x with probability proportional to 3(z). Under the conditions of Lemma C.I, Q(x) + U (xx), where U (œ) ) is the uniform distribution.
i.p-
Proof. Under the conditions of Wied & Weißbach (2012) (specifically, positive-definitensss and E integrability bounded domain), the kernel sum is a consistent estimator of the density. That is, the sum converges probability P(x).
Σ EDDD
k(z q) i.p.
P(z)
20

---

Published as a conference paper at ICLR 2026
Theorem C.1 shows that S(z) converges in probability to the sum (and thus to P(x)). By Slutsky's 's
Theorem, S(x) P()) for all x the support of the distribution (i.e. P(æ) o 0). The probability of
generating x as part of the sample is:
1 P(x)
(xx)
Pr[Seloct.r Generatez) Pr
S(x)
Because S())
for some constant c, we have that Q(x) -> c.
P(2)
Theorem C.2 demonstrates that our DENSITY sampler outputs a uniformly-distributed collection of points over the imput space (latent LLM representation space).
Algorithm 2 Inverse Propensity Sampling (IPS) via Kernel Density Estimation (KDE)
Input: Dataset D {œ1 ...-.. N of embeddings, sample size k, kernel & with corresponding
sensiive hash family H. (see Coleman & Shrivastava (2020)), hash range B, rows R, random seed
Output: Sampled data
1: Initialize: KDE sketch S < R
2: Generate R. independent hash functions hy...., hR from H. with range B and random seed s. Construct KDE estimator for D. Add Fn to the KDE estimator.
3: for 1 - do
4: for ror do
5: (r,h(a.,)+
6: Initialize list of scores S =
7: for = 1 do
Score each example T.
8: score =
9: for r = - do
Compute approximate KDE using
10:
score- Sir, (x,])
Append score/ to
12:
return elements from DD with probability p
S without replacement.
Like Sep,eDup, D4, and SSL prototypes, our DENSITY sampler requires access to embed- dings for each example in the training corpus. However, by eliminating the expensive clustering step, we eliminate a significant computational overhead. Our DENSITY sampling routine required just 80MB of memory and linear passes through the dataset to score all 364M embeddings. This is significantly less expensive than clustering.
Cost:
Tuning: We also eliminate large number of hyperparameters, improving tuning, Cluster-based samplers must choose the number of clusters, clustering optimizer and objective, and per-cluster sampling rate or deduplication similarity. Kernel density estimation, on the other hand, has just two hyperparameters: the choice of kernel and the bandwidth. We did not observe a significant performance variation among different bandwidth and kernel choices (e.g., the and cosine kernels of Coleman & Shrivastava (2020) perform nearly identically). This is likely because all positive- definite kernels enjoy strong guarantees on the distribution approximation error (Devroye, 1983).
D DATA-CURATION TECHNIQUES
D. RANDOM SAMPLING
The de-facto standard for obtaining samples large datasets where we sample training examples uniformly at random. Notably, random sampling has also been accompanied with strong results in variety applications in the data-curation literature primarily due to unbiased sampling (Ayed Hayou, 2023b; Guo et al., 2022b).
21

---

Published as a conference paper at ICLR 2026
100% sample of C4 = 184B unique tokens 2.8 epochs
80% sample of E\t 1478 unique tokens E\r 3.5 epochs
60% sample of C4
1108 unique tokens =\r 4.7 epochs
40% sample of C4
73.68 unique tokens 7.1 epochs
20% sample of C4 36.88 unique tokens 14.2 epochs
sample
unique tokons
apachs
Figure We consider a setup where all of our models are trained on exactly 524B tokens, causing us to repeat the same examples for more epochs when we downsample. We borrow the format of this graphic from Muennighoff et al. (2023), who consider a similar setting.
D.2 DENSITY SAMPLING
See Section 3.2 for technical details about the DENSITY sampler. We use Sentence-T5-Bass (Ni al., 2021) as our embedding model for training samples, primarily due to contrastive training. giving confidence for computing distances amongst its 768-dim embeddings. We use the PStable hash (Datar et al., 2004) to hash the embeddings, along with [1, 000 x 20, 000) sketch matrix.
D.3 SEMDEDUP
The key idea is to perform (coverage maximizing) semantic deduplication inside clusters of the original dataset (?). We re-use the Sentence- Base embeddings data-points (Section D.2), and perform h-means clustering obtain 10, 000 clusters of the entire dataset.
D.4
SSL PROTOTYPES
They key idea is to remove prototypical points in a dataset (Sorscher et al., 2022). As a meaningful proxy, this method removes the points closest to cluster centroids of dataset. For brevity, we use the name "Prototypes" when reporting our results. We re-use the same embeddings and clustering for both SemDeDup and Prototypes.
D.5 PERPLEXITY FILTERING
A popular quality-filtering approach in the literature is to use the perplexity of proxy language models to filter data-points with a high-perplexity under that language model. While the literature historically used small language models for perplexity filtering (Wenzek et al., 2019; Muennighoff et al., 2023), recent work (Marion et al., 2023) suggests improved filtering performance when using LLMs for this task. To this end, we employ perplexity filtering with (S-(Small, Base, Large, XL, XXL) models; as well as intermediate checkpoints during the course of training T5-Large: (20k, 100k, 300k, 500k, 700k].
D.6 TEXT-QUALITY CLASSIFIER (Q- (Q-CLASSIFIER)
First proposed by GPT-3 for curating its pretraining dataset (Brown et al., 2020), and later used by various state-of-the-ar LLMs at their time (Chowdhery et al., 2023; Anil al., 2023; Gao et al., 2020; Du et al., 2022) another popular quality filtering approach is to train linear classifier for distinguishing web-scrape data vs. known reference high-quality data. Consistent with existing usage of this technique (Gao et al., 2020; Xie al., 2024: Brown et al., 2020; Chowdhery al., 2023; Du et al., 2022), we train a hashing-basee linear classifier with hash size of 262k trained to classify if a document is either from (negative) C4 or (positive) Wikipedia BookCorpus. We train this classifier
22

---

Published as a conference paper at ICLR 2026
for total of 218k steps (equivalent to 14 Trillion unigrams), and based recent evidence (Xie et al., 2024) we sample the documents with the highest score according to this classifier.
D.7 DATA SELECTION WITH IMPORTANCE RESAMPLING (DSIR)
Proposed by Xie et al. (2024), DSIR performs importance sampling using a bag-of-words estimator (we use unigram and bigram features) over some "high-quality" target data-source. This approach is, in spirit, quite similar the aforementioned texi-quality classification approach but performs distribution distributionmmatcinng in non-parametric way, and without the hassle of training classifier large piles of data. To consistent, we use Wikipedia + BookCorpus as the target source for DSIR as well. We re-use the official public implementation for DSIR
D.8 ASK-LLM SAMPLING
See Section 3.1 for technical details about the ASK-LLM sampler. Since ASK-LLM relies on the reasoning capabilities instruction-ture models, we use Flan-T5-(Small, Base, Large, XI. XXL) (Longpre et al., 2023a) and instruction tuned Gemma-7B (Team et al., 2024) models for obtaining the quality scores in ASK-LLM.
E EFFECTIVE MODEL SIZE
It is challenging to cisely summarize perfor- mance using a single mea- sure, primarily because our evaluation consists of 111 individual tasks, all of which respond at different rates to data and model op- timizations. To provide
Average Downstreom Performance
80
25
Taisk Set
80
BUSH
SQUAD
Reasoning (FLAN]
MMLU (FLAN)
QIA IFLAN
3g
holistic view of perfor-
mance, we a paramet- ric model, a.k.a scaling law (Hoffmann et al., 2022; Muennighoff et al., 2023), of the "model-size <- qual- ity" curve for the original T5 models (i.e., trained on the full C4 dataset) over var- ious downstream tasks.
TS-Small
TS Base
500M
Larearg
18
1.58
28
2.58
3.58
Parameters
Figure Empirical scaling laws for T5-models trained on the entire C4 dataset for various downstream tasks.
More specifically, we fit functions the following form, for each downstream task separately:
ModelSize A exp(B ExalPerformanco+C))
and use scip py. optimiiii ze. curve_ft fit to estimate the A, B,C parameters based on the evalu- ations of (Small, Base, Large, XL.. See Figure for a visual interpretation of the parametric models we fit.
Finally, given the performance of a model trained on downsampled data, the effective model size is defined the predicted model size by plugging in the observed downstream performance into the parametric scaling law estimated via Equation (1), taking a median over various downstream evaluation tasks listed in Figure 9.
23

---

Published as a conference paper at ICLR 2026
F DOWNSTREAM EVALUATION TASKS
F.1 PERPLEXITY
Defined as the exponentiated average negative log-likelihood an average sequence in the dataset; report the perplexity computed over the target tokens in T5's denoising objective (Raffel et al., 2020) over the default validation set provided by C4. Note that C4's validation set is a random sample of the dataset, it is prone be of much lower quality than curated sources, and hence, less reliable indicator of true model quality.
F.2 HQ PERPLEXITY
As our best effort to devise an inexpensive-to-coppii metric that is better aligned with model quality than perplexity on C4's validation set, inspired by the evaluation conducted in Tirumala al. (2023), we construct high-quality validation set from non web-scrape sources. We collate the validation sets from (1) English portion of wiki40b (Guo et al., 2020), (2) realnews and webtext subsets of C4, and (3) news commentary from the LMIB dataset (Chelba et al., 2013).
F.3 GLUE
A popular natural language understanding meta-benchmark comprising of eleven different tasks (Wang et al., 2018). Note that we report the average score for all individual tasks, after finetuning on the concatenation of all individual tasks' training sets, as is done in the original T5 implementation.
F.4 SUPERGLUE
A harder weeaa-benchmark (vs. GLUE) built further test the natural language understanding abilities of language models (Wang et al., 2019). Similar to GLUE, we report the average score of all tasks, and conduct fine-tuning all tasks' concatenated train-set.
F.5 CNN/DM
We use the CNN/DM dataset (Hermann et al., 2015) for testing our models' abstractive summarization abilities. Like the T5 original setting, we finetune on the train-set, and report the ROUGE-2 scores.
F.6
SQUAD
A popular dataset (Rajpurkar et al., 2016) used to evaluate question-ans answering capabilities of language models, we compare the finetuned performance of our models using exact-match the metric.
F.7 FLAN INSTRUCTION TUNING
popular application LLMs has been mnstruction-following and chatting capabilities. To test our model's 's quality on this front, we finetune our models on the FL.ANv2 dataset (Longpre et al., 2023a), and test the instruction-tume models' performance from four fronts:
5-shot MMLU (Hendrycks al., 2020): popular benchmark consiting of exam questions from 57 tasks.
3-shot Big Bench Hard (BBH) (Srivastava et al., 2022): a popular set of 23 hardest tasks from big bench.
Reasoning: macro-average 8-shot performance on GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDIV (Miao et al., 2021), and Strategy (Geva et al., 2021) benchmarks.
QA: macro-average 0-shot performance on UnifiedQA (Khashabi et al., 2020), BoolQ (Clark et al., 2019), Are-Easy and Are-Chale Are-Challenge (Clark et al., 2018) benchmarks.
Average: macro-average of all the four benchmarking suites listed above: MMLU, BBH, Reasoning, and Q/A.
24

---

Published as a conference paper at ICLR 2026
680
1600
0003 Quality Suand
Dit Qual-ty Stare
Dute Quulity Score LT1
Dute: Qualiyy Scaro
Detoo Qualty SCar
Figure 10: Score distribution of various data curation techniques. The plots for Flan-T5-* models are for ASK-LLM, whereas ones using T5- models are for perplexity filtering.
Please note that all of our reported numbers are based on single checkpoint evaluations, iee, we first select the hest checkpoint during FL.AN finetuning using the average performance on all tasks, and report the individual task performance on that checkpoint.
G ADDITIONAL RESULTS
G.1 (FIGURE 10) QUALITY-SCORE DISTRIBUTION FOR DIFFERENT SAMPLERS
For different data curation techniques listed in Section D, we examine the distribution of estimated data-quality scores normalized in way that higher represents better data quality.
For the DENSITY sampler, the plotted score is proportional to the likelihood of the example under the kernel density estimate.
For the Prototypes sampler, the plotted score represents the negated cosine similarity of data-point with its assigned cluster centroid.
For the SemDeDup sampler, the plotted score represents the negated maximum cosine similarity of datapoint all other datapoints in its respective cluster.
For the perplexity filtering sampler, the plotted represents the negated perplexity of a training sample.
For the ASK-LLM sampler, the plotted score represents the log probability of the token "yes" given the prompt in Figure 3.
G.2 (FIGURE 11) ANALYSIS FOR DIFFERENT SAMPLERS' AFFINITY TO DIFFERENT TOPICS
Since the different sampling strategies explored in this paper operate with different implicit biases, we try to understand if certain samplers exhibit more affinity to certain topics in the data compared to others. To visualize this phenomenon, we conduct the following procedure:
1. Load random sample of 500k datapoints, along with their respective data-quality scores.
2. Perform topic-modeling (via LDA) on the 500k datapoints with 9 topics.
Manually inspect the most common word associations in each of the 9 topics, and label a "high- level description" for each topic.
4. Assign each of the 500k datapoints to the LDA topic with the highest likelihood and analyze the differences between the distribution of scores within each topic and the global score distribution.
conducted a one-way ANOVA, one-vs-rest style, to determine whether the averages were statistically significant. Because N 500k, all effects were significant at the < 0.01 level. We measure the effect size using Cohen 's and report results Figure 11.
From the topic affinity analysis in Figure 11, we can observe a few interesting common trends:
The perplexity filters have relatively low variance in their scores, indicating much less biased sampling. This is expected, because perplexity filtering primarily biases toward "well-written text" which is relatively task/topic agnostic.
25

---

Published as a conference paper at ICLR 2026
1,00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
Entertainment
Business
6,22
0.26
2.0
0.3
0.0
0.24
0.30
2.22
6.25
019
0.37
0.00
5.23
-.29
Food
6.22
0.36
0.33
318
0.32
2.29
0.8
0.33
0.06
4.05
0.05
10.0 107 0..55 6,01 0M
10.55 -001
0.00 0.33
0.82
Features
Medical
6.07
0.38
3.83
0.21
0.29
0.01
0.20
-0.02
6.22
0.00 .0.2 -202 0.01
Politics & Relegion on
14.5
0.69
06.22
3.2
0.31
0.66
0.51
3.11
10.00
-0.9
10.08
-3,8
0.00
4.11
Products & Services
School & University
4.27
0.04
2.13
0.5
1608
9.3
0.28
0.07
0.10
9.30
0.06
0.08
0.08
0.77
9,07
0.01
1692
9,8
Tech
-0.30
2.02
0.03
451
3.54
6.32
0.30
6.13
013
0.1
0.00
0.0
DSIR
C-Casslfeer
Gemma 7B
Flan T5- 35mmmll)
Flan T5-8ase
Fan T5-XL
Flan T5-XXL
T5-Smal
T5-8ase
Large
T5-XL
T5-X00
T5-Large (20k)
TS-Large (100k)
TS. Large (300k)
TS-Large (500k)
T5 L--aare (rock)
Density
Prototypes
SemDeDup
Flan TS
Figure 11: Estimated topic affinity for quality filters (first 8), perplexity filters (next 10), and coverage- based samplers (last 3) over 500k randomly selected training samples.A higher score represents more affinity. All effects significant at the < 0.01 level.
The quality- samplers (LSK-LLM, DSIR, Q-Classifier) exhibit much stronger variance in their scores, with a common liking towards business, political, and religious content; and a common disliking towards tech, art, and entertainment content.
Consistent with the score correlations in Figure the prototypes and SupDeDup samplers exhibit inverse correlation with most other samplers when comparing topic affinity too.
Density sampling, as expected, exhibits no special affinity to any particular topic because it's objective is to only maximize coverage.
G.3 (FIGURES 12 TO 20) DATA-QUANTITY vs. MODEL- QUALITQU FOR DIFFERENT SAMPLERS
For different data curation techniques listed in Section D, investigate the tradeoff between the sampling rate and the respectively trained model's quality on various downstream evaluations listed in Section F. We plot our results in the following figures:
(Figure 12) T5-Small, coverage: Pre-training T5-Small on different amounts of data sampled by [Random sampling, DENSITY sampling, Self-supervised Prototypes sampling. SemDeDup).
(Figure 13) T5-Large, coverage: Pre-training La-Large on different amounts data sampled by [Random sampling, DENSITY sampling, Self-supervised Prototypes sampling, SemDeDup).
(Figure 14) T5-Small, ASK-LLM: Pre-training T5-Small on different amounts of data sampled by Ask-LLM using the (Flan-T5-Small Flan-T5-Base, Flan-T5-Largee Flan-T5-XL Flan-T5-XXL] scoring models.
(Figure 15) T5-Large, ASK-LLM Pre-training TS-Large on different amounts of data sampled by ASK-LLM using the TS-mal,,,,, Flan-T5-Base, -S-L-gee Flan-T5-XL, F5-n-T5-XXL) scoring models.
(Figure 16) T5-Small, Other quality-based Filters: Pre-training T5-Small on different amounts of data sampled by (Random sampling, DSIR, Q-Classifier, ASK-LLM (G.7B), ASK-LLM (XL))
scoring models.
(Figure 17) T5-Large, Other quality- based Filters: Pre-training T5-Large on different amounts of data sampled by (Random sampling, DSIR, Q-Classifier, ASK-LLM (G.7B), ASK-LLM (XL))
scoring models.
(Figure 18) T5-Small, Perplexity filtering: Pre-training T5-Small on different amounts of data sampled by Perplexity filtering using the (T5-Small, T5-Base, T5-Large, T5-XL, T5-XXL scoring models.
26

---

Published as a conference paper at ICLR 2026
(Figure 19) T5-Large, Perplexity filtering: Pre-training T5-Large different amounts of data sampled by Perplexity filtering using the (T5-Small, T5-Base, T5-Large, T5-XL, T5-XXL} scoring models.
(Figure 20) T5-Large, Perplexity filtering: Pre-training T5-Large on different amounts of data sampled by Perplexity filtering using the (20k, 100k, 300k, 500k, 700k) intermediate checkpoints of TarLa as data quality scoring models.
Effective Model Size
Perplesity Reaconing mini (Post-FLAN) Aug. (PoIt-FLAM) FLAM]
(Post-FLAM0
30.0
T5-Sral
28.00
25.32
MMU (Post-LAAM) (LANT
00:0
3.75
Supee-GLUIE
SQUAD
CNN-DM
1830
75.75
37.50
72.30
150
164
Dataset Size (Billion Tokers))
Datasel Size (Billion Tokens)
Dataset Size (Billion Tokems)
Full data
Random
Density
Prototypes
SemDeDup
Figure 12: Tradeoff between data quantity and model quality while pre-training T5-Small. Each point in this plot comes from the converged pre-training run over sampled dataset. See Section for description about the metrics used in this plot.
27

---

Published as a conference paper at ICLR 2026
Effective Model Stzo
Reasoning (POSt- Ava. (FozD.FLAN)
5
DBI (Post-FLAMM) 10
253M
203M
433M
MMILI LAN)
O1A (Post-FLAN)
13
40
3
18
430
Perplesiiyy (HQ)
Super-LLE
15
370
$9.15
$7.6)
87.03
CHI-OM -44
26.85
GUUE
86.43
26.00
26.55
07.75
85.23
115
28
184
150
(Billion
(Billion
Full
Random
Density
Prototypes
SemDeDup
Figure 13: Tradeoff between data quantity and model quality while pre-training T5-Large. Each point in this plot comes from the converged pre-training run over a sampled dataset. See Section for description about the metrics used in this plot.
Effective Model Size
Reassning (Post-FLAM) Avg. Aug. (Post.FLAN)
(PoSt-FUANG
660
20.23
25.43
8
Perplexity (Post-
3
34:00
14.00
34.00
2.13
79.01
18.00
16.70
79.23
CNN-DM
18.40
10
150
40
315
190
115
153
164
Dataset Size (Billion Tokers))
Dataset Size (Billion Tokens)
Dataset Size (Billion Tokens)
Full data
LLM (Small)
AsK-LLM (Large)
Ask-LLM (XXL)
Random
Ask-LLM (Base)
Ask-LLM (XL)
Figure 14: Tradeoff between data quantity and model quality while pre-training T5-Small. Each point in this plot comes from the converged pre-training run over sampled dataset. See Section for description about the metrics used in this plot.
28

---

Published as a conference paper at ICLR 2026
FLAN) Effective Model Size
1.28
Avg. (PozD-LAAM)
DBI (Post-- LAN) 22
1:18
19
8 18
Reasoning (POST-
QMA (Post-FLAN)
8
78
72
420
Perpleaity (HQ)
CHI-DX Supar-GLLE
$3.40
43.40
00.00
3.20
23.00
68.50
SQUAD
20.00
69.20
26.70
07:60
26.90
87.30
42
123
150
184
130
(Billion
(Billion
data
(Smail)
(Large)
(XXL)
Random
ALM (Base)
ALM
Figure 15: Tradeoff between data quantity and model quality while pre-training T5-Large. Each point in this plot comes from the converged pre-training run over a sampled dataset. See Section for description about the metrics used in this plot.
Effective Model Size
0011 (Post- FLANI
20.00
24
62M
13
36M
Reaconing (Post-FLAN) Aug.
28.40
25.00
24:00
13.00
34:00
20.39
MMU
25.00
24.73
7.30
Perplesity (HQ)
6:90
6:00
6.73
8.50
18.70
$0.40
CNN-DM
40.00
70.00
74:00
38.0
1840
29.20
10
150
115
150
115
153
16
Dataset Size (Billion Tokers))
Datasel Size (Billion Tokens)
Dataset Size (Billion Tokens)
Full data
Random
Ask-LLM OXL)
Ask-LLM (G. 78)
Q-Classifler
DSIR
Figure 16: Tradeoff between data quantity and model quality while pre-training T5-Small. Each point in this plot comes from the converged pre-training run over sampled dataset. See Section for description about the metrics used in this plot.
29

---

Published as a conference paper at ICLR 2026
Effactivn Modell Stee
DD1 (Poxt-FLAM 88
1.20
8
2330
Reazonina (Pozt-FLAN)
O/A (Phstt-FLAN)
460
8.40
Perplexity (HQ)
Suppr-GLLE
04.00
14.90
Parplesity
63.00
03.00
3.0
32.00
43.40
3.6
GLUE
60.40
26.90
26.00
20.50
87.28
60
115
153
166
42
135
150
164
115
166
144
(Billion
(Billion
Full data
Random
Ask-LLM (XL)
ASK-LLM (G.78)
Q-Classifier
DSIR
Figure 17: Tradeoff between data quantity and model quality while pre-training Larger Each point in this plot comes from the converged pre- training run over a sampled dataset. See Section for description about the metrics used in this plot.
Effective Model Size
66M
Reasoning (POS-FLANN) Aug. (POTT-PLAM)
28.22
(Post-FLAN) FLANI
27.01
26.0
133mral
27.01
5TM 54N 53N
27.00
25.82
27.00
36.23
MHLU (PodsFLLAM)
26
23.00
QUA
Perplexity (HQ)
8.13
Supar-GLUE
7.83
1.00
79.23
18:00
76.4
CHI-DM
18.56
78.40
73.40
10
11A
150
184
40
115
150
115
153
16
Dataset Size (Billion Tokers))
Dataset Size (Billion Tokens)
Dataset Size (Billion Tokens)
data Ra Random
Perplexity (Small)
Perplexity (Large)
Perplexity (XXL)
Perplexity (Base)
Perplexity (XU)
Figure 18: Tradeoff between data quantity and model quality while pre-training T5-Small. Each point in this plot comes from the converged pre-training run over sampled dataset. See Section for description about the metrics used in this plot.
30

---

Published as a conference paper at ICLR 2026
Efrective Model Size
08H (Post-FLAM)
134agg
Recsoning (Post-FLAN) Aug. (Post-FLAN)
22
24
20
QVA (Post-FLAM)
24
4.00
Perplcity
1.05
Perplecity
Super-
2.00
$1.00
CWN-DM
20.0
20.42
20.2
$5.05
46
115
150
164
40
75
115
160
184
19.75
45
115
155
164
(Billion
Perplexity (Small)
Perplexity (Large) Perplexity (XL)
Perplexity
Ra Random
Perplexity (Base)
Figure 19: Tradeoff between data quantity and model quality while pre-training LarLar Each point in this plot comes from the converged pre- training run over a sampled dataset. See Section for description about the metrics used in this plot.
Effective Medel Size
120
80H (Post-FLANN)
1348gg
603M
4038
Roaconing (POSt:FLAN) Aurg
2034
(Post-LANN
OIA (Poxt-FLAM)
24
34
4.50
6.1
Super-GULE
Perplesitty
3.70
2.50
19:55
CHN-DM
20:0
$7.20
20:5
$5.50
115
Fio
184
40
115
190
144
115
153
164
Dataset Size (Billion Tokers)
Dataset Size (Billion Tokens)
Dataset Size (Billion Tokems)
Full data Random
Perplexity (20k)
Perplexity (300k)
Perplexity (700k)
Perplexity (100k)
Perplexity (500k)
Figure 20: Tradeoff between data quantity and model quality while pre-training T5-Large. Each point in this plot comes from the converged pre- training run over sampled dataset. See Section for description about the metrics used in this plot.
31

---

Published as a conference paper at ICLR 2026
G.4 (FIGURES 21 TO 29) QUALITY OF FRESH REPEATED TOKENS FOR DIFFERENT SAMPLERS
We investigate the data-efficiency for different data curation techniques listed in Section D over various downstream evaluations listed Section F, when stratifying by the maximum number of repetitions allowed over the sampled dataset. We plot our results in the following figures:
(Figure 21) T5-Small, coverage: Average data-efficiency of pre-training T5-Small on data sampled by (Random sampling, DENSITY sampling, Self-supervised Prototypes sampling. SemDeDup), stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 22) T5-Large, coverage: Average data-efficiency of pre-training T5-Large on data sampled by (Random sampling. DENSITY sampling, Self-supervised Prototypes sampling. SemD:Dup). stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 23) T5-Small, ASK-LLM: Average data-efficiency of pre-training T5-Small on data sampled by ASK-LLM using the (Flan-T5-Small Flan-T5-Base, Flan-TSi T5-Large, Flan-TS-XL, Flan-T5-XXL) scoring models, stratified by the maxmimum number of allowed repetitions the sampled dataset.
(Figure 24) T5-Large, ASK-LLM: Average data-efficiency of pre-training T5-Large on data sampled by ASK-LLM using the (Flan-TS-Smalll Flan-T5-Base, Flan-T5-Large, Flan-T5-XL, TX-X-L)-XXL)) scoring models, stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 25) T5-Small, Other quality-based Filters: Pre-training T5-Small different amounts data sampled by (Random sampling, DSIR, Q-Classifier, ASK-LLM (G.7B), AsK-LLM (XI)) scoring models, stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 26) T5-Large, Other quality- -basd Filters: Pre-training T5-Large on different amounts of data sampled by (Random sampling, DSIR, Q-Classifier, ASK-LLM (G.7B), AsK-LLM (XL)) scoring models, stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 27) T5-Small, Perplexity filtering: Average data-efficiency of pre-training T5-Small on data sampled by Perplexity filtering using the (T5-Small,, T5-Base, T5-Large, T5-XI., T5-XXI.) scoring models, stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 28) TS-Large Perplexity filtering: Average data-efficiency of pre-training T5-Large on data sampled by Perplexity filtering using the (T5-Small, T5-Base, T5-Large, T5-XL, T5-XXL) scoring models, stratified by the maxmimum number of allowed repetitions over the sampled dataset.
(Figure 29) T5-Large, Perplexity filtering: Average data-efficiency of pre-training T5-Large data sampled by Perplexity filtering using the 120k, 100k, 300k, 500k, 700k) intermediate checkpoints of La-Large as data quality scoring models, stratified by the maxmimum number of allowed repetitions over the sampled dataset.
32

---

Published as a conference paper at ICLR 2026
Max epochs
Max epocte
Nax 5 epachs
Nax 10 epochs
Max epochs
Max 30 eseche
13
Tul Data
Random
Denudly
Propotypes
Sem
Figure 21: Average data-efficiency of pre- tr-iiriinnn T5-Small on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point in this plot represents the perfor- mance of an intermediate checkpoint averaged over sampling ratios, as long as the maximum allowed repetitions have not been reached. See Section F for description about the metrics used in this plot.
Takee Gen
Figure 22: Average data-efficiency of pre- pre-training T5-Large on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point this plot represents the perfor- mance of an intermediate checkpoint averaged over all sampling ratios, long as the maximum allowed repetitions have been reached. See Section F for description about the metrics used in this plot.
33

---

Published as a conference paper at ICLR 2026
Max epochs
Max epochs
Nax 5 epects
Max 10 epochs
Max 20 epocts
Max 30 epochs
10
Tall Data
Rendom
Auc-LIM [5mall
ASC-LEI [Basel
LUN (aarge)
AUE-LIN OLL
ASKLIM COOL
Figure 23: Average data-efficiency of pre- training T5-Small on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point in this plot represents the perfor- mance of an intermediate checkpoint averaged over sampling ratios, as long as the maximum allowed repetitions have not been reached. See Section F for description about the metrics used in this plot.
spochs
Takees Gen
Figure 24: Average data- data-efficiency of pre-training T5-Large on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point this plot represents the perfor- mance of an intermediate checkpoint averaged over all sampling ratios, long as the maximum allowed repetitions have been reached. See Section F for description about the metrics used in this plot.
34

---

Published as a conference paper at ICLR 2026
Max epochs
Max epochs
Nax 5 epects
Max 10 epochs
Max 20 epocns
Max 30 epochs
Full Dats
Random
Asd LIM (XL)
ANCALI (G.78)
D-Casuller
DSR
Figure 25: Average data-efficeency of pre- mre-training T5-Small on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point in this plot represents the perfor- mance of an intermediate checkpoint averaged over sampling ratios, as long as the maximum allowed repetitions have not been reached. See Section F for description about the metrics used in this plot.
spochs
Takees Gen
Figure 26: Average data-efficiency of pre- pre-training T5-Large on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point this plot represents the perfor- mance of an intermediate checkpoint averaged over all sampling ratios, long as the maximum allowed repetitions have been reached. See Section F for description about the metrics used in this plot.
35

---

Published as a conference paper at ICLR 2026
Max epochs
Max epochs
Max 5 spects
Max 10 epochs
Max 20 epocts
Max 30 epochs
10
26
Dul
Random
Perplesly 15mm
Perplecty (Bass)
Peeplesity (ange)
Perplestyy OL)
Perpleatly (XLL)
Figure 27: Average data-efficiency of pre- training T5-Small on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point in this plot represents the perfor- mance of an intermediate checkpoint averaged over sampling ratios, as long as the maximum allowed repetitions have not been reached. See Section F for description about the metrics used in this plot.
00
Can
Takees Gen
Perpienky
Peeplexity
Perplesity
Figure 28: Average data data-efficiency of pre-training T5-Large on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point this plot represents the perfor- mance of an intermediate checkpoint averaged over all sampling ratios, long as the maximum allowed repetitions have been reached. See Section F for description about the metrics used in this plot.
36

---

Published as a conference paper at ICLR 2026
Max epochs
Max epochs
Nax 5 spects
Max 10 epochs
Max 20 epocns
Max 30 eprchs
13
Pull Dala
Random
Pepplesity (294)
Perplecty (100c)
prppeccly (300k)
Perpreckyy (500k)
Per perpiccyy (800c)
Figure 29: Average data-efficiency of pre- me--trining T5-Large on sampled data, stratified by maximum number of allowed repetitions on the sampled dataset. Each point in this plot represents the perfor- mance of an intermediate checkpoint averaged over sampling ratios, as long as the maximum allowed repetitions have not been reached. See Section F for description about the metrics used in this plot.
G.S (FIGURES 30 TO 36) DATA-EFHICIECCY OF DIFFERENT SAMPLERS
We investigate the data-efficiency for different data curation techniques listed in Section D over various downstream evaluations listed Section F, when stratifying by the sampling ratio or the size of the sampled dataset. We plot our results in the following figures:
(Figure 30) T5-Small, ASK-LLM: Data-efficiency of pre-training T5-Small on data sampled by AsK-LLM using the (Flan-T5- Tm-lll,,,, Flan T5 75-Base, -a-n-T--aare. Flan -aan-T--XL, Man-T5-XXLL scoring models, stratified by the sampling ratio.
(Figure 31) T5-Large, ASK-LLM: Data-efficiency of pre-training T5-Large on data sampled by AsK-LLM using the Flan-T55-Small scoring models, stratified by the sampling ratio.
15-Base, Flan-T5 Ta-Lerge, Flan-T5-XX Flan-T5-XXL.
(Figure 32) T5-Small, Other quality-based Filters: Data-efficiency of pre-training T5-Small on data sampled by (Random sampling, DSIR, Q-Classifier, AsK-LLM (G.7B), ASK-LLM (XL)) scoring models, stratified by the sampling ratio.
(Figure 33) T5-Large, Other quality-based Filters: Data-efficiency of pre-training T5-Large on data sampled by (Random sampling, DSIR, Q-Classifier, ASK-LLM (G.7B), AsK-LLM (XL)) scoring models, stratified by the sampling ratio.
(Figure 34) T5-Small, Perplexity filtering: Data-efficiency of pre-training T5-Small on data sampled by Perplexity filtering using the (T5-Small, T5-Base, T5-Large, T5-XL, T5-XXL scoring models, stratified by the sampling ratio.
(Figure 35) T5-Large, Perplexity filtering: Data-efficiency of pre-training T5-Large on data sampled by Perplexity filtering using the (T5-Small, T5-Base, T5-Large. T5-XL, TS-XXL| scoring models, stratified by the sampling ratio.
(Figure 36) T5-Large, Perplexity filtering: Data-efficiency of pre-training T5-Large data sampled by Perplexity filtering using the (20k, 100k, 300k, 500k, 700k) intermediate checkpoints of Large as data quality scoring models, stratified by the sampling ratio.
37

---

Published as a conference paper at ICLR 2026
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
Perploity (04)
203
530
282
336
254
460
244
Parpteatty (ni)
163
1.40
1.34
80.00
60.13
12.00
18.00
7650
73.59
Super GAUIE
303
100
Traising Bikens (Billion)
Traniing Toxonn (Billion)
Taining Tokens (Bilioo))
Traning Tekena (Billon))
data
Ask-LIM (Small)
Ask-LLM
ASR-LLM
Figure 30: Data efficiency comparison of different samplers while training T5-Small for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
3.00
3.00
243
4.3
4.30
4.20
Perpaary
1.10
3.0
2.00
230
104
X5
130
253
11.00
10.00
100
200
303
180
200
333
500
200
Thiiiing Theens [Billion]
Triiting Tolern (tillinn
Twining Tikeets (oi))
Thiining Tokers (billlen)
data
Random
Ask-LIM (Small)
Adk (Base)
Ask-LLM
ALk IKL)
AsR-LLM (XXL)
Figure 31: Data efficiency comparison of different samplers while training T5-Large for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
38

---

Published as a conference paper at ICLR 2026
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
222
1.4
7.00
630
1.30
200
44
300
544
$3.00
1050
304 200
400 Paining Toklns (Billon)
Training Toiens :Blliont
Training Tokken (Sillin)
Traniigg Takons (Biliion)
DSIR
Classiifeer
(G.78)
OXL)
Figure 32: Data efficiency comparison of different samplers while training T5-Small for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
(C41
3.50
3.70
4.30
4.2
433
290
100
104
130
253
500d3 GLU
303
304
200
333
480
500
200
Thining Theens [Billion)
Triiting Tolern (tillond
Taining Tikeets (Bll)o))
Thiining Tibens (billlen)
Random
DSIR
G-Classilee
Ast-LLM (G.78)
AdkLLM OLL)
Figure 33: Data efficiency comparison of different samplers while training T5-Large for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
39

---

Published as a conference paper at ICLR 2026
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
Perpbecky (CA)
600
6.0
283
628
Parplecty (HQ))
650
7.50
7.40
1.0
120
20
304
X1
19.00
0930
Supar- QUUEE
100 238
309
200
492
150
200
500
205
Traiiing Biken (Billion)
Traniing Toxons (Bilion)
Taining Tokens (Billoo))
Trining Takena (Billon)
Randam
Perplexity
Parplixity
Perplexity
Perplexity
Perplexity (OLL)
Figure 34: Data efficiency comparison of different samplers while training T5-Small for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
3.00
272
222
4.00
3.00
300
100
GLLE
11:00
00.1
Super
100
303
24
XX
383
203
480
300
Twiining Thoess [Billio]
Trining Tolers (lillon)
Taining Thleens (Bll)
Thiining Tibens (billen)
Random
(Small)
(Bare)
Perplexity (Large)
Perplexity PXL
Perplexity OOLL
Figure 35: Data efficiency comparison of different samplers while training T5-Large for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
40

---

Published as a conference paper at ICLR 2026
20% Sampling Ratio
40% Sampling Ratio
60% Sampling Ratio
80% Sampling Ratio
Pepplealt (H10)
4.50
432
0.23
4.2
139
225
GLIVE
SuperGurE
Wairing Iakere
Training
18llio))
Traning (billon)
Training Tocans (Bilion)
--- Fullidata
Bardom
Per wwrplclyy (204)
Perpissity (100k)
Perpleslly (300k)
Perploxity (500k)
Perplecity (700k)
Figure 36: Data efficiency comparison of different samplers while training T5-Large for various sampling ratios. Each point in this plot is the performance an intermediate checkpoint during the course of training sampled data.
H QUALITATIVE RESULTS
In this section we look at some qualitative training samples, sorted according to various criteria of data-quality scores. Along with the textual content of each training sample, we also list estimated data-quality percentile for AsK-LLM and perplexity filtering samplers, i.e., the percentile of the given data- lata-point's quality score amongst the entire training set. A high percentile represents that the sampler estimates this training sample to have higher quality compared to other training samples in the dataset. We manually don't include any NSFW examples to the best of our knowledge.
H.1 HIGH-QUALIT SAMPLES IDENTIFIED BY ASK-LLM
We look at the training samples that all ASK-LLM scoring models, average, think are good (i.e., have a high percentile). To the best of our understanding, the overarching conclusions we make by observing these qualitative samples are:
AsK-LLM doesn't seem to have any length bias for good examples.
AsK-LLM can accurately tag high-tuality training samples that contain a lot of proper nouns and named entities. Perplexity filtering gets these kind of samples wrong.
Even looking at this slice of only the highest-quality data tagged by AsK-LLM, perplexity filtering scores don't seem to correlate well with ASK-LLM scores as suggested by Figure 7.
Example I: Estimated Data-Quality (Percentile - Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
XL
XXL
Small
Large
XL
XXL
93.33%
88.21%
88.11%
100.0%
99.99%
50.29%
30.34%
32.56%
31.61%
25.62%
What constitutes overtime for part-time employee? Question: What is overtime for a part-time employee? Overtime for part-time employee is time that is beyond the part-time employee's 's ordinary hours of work or outside the agreed number of hours of work, as specified in their employment contract.
41

---

Published as a conference paper at ICLR 2026
Example 2: Estimated Data Quality (Percentile Higher better)
Ask-LIM
Perplexity Filtering
Small
Base
T Large
X1.
XXL.
Small
Base
Large
XT.
XXI.
98.54%
96.4%
96.3%
46.2%
54.65%
46.2%
49.85%
20.33%
Viva La Vegan! Can Vegan Lifestyle Help to Get Rid of Ocean Dead Zones? Can a Vegan Lifestyle Hclp to Get Rid of Ocean Dead Zones? A dead zone is an area at the bottom of the ocean that oxygen depleted and cannot maintain any marine life. The biggest cause of these dead zones is an overflow of fertilizers, sewage and industrial pollutants being pumped into rivers all over the world. Thankfully dead zones can be reversed and living vegan lifestyle can help enormously and I'll show you how. What are Ocean Dead Zones?
Vegans don't 't want to harm the planet. On the contrary they want save it and what better way than living with nature instead of against it and helping the planet in ways we probably never even realised, like helping reverse oceans dead zones. Next time you think about buying something you don't 't need, or eating food that is highly processed or non-organic, spare thought for the largely unknown dead zones and how overconsumption and an unnatural lifestyle is slowly killing both you and them.
Example 3: Estimated Data- Qualt-y (Percentile Higher better)
ASK-LLM
Perplesity Filtering
Small
Base
Large
XXL
Small
Base
Large
XXL
98.81%
98.96%
95.42%
99.53%
99.56%
88.1%
80.99%
77.13
65.89%
73.79%
Question: necessary to dredge ponds and lakes in the upper coastal region of South Carolina? Answer: It is necessary to dredge ponds and lakes in South Carolina, in the upper coastal region South Carolina. Each lake and each pond a different environment and as years pass, these environments accumulate lot of sediment. They tend fill in with storm water runoff, they tend from natural leafy materials, storm water fun off, sand, silt, sediment, muck, mire. All of these produce in the bottoms of pond beds and lake beds. So it is absolutely necessary to do an evaluation every so many years to determine whether not you need remove the sediment that's accumulated.
be grass clippings, leafy
42

---

Published as a conference paper at ICLR 2026
Example 4: Estimated Data- Quality (Percentile Higher better)
Perplexity Filtering
Small
Base
Large
XL.
XXL.
Small
Base
Large
XT.
XXT.
88.93%
95.14%
93.44%
26.83%
34.32%
32.98%
31.1%
28.35%
S
However, it's a long and challenging way mass production. New Tesla Model 3 an electric game-changer worth $35,000 and comes in classic black color. A single masterpiece in black now belongs Tesla's CEO and co-founder Elon Musk. Why not mass market yet? Company has quite complicated reason. Tesla needs to make sure that it can build, deliver and service enormous numbers of these awesome electric cars without sacrificing quality. Tesla will present 30 first cars at launch celebration dated on July 28. 100 cars with production speed cars per day dated for August. 1,500 cars will be ready September.
Owners new Teslas will also enjoy exquisite acrodynamic wheel face. An itemized list of the Tesla Model 3's features, specs, and pricing expected to be revealed July 28, the car's launch party. 5.6 seconds is what it gets the Model to go from zero to 60 miles per hour, as May news says. Hot, right? accelerates even faster than the base model BMW 3 Series or the famous Meereeee--ennt C Class, which are leaders in the compact luxury space. single charge will allow minimum 215 miles of single drive. The roof in Model 3 made almost entirely of glass, providing an incredible sense of space and infinity. Moreover, it blocks UV rays and manages the level of heat.
Example 5: Estimated Data-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
XL
XXL
Small
Base
Largo
XXL
89.28%
98.111%
98.93%
98.7%
96.32%
26.24%
19.14%
26.25%
26.05%
24.29%
Landmines. Every month, 1200 people are maimed, and further 800 killed throughout the world due landmines. Landmine removal efforts are clearing about 100,000 mines a year, but at rate it will still be over 1000 years to get them all. The cost of clearing them is huge, with estimates excess of $50 billion. Worse still, for every 5000 mines cleared, one person will die in the process.
Hopefully the work that people like Vandiver and Tan can be built upon and further progress can be made in the fight to clear the world of landmines. The video below shows a group of minesweepers working with the kits- and it is clear understanding as to how the mine operates is already improving- giving them the knowledge they need to safely diffuse any mines they encounter.
watching them that the level of
43

---

Published as a conference paper at ICLR 2026
Example 6: Estimated Data Quality (Percentile Higher better)
Perplexity Filtering
Small
Base
Large
XI.
XXI.
Small
Base
Large
XT.
XXL.
87.79%
98.52%
88.09%
19.72%
17.88%
16.95%
By all measures successful chemical engineering undergraduate Oregon Agricultural College, and wanting very much to continue his education and carn his PhD in chemistry, Linus Pauling wrote to several graduate programs across the country, inquiring in particular about fellowships. Though he had proven himself to be prodigious talent student and, already, teacher, Pauling's s location in Corvallis didn't 't carry great deal cache with the country's elite institutions. And given his family's 's shaky financial health, some measure institutional funding was going be required if were to advance the academy.
During his sparse free time, Pauling wrote letter after letter to his girlfriend. Ava Helen Miller, who remained in Corvallis continue work her Home Economics degree OAC. Having expressed a desire to marry at least twice before Linus left for California, only to be rebuffed by their families, the two decided in their letters that they would absolutely be wed once Pauling had finished his first year of classes and just prior to his resumption of more construction work during the summer. Their plan came to fruition Salem, Oregon on June 17, 1923, and Ava Helen moved to Pasadena that fall to accompany her new husband during his second year graduate student.
Example 7: Estimated Data- Quality (Percentile Higher better)
Perplexity Filtering
Small
Base
Large
XI.
XXL.
Small
Base
Large
XL.
XXI.
87.08%
89.33%
95.26%
99.94%
98.09%
97.52%
98.83%
97.39%
Bonelli, N.; Giordano, S.; Procissi, G. Enif-Lang: Specialized Language for Programming
Network Functions on Commodity Hardware. J. Sens. Actuator Netw. 2018, 7, 34. Bonelli Giordano S, Procissi G. Enif-Lang: Specialized Language Programming Network Functions on Commodity Hardware. Journal of Sensor and Actuator Networks. 2018; 7(3):34. Bonelli, Nicola; Giordano, Stefano: Procissi. Gregorio. 2018. "Enif-Lang Specialized Language for Programming Network Functions on Commodity Hardware." J.
Sens. Actuator Netw. 7, no. 34.
44

---

Published as a conference paper at ICLR 2026
Example 8: Estimated Data Quality (Percentile Higher better)
Ask-I.MM
Perplexity Filtering
Small
Base
T Targe
XT.
XXT.
Small
Rase
Large
XI.
XXT.
96.41%
97.38%
34.7%
44.8%
56.87%
77.25%
S
"What your number one secret productivity?" In recording their responses, Kruse came across some fascinating suggestions. What follows are some of my favorites. They focus on minutes, not hours. Most people default to hour and half-hour blocks on their calendar; highly successful people know that there are 1,440 minutes in every day and that there is nothing more valuable than time. Money can be lost and made again, but time spent can never be reclaimed. As legendary Olympic gymnast Shannon Miller told Kevin, "To this day, keep a schedule that is almost minute by minute." You must master your minutes to master your life.
Energy is everything. You can't make more minutes in the day, but you increase your energy to increase your attention, focus, and productivity. Highly successful people don't skip meals, sleep, breaks in the pursuit of more, more, more. Instead, they view food as fuel, sleep as recovery, and breaks as opportunities to recharge in order to get even more done. Author #1 bestselling book, Emotional Intelligence 2.0, and president of TalentSmart, world's leading provider of emotional intelligence.
H.2 LOW-QUALITY SAMPLES IDENTIFIED BY ASK-LLM
We look the training samples that all AsK-LLM scoring models, on average, think bad (i.e., have a low percentile). To the best of our understanding, the overarching conclusions we make by observing these qualitative samples are:
ASK-LLM doesn't seem have any length bias for bad examples.
ASK-LLM filters hateful or toxic examples that might hurt LLM training.
Ask-LLM rejects non-contextual samples, e.g., having only questions with no answers, repeated non-sensical content, etc. Notably, perplexity filtering performs bad in these cases, as these low quality examples tend to have a low perplexity score.
Example 9: Estimated Data hata-Quality (Percentile Higher better)
AsK-LLM
Perplexity Filtering
Small
Base
Large
XL
XXL
Small
Base
Large
XL.
XXL
0.01%
0.01%
0.01%
0.0%
0.0%
40.46%
25.66%
27.42%
25.6%
28%
Format iso Juiced 2: HIN evolves the current street racing scene, letting players experience PC Repack DiRT Rally v1.17 Black Box Bears Cant Drift PC torrent uploaded. Juiced ?2 ???? ????
Release name Juiced2. Horeed..Hot..mmggg Import. Miipoo..iggtt-MMuiii. Nii.ss--MlliS-RRLLAAAAAA
? ??? ? 2?
????!
HIN evolves the current street racing scene, letting players experience the culture of the real-life HIN tour, the nation?s largest lifestyle custom. Juiced Hot Import Nights Torrent. Bittorrent 729. 64 MB. Juiced 2 Hot Import Nights Download free torrent Largest Bittorrent Source with Several Listed Files. Now can upload screenshots or other images (cover scans, disc scans....
45

---

Published as a conference paper at ICLR 2026
Example 10: Estimated Data- Quality (Percentile Higher is better)
AsK-LLM
Perplexity Filtering
Small
Base
Large
XT.
XXT.
Small
Base
Large
XT.
XXT.
5.41%
3.86%
6.24%
62.97%
75.91%
86.3%
85.26/
You were a good daughter the first day two. Now, you are only showing the worst sides of yourself. can only be sad and disappointed in you.
Example 11: Estimated Data- Quality (Percentile Higher is better)
Perplexity Filtering
Small
Rase
Large
XI.
XXI.
Small
Base
Large
XL
XXL.
0.41%
1.44%
35.97%
31.46%
$1.15%
38.19%
$
Kids can help you enrich your life? Be a better person? Learn think about someone else? Apparently whoever said these things has never had children because from everything we have seen and experienced, kids are flat out horrible. College can't come fast enough.
Example 12: Estimated Data- Qual-Quality (Percentile Higher is better)
AsK-LLM
Perplexity Filtering
Large
Large
1.89%
3.58%
3.11%
6.02%
0.09%
18.09%
22.8%
25.61%
19.14%
47.01%
Events is how can go ice skating with real penguinsGrab your tickets before they sell out! Can you spot anyone you know these fun pics? EventsHow do get tickets for Wimbledon 2018?
Example 13: Estimated Data- -uulaQyuattyyyy (Percentile
Higher is better)
Ask-LLM
Perplexity Filtering
Large
Large
2.17%
1.11%
3.75%
2.0%
5,31%
92.49%
89.88%
86.79%
97.04%
96.78%
That don't make you happy? We can start all over some day? Somewhere, are you dreaming of me? Won't you come back home me?
46

---

Published as a conference paper at ICLR 2026
Example 14: Estimated Data- Quality (Percentile Higher is better)
Perplexity Filtering
Small
Base
Large
XI.
XXT.
Small
Base
Large
XL.
XXL.
0.04%
0.07%
68.86%
51.15%
44.08%
35.81%
19.28%
2 ?
? (1395).
???? 26(2), 145-159.
? 1
?
???????? 26, 2, 1395, 145-159. 2...
?
(1395).
? ?????
26(2). pp. 145-159. ?
? ? ? ??????
???? 1395; 26(2): 145-159. 211159999999 ?
BHT 2
DPPH ?
(HPMC)
2
Effect of the plasticizer permeability, mechanical resistance and thermal behaviour of composite coating films. Powder Technology 238:14-18 Martos MV, Mohamady MA, Femm'ndez?l.pee.. J, Abd ElRazik KA,, Omer FA,, P?rez?Alvarez JA and Sendra E, 2011. In vitro antioxidant and antibacterial activities of essentials oils obtained from Egyptian aromatic plants. Food Control 22: 1715?1722. Phoopuritham P. Thongngam M, Yoksan and Suppakul P. 2011. Antioxidant Properties of Selected Plant Extracts and Application in Packaging as Antioxidant Baled Films for Vegetable Oil. Packaging Technology and Science 25: 125?136. Rojas?Gra? MA, Avena Bustillos RJ, Olsen C, Friedman M, Henika PR, Martin Belloso O, Pan Zh and McHughTH, 2007. Effects...
Example 15: Estimated Data- Auta-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
XXL.
Small
Base
Large
XL.
XXI.
0.01%
0.02%
0.02%
0.01%
0.0%
59.41%
36.81%
23.01%
12.95%
17.24%
Showing results for tags AA3rmma start'. have Error mesage "Addon 'A3_ _3__epoch____ server' requires addon 'A3 @A3_epoch____ "conhig"""""" why is that and how fix this? When i click Ok get this My Start.cmd losk like this: arma3server.exe (email protected]; @ EpochHive; -config=C: arma 3 ? SC config.cfg -i2= 68.77.2.4. -pott=2301 -profile=SCC -efg=C: arma SC ? basic.cfg name=SC This is my RPT file:
C: arma 3 arma3server.exe == arma3server.exe [email protected]; @EpochHive; -config=C: arma 3 SC
2:05:23 Updating base class Rsstooss,oxx by ui ui_f? config.bin/Rse/GJJI....
2:05:23 Updating base class ->RscListNBox. by a3 ? ui f? Banfg..binRss/GGUJLssNNNBBo
2:05:23 Updating base class Text, by a3 ui f? config, BiiiggiiiinscBBerruunddll 2:05:23 Updating base class Text,,,tt, by a3 ui f config. www.///ceBBccggruunuUUUU// 2.00://ummmm.222..2.2202323 2:05:23 Updating base class ->scPicture, by a3 ? ui ff wwa.gg.iinRssBBaccoonnn........
2:05:23 Updating base class >>RscPicture, by a3 ? ui f?
config. heiigg/iiiiisBeeeggnnnnnGGIR 2:05:23 Updating base class Piccture by a3 wi
? ConfgghhnRKsshaegggoonnCCCTTooOo 2:05:23 Updating base class Recttext, by a3...
47

---

Published as a conference paper at ICLR 2026
Example 16: Estimated Data- Quality (Percentile Higher is better)
Perplexity Filtering
Small
Base
Targe
XI.
XXT.
Small
Base
Large
XT.
XXI.
0.47%
3.79%
193%
L.8%
10.22%
51.15%
46.92%
63.04%
44.77%
41.35%
my wife) in my name and land (plots). Recently sold one of flat (100% cheque payment). Can I reinvest the Capital gains arriving out of sale purchasing flat? Note: had reinvested earlier on (4 years ago) the similar captial gains to buy land from house sale.
10 February 2019 have houses (joint-
H.3 INCREASING- SAMPLES IDENTIFIED BY ASK-LLM
We look at the training samples that ASK-LLM scoring models disagree on go from Flan-T5- Small + Flan-T5-XX T5-XXL. Specifically, we look at training samples that quality, whereas X5-X-L thinks otherwise. To the best of our understanding, our overarching conclusions by observing these qualitative samples are:
thinks are of low
Larger scoring models in ASK-LLM able identify training samples containing tail-end of knowledge, egg., rare world-events, rare named entities, etc.
The increasing quality trend going from Flan-T5-Small + Flan-T5-XX isn't correlated with the quality scoring model size in perplexity filtering.
Example 17: Estimated Data- aata-Quality (Percentile Higher is better)
Perplexity Filtering
Small
Base
Lange
XL
XXL
Small
Base
Large
XL
XXL
30.45%
57.41%
78.17%
97.41%
15.56%
31.02%
24.14%
50.59%
49.64%
The historic city of Manchester features of the most interesting public art installations that art lovers have ever witnessed. Design studio, Acrylicize installed five giant lamps in Piccadilly Place that represent the many historic periods that the city has gone through. including; Art Deco, Art Nouveau, Victorian, mid-century, and contemporary. The installation is without any doubt, great piece of art but unlike other artworks, these are absolutely functional well. Each lamp provides the many visitors with seating, shelter, light and even heat in the winters. The admirers can also witness the historic stories of Manchester via graphic illustrations on the lamps.
Example 18: Estimated Data- Quality (Percentile Higher is better)
Perplexity Filtering
Small
Base
Large
XL
XXL
Small
Base
Large
XL
XXL
10.48%
31.26%
54.17%
84.17%
97.93%
30.52%
39.49%
35.79%
30.89%
25.39%
The Cokin Yellow and Pink Center Spot filter has clear center and diffused yellow and pink edges. Theses diffused edges will be produce blur while leaving the center sharp. The filter effect is directly influenced by the f-stop and the focal length. A lens shot at f/1.4 will see greater blurring effect than f/8.0 and 85mm lens will see more blur than 28mm. Additionally, longer focal length lens will visually increase the size of the center spot area because it sees less of the filter area.
48

---

Published as a conference paper at ICLR 2026
Example 19: Estimated Data- Quality (Percentile Higher is better)
Ask-I.M
Perplexity Filtering
Small
Base
Large
XI.
XXI.
Small
Base
Large
XT.
XXT.
20.29%
38.23%
63.94%
22.41%
14.8%
12.69%
S
Provide hoist coverage and 200 degree rotation for individual use in bays, along walls, or columns of plants, or as supplement to an overhead crane or monorail system. This jib has the advantage of providing maximum lift for the hoist, since can be installed very close to the underside of the lowest ceiling obstruction. It is composed of vertical mast mounted to brackets on wall or vertical building beam with a boom that cantilevers out, perpendicular from the wall at the top.
Example 20: Estimated Data- aata-Quality (Percentile Higher is better)
Ask-LLM
Perplexity Filtering
Large
Large
20.76%
45.81%
60,225
73.95%
84.14%
2.98%
2.94%
3.49%
2,51%
2.09%
The mighty Adyar River that flows through Chennai has tale to tell. Arun Krishnamurthy, founder, Environmentalist Foundation of India has documented the origin of the river, the journey and the culmination all captured in images aimed at sensitizing citizens of Chennai treasure that they are being denied. Titled Urban Waters, the photo exhibition on Adyar river will bring out Adyar's 's rich history. fine ecology, urban exploitation and her innate beauty through framed images. The exhibition is organised at Max Mueller Bhavan Chennai. Goethe Institut, Max Mueller Bhavan is at 4, 5th Street, Rutland Gate, Chennai.
Example 21: Estimated Data- auta-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
XL
XXL
Small
Large
XL
XXL
4.27%
22.22%
47.57%
82.58%
92.4%
6.34%
4.77%
3.89%
8.75%
7.55%
The Pendaries Village Skyline Subdivision is located near both the Santa Fe National Forest and the Pecos Wilderness in North Central New Mexico. It has the charm of small town New Mexico, perhaps even more so than its better known nearby sister cities. It offers a unique opportunity for people wishing enjoy the quiet beauty of Northern New Mexico.
49

---

Published as a conference paper at ICLR 2026
Example 22: Estimated Data- Quality (Percentile Higher is better)
AsK-LLM
Perplexity Filtering
Small
Base
Large
XI.
XXI.
Small
Base
Large
XT.
XXT.
22.09%
66.57%
76.56%
85.51%
96.98%
17.42%
15.55%
S
Anderson PPaak's 's new album, Oxnard, is nod the Southern California city where Anderson grew up. It is the Grammy- nomumntto artist's third studio album and the first to be released on Dr. Dre's label Aftermath Entertainment. Oxnard includes his latest single, Tints featuring Kendrick Lamar along with album features from J Cole, Pusha T and many more. This is the album he dreamed of making in high school, when was listening Jay-Z's The Blueprint, The Game's is The Documentary, and Kanye West's The College Dropout. The classic fourth album from the rap-god Eminem.
Example 23: Estimated Data- Data-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Basc
Large
XL
Small
Base
Large
XL
XXL
0.98%
24.84%
53.36%
88.98%
98.18%
2.3%
1.48%
2.03%
2.1%
3.07%
The Disknet is networking solution which uses the external floppy drive port of the Amiga. uses the same coax cabling as 10Base2 Ethernet (RG-58U/500hm) but is NOT compatible and is capable of transferring at around 45k/sec. The Disknet may be the same device as the AmigaLink, but this has not been confirmed.
H.4 DECREASING-QUALITY SAMPLES IDENTIFIED BY ASK-LLM
We look at the training samples that ASK-LLM scoring models disagree on as we go from Flan-T5- Small - Flan-T5-XXL. Specifically, we look at training samples that Flan-T5-XXL thinks are of low quality, whereas T5-mall- thinks otherwise. the best of our understanding, our overarching conclusions by observing these qualitative samples are:
Smaller quality-scoring models sometimes mislabel non-informative training samples, that contain, e.g., non-informative content, or repeated content.
The decreasing quality trend going from Flan-T5-Smal Flan-T isn't correlated with the quality scoring model size in perplexity filtering.
Example 24: Estimated Data- quta-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
XL
XXL
Small
Base
Large
XL
XXL
64.05%
46.39%
35.92%
25.29%
9.63%
4.3%
10.21%
3.47%
3.34%
3.35%
one filled with goodwill and cheer. who have supported me thru the year. wouldn't 't be changing carcers. instead of strange people's rears. Wishes You Healthy, Happy Holidays! Ah, how the mighty have fallen! And a Merry fave to you ... and happy new rear. From Xmas humor story another, enjoyed this! Thanks Jack & Susan! Doug, I checked him out- wonderull stuff! Will pass along the good word. Fun and funny-as always! Thanks for the cheer! I can only fave this once, but I've looked at it repeatedly been a bizarre week- and each time you't 'e given me laugh. That's gift Bob and I'm grateful! Best of holidays you and great New Year!
what has
50

---

Published as a conference paper at ICLR 2026
Example 25: Estimated Data- Quality (Percentile Higher is better)
AsK-LLM
Perplexity Filtering
Small
Base
Large
XI.
XXL.
Small
Base
Large
XT.
XXI.
91.25%
53.1%
24.11%
4.53%
32.4%
36.56%
46.53%
48.19%
54.84%
S
hear people saying that vinyl records have better sound quality than CDs or even DVDs. A mini LP is CD version of something that was originally released as 12" (12 inch) vinyl LP.. In many cases the packaging is superior to, at least. Vitalogy; Studio album by Pearl Jam; Released: Vinyl: November 22, 1994 CD: December 6, 1994: Recorded: November 1993 October 1994: Studio: Bad Animals Studio. Browse best sellers, new releases, AutoRip CDs and vinyl records, deals, vinyl Audio CD. 7.99. From A Room: Volume 1. Chris Stapleton. Audio. The one and only CD, DVD, VIDEO, DJ, VINYL, ERO store. Search full catalog. Recordstore.co .tcitorrecc...... uk. The UK's leading online record store. Buy and exclusive signed bundles, CDs, LPs, Merchandise and box sets. Recordstore Day, every. Vinyl Records CD Conversion Cheapest the net! High-quality, standards-compliant CD-Audio of your favorite vinyl records, saved for posterity. Custom CD, DVD Vinyl Packaging You'le just click away from gorgeous, retail-ready CD or DVD in professional disc packaging. We also offer full-range of Vinyl.
Buy with confidence as the. Mar 4, 2017 Despite the decline in mainstream CD usage, some consumers still have CD recording needs for radio, vinyl and other formats. Here are our. 12 results. You can finally burn your cassettes and vinyl records to CD with Crosley's s Memory Master II CD Recorder. Just play your cassette or record One Nation back after the Sold Out New Years Eve event with yet another From its esoteric origins releasing field recordings of steam engines on vinyl to our latest critically acclaimed Ultradisc THRTM SACDs, Mobile Fidelity Sound. How much are worth and valued your rare and collectable vinyl and ed by searching on Music Price Guide archive. Heel veel CD, LP, Vinyl SACD op voorraad, snelle levertijden altijd superscherp geprijsd lage verzendkosten, voor 17:00 besteld morgen Some of the greatest music ever made isn available digitally, on mp3, or on CD; but rather is only available vinyl. Moreover, if you already have purchased.
51

---

the seven Published as a conference paper at ICLR 2026
Example 26: Estimated Data- Quality (Percentile Higher is better)
AsK-LLM
Perplexity Filtering
Small
Base
T Large
XI.
XXL.
Small
Base
Large
XT.
XXT.
96.67%
76.07%
47.33%
30.0%
32.02%
24.31%
25.77%
brilliant performance by Year based on The Lion King. Brilliant singing and acting from everyone, congratulations Year 6! A big thank you to all the staff that helped with everything from costumes, set design, make up and directing. A wonderful commemoration years that Year 6 students have spent at The Good Shepherd. Thank you to all of the parents and staff for attending this celebration and we wish all of the children continued success in their new schools and hope they continue to do themselves proud. Well done to Foundation for showing us what is to be good friends! This week we have been looking all countries in the world that speak Spanish their native language, there are 21! So throughout school we spent a day learning lots of wonderful things about our chosen country. We looked at maps, flags, famous people, food and much more! Below is little glimpse into our fabulous week.
Click on the links to take a look at some of the brilliant things we got up to! Faith in Families is charity based here in Nottingham who believe, as we do, that all children have the right to grow up as part of a loving and nurturing family and they provide services for children and families. We learnt lots about adoption and what it can mean for children and their family. We learnt about Fairtrade and all the fantastic work they do around the world. We also discovered lots of products that we did know were Fairtrade. There was also sell out Fairtrade food sale, well done everyone! Year have been able to show off our brilliant new high visibility jackets! Now we will be able to stay safe and visible on any out school trips. We very lucky to have these donated by Walton & Allen. Thank you! Click on the high visibility jacket to take a look at our super jackets! Year have wowed us with their acting skills in brilliant performance of Ali Baba well done Year 4! Year...
52

---

Published as a conference paper at ICLR 2026
Example 27: Estimated Data- Quality (Percentile Higher is better)
Perplexity Filtering
Small
Base
Large
XI.
XXI.
Small
Rase
Large
XI.
XXT.
75.97%
18.06%
16.88%
14.36%
13.67%
S
Search result for For Sale We supply Germany made embalming powder in small quantities from kg at affordable prices. We have white and pink 100% hot and 98% pink in stock. Call us on +27786893835 for details. EMBALMING. EMBALMING POWDER CALL +27786893835 Hager Werken Embalming Compound Pink Powder call +27786893835 General items from Germany Embalming compound in powder form both PINK and WHITE Radio active.. Sierra Residences Type B, Sg Ara near PISA, Factory.Air-por Sierra Residences (ID: 5695)
Monthly Rent: RM 1,000 BU: 1182 sq.ft. Newly Renovated/NOT Furnished 3.. Very Strategic and Highly Potential LAND 9.7 Acres Converted Residential Land For Sale in Taman Melawati !!!!!
Taman Melawati development land, Titile Freehold, non bumi land. Status... I am Certified Private Loan Lender, Do you need a Fast and Guarantee loan to pay your bills or start up a Business? I offer both local and international Ioan services to meet your financial needs..
Introducing our mining company to you for a very fruitful business transaction. we are miners who have come together to upgrade our production through the introduction of modern technology and.. Commercial land for sale. Location near to Premium Outlet. Size 32 acres Good land shape and very suitable for development. Selling price RM per sf. Interested party kindly contact.. Keterangan: Tanah yang rata dan sangat startegik untuk buat rumah kediaman/rumah rehat (homestay), atau untuk rumah penginapan sendirian/Percutte (vacation home) Tanah lot tepi berdekatan.. Limited gated Semi D Sri petaling, fully furnish with lift and move in condition. practical.Prim Prime location for stay.good gated security and easy access few main.. Land for sale in MELAKA Price: RM 65 per sq fit (or roughly USD 17 per sq fit Size : 53000 sf Property type 144ffrechold housing land Location Jalan Laksamana Cheng Ho.Å
muit,modern,spatt
and
53

---

Published as a conference paper at ICLR 2026
Example 28: Estimated Data- Quality (Percentile Higher is better)
Ask-LLM
Perplexity Filtering
Small
Base
T Large
XI.
XXT.
Small
Rase
Large
XI.
XXT.
94.72%
78.07%
5.75%
9.63%
13.12%
17.51%
17.12%
FIFA 20 CONFIRMED TRANSFERS SUMMER 2019 RUMOURS w/ ALEX SANDRO BALE & NEYMAR JR. TO BARCELONA!! Top 10 Worst Transfers In Football History!
Thank You For FIFA 20 CONFIRMED TRANSFERS SUMMER 2019 & RUMOURS w ZIDANE COUTINHO & RONALDO BACK TO R.MADRID! REBUILDING REAL MADRID DREAM TEAM LINEUP 2019-2020 POTENTIAL TRANSFERS w/ NEYMAR & RONALDO! FIFA CONFIRMED TRANSFERS SUMMER 2019& & RUMOURS w BALE FEKIR UMTITI & NEYMAR E300M TO MADRID! SUBSCRIBE htpp://il.yyScccerAMubbb Dean from 442oons is back with his list of the top deals that done transfer deadline day. Do you agree with... FIFA 20 CONFIRMED TRANSFERS SUMMER 2019 & RUMOURS w STERLING JAMES AUBAMEYANG GRIEZMANN! SUBSCRIBE FOOTBALL DAILY: http://ii..y/ffsubssset /tdddb//ddbbbbbeeeeeeee Last week we broke down our best signings of the summer so far. Now lets expose the worst! Top 150 confirmed transfers signings of the summer transfer window 2018 ft. Ronaldo, Mbappe, Mahrez, Vidal, Courtois... THANK FOR WATCHING! FIFA 20 CONFIRMED TRANSFERS SUMMER 2019 & RUMOURS POGBA SANCHO THIAGO & MESSI TO INTER!!
70 CONFIRMED TRANSFERS JANUARY 2019 Watching
Like Subscribe
Example 29: Estimated Data- Mata-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
X1.
XXL.
Small
Base
Large
XL.
XXL
86.25%
69.2%
61.9%
46.57%
19.99%
76.61%
71.9%%
94.86%
92.93%
94.99%
Phone 1300 616 202 if you're looking for trustworthy, experienced and licensed Plumber Leopold. know that getting plumbing repairs Leopold can be a pain and you've got better things do than look for plumber. Clearwater Plumbing and Maintenance will save you from any unnecessary hassle and expense for a Plumber Leopold. We make sure that wherever you need Plumber Leopold, Clearwater Plumbing and Maintenance will assist you with your plumbing worries. Plumbing problems with your taps, toilets, gas, hot water and drains painful enough. You don't need the extra stress of finding Plumber Leopold that you can trust. And what about all of those plumbers in Leopold who don't 't clean up after themselves, leaving mud and materials all over your home? Our professional team are different!
Do you have hot water system repairs Leopold. We have highly experienced plumbers who know how to fix hot water systems Leopold. There can be many possible reasons why your hot water system Leopold is broken. Our Leopold plumbers diagnose problems. Our hot water system repairs Leopold plumbers are trained and qualified. To book appointment, please call 13300 616 202. We will do best get plumber you in Leopold as soon as possible. If you notice that there is water leaking from the bottom of your hot water system in Leopold, chances are the system is completely broken. In this scenario, you will need to replace your hot water system in Leopold. Our team of plumbers can help you to choose what hot water system you will need.
reliable, fast and know hot to
54

---

Published as a conference paper at ICLR 2026
Example 30: Estimated Data- Quality (Percentile Higher is better)
Perplexity Filtering
Small
Base
Large
XL.
XXI.
Small
Base
Large
XL.
XXT.
82.64%
75.2%
29.51%
8.94%
78.34%
82.07%
91.01%%
88.02%
You can now configure the minimum TLS protocol level for client connections and connections to other servers. Refer to the following page for more information: Advanced TL... You can now set an Integrated Capture Point (ICP) to stopped mode by changing the state of the corresponding configuration object to disabled; changing the state to enabled restarts the inbound cycle of the ICP. You can now set the minimum TLS protocol level for the Web Service Capture Point by configuring the option <sec-protocol> in the section <settings> of the Capture Point object.
Support for the following databases. See the Supported Operating Environment: eServices page for more detailed information and list of all supported databases. No special procedure is required to upgrade to release 8.5.201.05 Retrieved from NanaaaionnRnmmm-------SSSSi--n (2019-04-21 22:59:48)" This page was last modified on November 8, 2018, at 08:48.
Example 31: Estimated Data-Quality (Percentile Higher is better)
ASK-LLM
Perplexity Filtering
Small
Base
Large
XXL
Small
Base
Large
XXL
62.21%
54.71%
35.73%
22.64%
6. 6.76%
64.82%
85.95%
94.65%
93.35%
85,29
are willing to provide you with perfect services and striding for Display Stand For Boutique, Display Stand for Boutique, Display Stand for Phone Our product quality one of the major concerns and has been produced to meet the customer's standards. "Customer services and relationship" is another important area which we understand good communication and relationships with customers is the most significant power run it long term business. "We have quite a few great team customers very good at internet marketing, QC, and dealing with kinds of troublesome trouble while in the output approach for Display Stand For Boutique. Display Stand for Boutique, Display Stand for Phone. We set a strict quality control system. We' ve got return and exchange policy and you exchange within 7 days after receive the wigs if it in new station and we service repairing free for our solutions. You should feel free contact for further information and we are going to give you competitive price list then.
55


Reviews:
[{"review_id": "GsTQThCczH", "reviewer": "Reviewer_1AMo", "summary": "- This paper addresses the critical challenge of efficient data selection in LLM pre-training. The core research question investigates the trade-offs between data selection strategies based on data quality versus those based on coverage. \n- To explore this, the authors propose two novel data curation techniques: (1) ASK-LLM, a quality-based sampler that leverages the zero-shot reasoning of an instruction-tuned LLM to directly score the informativeness of each training example, and (2) DENSITY, a coverage-based sampler that uses kernel density estimation on data embeddings to select a diverse subset of the data.\n- The authors conduct an extensive empirical study by pre-training T5-style models (60M and 800M parameters) on various subsets of the C4 dataset curated by 22 different methods.", "questions": "Same as the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The motivation of this paper is clear: selecting high-quality and high-coverage data for efficient pretraining.\n- While model-based data selection has been explored, the use of an instruction-tuned LLM's explicit, prompt-based reasoning to assess the quality of individual pre-training samples is a novel and powerful concept.\n- The experimental results are highly insightful, demonstrating that (1) compared to other model-based methods (e.g., PPL), the proposed ASK-LLM leverages the reasoning and context ability from the instruct-tuned LLM for long-tail high-quality data selection. (2) With the improvements of the scoring model, there are increasingly beneficial effects as the scaling of the to-be-trained model. (Perplexity filters do not seem to exhibit such trends.) \n- The paper is well-written, well-organized, and easy to follow.", "weaknesses": "- This work conducts all experiments on the T5 series encoder-decoder LLMs. As of late 2025, the vast majority of state-of-the-art open-source research and applications are centered on decoder-only LLMs, like Qwen/Llama/Gemma/Mistral. While the authors acknowledge this limitation, the absence of any experiments on a decoder-only architecture makes it difficult to assess the direct generalizability of the findings. \n- The paper argues that the high inference cost of ASK-LLM is amortized over multiple training runs. However, scoring a trillion-token dataset is much expensive. The paper would benefit from a more concrete cost analysis. For example, a simple table estimating the TPU/GPU-hours required to score the full C4 dataset with Flan-T5-XL versus the compute saved from the accelerated T5-Large training run would provide a much clearer picture of the return on investment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQdM8BRYFw", "reviewer": "Reviewer_WkEw", "summary": "This paper investigates data-efficient pre-training methods for LLMs to reduce high training costs. It contrasts two primary data selection strategies: data quality and data coverage. The authors introduce two novel methods to represent these strategies:\n1. ASK-LLM: A quality-based sampler that uses an instruction-tuned LLM to perform a zero-shot, prompt-based assessment of a data point's quality.\n2. DENSITY: A coverage-based sampler that selects a diverse set of examples by modeling the data distribution in the feature space.\n\nIn a large-scale experiment testing 22 different curation techniques on T5-style models, ASK-LLM was the top performer. The key finding is that while coverage-based samplers typically match the performance of training on the full dataset, the quality-based ASK-LLM consistently outperforms full-data training.", "questions": "1. The paper's evaluation relies entirely on post-finetuning performance across 111 tasks . How do you expect the ASK-LLM curation to affect the zero-shot and few-shot reasoning capabilities of the pre-trained models? Is there a risk that filtering for \"informative\" data (as defined by the prompt) optimizes for finetuning performance at the expense of the broad, general-purpose in-context learning abilities that are a key benefit of pre-training? \n2. The ASK-LLM sampler requires a full inference pass over the entire dataset, which is a significant, up-front computational cost. While you argue this cost is amortized, could you provide a more concrete cost-benefit analysis? For example, using the T5-Large model, how many compute-hours did the ASK-LLM (XL) scoring pass on the full C4 dataset take?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's conclusions are supported by a massive-scale experiment. The authors tested 22 different data curation techniques , involving 220 pre-training runs and 1,100 distinct fine-tuning runs , all evaluated on 111 downstream tasks. This comprehensive and costly analysis provides high-confidence evidence for their claims.\n2. The primary strength is the clear finding that quality-based filtering (ASK-LLM) can train models that outperform models trained on the entire dataset, even when using as little as 10% of the data. This is a significant result because it shows data curation is not just about matching full-data performance more efficiently, but about achieving a new, higher performance ceiling.\n3. The proposed ASK-LLM sampler is a novel and intuitive method that leverages an LLM's reasoning to score data quality. This approach is shown to be superior to traditional perplexity filtering because it captures a different, more valuable signal. The paper demonstrates that ASK-LLM scores have \"almost no ranking correlation\" with perplexity scores and that it correctly identifies high-quality, niche-topic examples that perplexity filtering wrongly penalizes.", "weaknesses": "1. The ASK-LLM method requires one full LLM inference pass for every single example in the training corpus. For a dataset like C4 (364M examples), this is a massive computational cost that could rival the cost of training itself. While the authors argue this cost is \"amortized\" over many training runs, this presents a very high, practical barrier to entry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qzoJEYgM3v", "reviewer": "Reviewer_rgfd", "summary": "This paper studies data-efficient pretraining of large language models (LLMs) by systematically evaluating two data selection strategies integrated into the pretraining pipeline.The proposed ASK-LLM method uses an instruction-tuned model to score each pretraining sample via prompting for quality judgment, selecting data with high predicted informativeness and linguistic coherence. In contrast, DENSITY estimates sample representativeness using embedding-space density and selects a diverse subset that maximizes coverage of latent topics. Both methods are applied before model training as a data filtering stage, followed by standard T5 pretraining on the selected subsets under a fixed compute budget. The study compares 22 sampling schemes on subsets of the C4 corpus using T5-small and T5-large, and evaluates downstream on 111 tasks. Results show that ASK-LLM substantially improves data efficiency—matching or surpassing full-data training with 10–60% of data and converging up to 70% faster—while DENSITY performs best among coverage-based baselines. Random sampling remains competitive but inferior to the LLM-guided quality filtering approach.", "questions": "- Did the authors experiment with alternative prompt formulations or response schemes for ASK-LLM—for example, using scalar or graded scoring instead of binary yes/no judgments? If not, how confident are they that the binary formulation captures sufficient quality signal?\n- How sensitive are the ASK-LLM filtering results to the choice of the LLM judge? The paper suggests that different Flan-T5 judge sizes yield similar filtering behavior—could the authors clarify where this consistency arises from?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper studies how different data sampling strategies influence the efficiency of LLM pretraining under a clearly defined and reproducible setup.\n- It conducts extensive and controlled experiments, covering 22 sampling strategies, over 200 pretraining runs, and 1,100 downstream fine-tunings under a fixed-compute regime.\n- The methodology is clearly framed through a contrast between quality-driven (ASK-LLM) and coverage-driven (DENSITY) sampling, offering a structured perspective on data selection principles.\n- Empirical results are consistent across model scales and sampling ratios: ASK-LLM yields larger gains in low-data regimes, while DENSITY maintains stable coverage performance, supported by convergence and ablation analyses.\n- The experiments include explicit reporting of compute budgets, dataset proportions, and evaluation metrics, allowing clear interpretation and comparison of the data-efficiency results.", "weaknesses": "- The ASK-LLM sampler requires one inference per training example, which introduces notable computational cost. While the authors mention this can be amortized across multiple runs, the initial scoring overhead may affect scalability to larger datasets.\n- The baselines do not include comparisons with more advanced recent methods such as Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025).\n- All experiments are conducted on encoder–decoder (T5) models using text-only corpora, and the generalization of the findings to decoder-only or multimodal architectures has not been evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RQZ1WyIq3I", "reviewer": "Reviewer_72fC", "summary": "This paper explores how to efficiently train Large Language Models (LLMs), with its core focus on investigating the impact of data selection on pre-training efficiency and model quality. It proposes two new data sampling methods:  \n\n#### ASK-LLM (Quality-Based)  \nThis approach that leverages the zero-shot reasoning capability of an existing, instruction-tuned LLM (a \"proxy model\"). Through specific prompts, it \"asks\" this proxy model to directly evaluate whether each training sample is \"information-rich and high-quality.\"  \n\n#### DENSITY (Coverage-Based)  \nThis method aims to maximize data diversity. It models the data distribution and prioritizes selecting samples from regions that are \"under-represented\" in the distribution.  \n\n\nThe paper conducts large-scale empirical research: it tests up to 22 different data curation techniques on the T5 model, involving hundreds of pre-training runs and thousands of fine-tuning evaluations.  \n\nIt draws key conclusions regarding \"quality\" vs. \"coverage\":  \n- Coverage-based sampling (e.g., DENSITY) can usually \"recover\" or \"match\" the performance achieved by training with the full dataset.  \n- Quality-based sampling (e.g., ASK-LLM) is capable of \"surpassing\" the performance of training with the full dataset.  \n\nExperiments show that even when using only 10% of the original data, ASK-LLM trains models that outperform those trained on 100% of the data—and with a 70% faster convergence rate.", "questions": "1. What is the true cost-benefit trade-off?  \nIn Section 6 and Appendix C.1, the paper brilliantly argues that the high scoring cost of ASK-LLM is a \"one-time amortizable\" investment.  \n\nMy question is: Given today’s dataset scales of 10T+ tokens, using a powerful proxy model (7B+ parameters) to perform a full inference on every training sample makes this \"one-time\" cost extraordinarily high.  \n\nThe paper claims that training speed is accelerated by 70%, but if the scoring process itself requires computational resources equivalent to (or even exceeding) the total training time, what would the net benefit of this method be in practice?  \n\nI would very much like to see a more direct cost analysis: (scoring cost + 70% of training cost) vs. (100% of training cost). The paper fails to provide this critical comparison.  \n\n\n2. What exactly is the \"Gemma 3 variant\"?  \nThe paper’s strongest (and most clever) response to concerns about \"scalability\" comes in Section 6, where it states: \"...a variant of our ASK-LLM technique was used in the development of the Gemma 3 family of models.\"  \n\nMy question is: The paper’s key scalability claim relies entirely on the word \"variant.\"  \n\nHow different is this \"variant\" from the method proposed in the paper (Figure 3)?  \n\nDoes it still involve \"asking\" with a complex prompt containing multiple criteria (informativeness, format, safety)? Or has it been reduced to a small component within a larger, more complex filtering pipeline?  \n\nIf this \"variant\" is significantly different from the original method, the generalizability of the paper’s main conclusions (drawn from T5-800M) to the scale of Gemma 3 becomes far less certain.  \n\n\n3. Prompt Sensitivity  \nThe entire magic of ASK-LLM is encapsulated in the prompt shown in Figure 3. This prompt itself is a complex, multi-dimensional instruction that requires the model to simultaneously judge \"informational signals,\" \"format,\" \"knowledge,\" and \"harmful content.\"  \n\nMy question is: How sensitive are the results to the wording of this prompt?  \n\nWhat would happen if I removed the criterion \"contain usable knowledge\" and only kept \"well-formatted\"?  \n\nIf I only asked \"whether it contains harmful content,\" would it function as a safety filter?  \n\nThe paper provides no ablation study on this newly introduced, complex \"prompt hyperparameter.\"  \n\n\nIn summary, these questions do not negate the merits of the paper, but they are the answers I would most want to know when considering deploying this method in practical production or conducting follow-up research.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The greatest strength of this paper lies in its proposal of the innovative ASK-LLM method.  \n\nInstead of adhering to traditional perplexity or classifier scoring, it pioneers the use of the zero-shot reasoning capability of an existing LLM. Through specific prompts, it \"asks\" the model to directly determine whether a data point contains \"informational signals.\"  \nASK-LLM explicitly resolves several key flaws of traditional perplexity filtering:  \n- **Avoiding meaningless content**: Perplexity filtering tends to select high-frequency, repetitive \"nonsense\" (e.g., \"Example 14\" and \"15\"), while ASK-LLM can identify such content as low-quality.  \n- **Preserving context**: Perplexity filtering incorrectly selects context-deficient fragments like \"questions without answers,\" whereas ASK-LLM can properly recognize these as non-\"informational\" data.  \n- **Retaining \"long-tail knowledge\"**: Perplexity filtering mistakenly discards valuable yet \"niche\" knowledge containing rare words (e.g., \"Example 17\"), but ASK-LLM can leverage its reasoning ability to identify and retain such \"long-tail knowledge.\"  \n\nThe paper features an impressive volume of experiments, making its conclusions highly credible.  \nThe paper compares up to 22 different data curation techniques.  To complete this comparison, the researchers conducted 220 independent pre-training runs and 1,100 downstream task fine-tuning evaluations. In the field of LLM pre-training, such a large-scale comparison is extremely rare and valuable.  \nExperiments were conducted on two model sizes (T5-Small and T5-Large) and five data sampling rates (10% to 80%), ensuring the robustness of the conclusions.  \n\nThe paper clearly answers the core question raised in the introduction: \"Which matters more, Quality or Coverage?\"  \n- The upper limit of coverage-based sampling (e.g., DENSITY) is to **\"recover\" or \"match\"** the performance of full-data training.  \n- Quality-based sampling (e.g., ASK-LLM) can **\"surpass\"** the performance of full-data training.  \n\nThe paper does not stop at superficial results but provides in-depth analyses:  \n- **Correlation Analysis (Figure 7)**: By calculating the correlation of scores from different samplers, the paper proves that ASK-LLM, Perplexity, and DENSITY indeed select distinctly different data—confirming that ASK-LLM represents a brand-new evaluation dimension.  \n- **Qualitative Analysis (Appendix H)**: It provides numerous real cases of data selected or discarded by ASK-LLM, intuitively demonstrating why it outperforms perplexity filtering.  \n- **Scoring Model Scaling (Figure 6)**: The paper finds that using a more powerful LLM (from Small to XXL) as the \"judge\" for ASK-LLM leads to better data filtering results. This indicates that the ASK-LLM method itself has excellent scalability and can become more effective in the future as \"judge\" models iterate.", "weaknesses": "1. Concerns Regarding Model/Data Scale and Method Generalizability (Scaling & Generalizability)  \n\nModel Scale  \nT5-Large (800M parameters) is indeed hardly qualified to be called a \"Large Language Model\" (LLM) today—even in the 2023-2024 period. Core challenges and phenomena in pre-training (such as emergent abilities) are most pronounced primarily in models with tens of billions or even hundreds of billions of parameters. Therefore, it is questionable whether the \"optimal\" data strategy derived from an 800M-parameter model can be directly generalized to models with 70B or 1T parameters.  \n\nData Quality and Generalizability  \nC4 is a relatively \"noisy\" corpus that has not undergone fine-grained filtering. For such datasets, the room for improvement through \"data quality filtering\" is inherently large (a scenario akin to \"picking the best from a bad lot\").  \n\nThis gives rise to a risk: the remarkable success of ASK-LLM—reducing data volume by 90%, cutting training time by 70%, while achieving better performance—may overestimate the method’s marginal benefits when applied to cleaner, higher-quality datasets. Examples of such datasets include RefinedWeb, Dolma, or the \"textbook-level\" data used for Phi-2. If a dataset is already of high quality, how much additional improvement can ASK-LLM bring? This is a question the paper cannot answer.  \n\n2. Another issue is that the related works cited in the paper are relatively outdated, with a lack of recent relevant studies. This makes it difficult to assess whether the paper still holds innovativeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IWszn7ZsSK", "reviewer": "Reviewer_THC1", "summary": "This paper explores how to balance data coverage and model quality in large language model training. The goal is to find sampling strategies that minimize the amount of data needed while still maximizing model performance, essentially pushing the Pareto frontier of efficiency.", "questions": "1.The ASK-LLM approach is innovative — using the model itself to assess data quality rather than relying on external heuristics — but it may risk creating a “self-reinforcing bubble,” where the model’s own biases limit data diversity.\n    \n2.The ASK-LLM method is computationally costly. It might be worth exploring hybrid strategies, such as integrating active learning to selectively annotate boundary samples and iterate.\n    \n3.While the empirical evidence is rich, the paper would benefit from a clearer theoretical explanation of why ASK-LLM captures data quality better than traditional perplexity-based filtering.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.The study is timely and relevant, offering solid empirical evidence that directly contributes to the ongoing conversation on efficient LLM training.\n    \n2.The experiments are extensive — covering a wide range of model sizes (from 60M to 800M parameters) and 22 different sampling strategies — giving the conclusions strong credibility.", "weaknesses": "1.All experiments are conducted on T5-style encoder–decoder architectures, with no validation on the more widely used decoder-only models (like GPT). This limits the generalizability of the results.\n    \n2,Although ASK-LLM demonstrates efficiency, it still requires scoring every single sample in the dataset, which makes it computationally expensive. The practical scalability and real-world deployment cost are not fully addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)