Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
Authors: Alex Fang
Hadi Pouransari
Matt Jordan
Alexander T Toshev
Vaishaal Shankar
Ludwig Schmidt
Tom Gunter
Venue: NeurIPS 2025 poster ()
Keywords: datasets
llm
repetition
deduplication
scaling
pre-training
language models
data filtering
Primary area: deep_learning

Abstract: Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, as large language model compute budgets continue to grow, the limited data volume provided by heavily filtered and deduplicated datasets will become a practical constraint. In efforts to better understand how to proceed, we study model performance at various compute budgets and across multiple pre-training datasets created through data filtering and deduplication. We find that, given appropriate modifications to the training recipe, repeating existing aggressively filtered datasets for up to ten epochs can outperform training on the ten times larger superset for a single epoch across multiple compute budget orders of magnitude. While this finding relies on repeating the dataset for many epochs, we also investigate repeats within these datasets at the document level. We find that not all documents within a dataset are equal, and we can create better datasets relative to a token budget by explicitly manipulating the counts of individual documents. We conclude by arguing that even as large language models scale, data filtering remains an important direction of research.

Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
Alex Fang 1.2 Hadi Pouransari!
Matt Jordan
Alexander Toshev
Vaishaal Shankar
Ludwig Schmidt
Tom Gunter*
Apple 2Stanford
3 Swork done while at UT Austin
*work done while at Apple
Abstract
Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, large language model compute budgets continue to grow, the limited data volume provided by heavily filtered and dedupli- cated datasets will become a practical constraint, In efforts better understand how to proceed, we study model performance at various compute budgets and across multiple pre-training datasets created through data filtering and deduplication. We find that, given appropriate modifications to the training recipe, repeating existing aggressively filtered datasets for up to ten epochs can outperform training on the ten times larger superset for single epoch across multiple compute budget orders of magnitude. While this finding relies on repeating the dataset for many epochs, we also investigate repeats within these datasets at the document level. We find that not all documents within dataset are equal, and we can create better datasets relative to token budget by explicitly manipulating the counts of individual documents. We conclude by arguing that even as large language models scale, data filtering remains important direction of research.
Introduction
Scaling data and compute has been the winning recipe for producing state of the art machine learning models. As costs increase with diminishing returns, recent work often points to high-quality data as key factor for high model performance (30 [). One method of obtaining high quality data is through data filtering. However, as frontier level large language models (LLMs) scale in compute through increases in both parameter count and token count, there concerns that will run out of training data. Data filtering reduces the dataset size, creating a tension between data quality and data quantity. It is currently unclear whether the performance gains from data filtering be sustained at scale through repetitions as an attempt to compensate for the reduction in dataset size.
Recent work has studied data filtering as means of efficiently training strong models with limited compute [5]. However,
unclear if the findings are practical for training larger models. Unfortunately, existing frontier level LLMs lack transparency when it comes to their data filtering pipelines 30 I6]. This gap knowledge adds to concerns about the practicality data filtering on scale.
Understanding the interaction between data filtering and scaling would be made casier if we better understood what it means to be better dataset. Currently, the quality of dataset is determined by using standardized benchmarks evaluate the performance of model trained on it. However, there are other traits we desire in the data beyond the performance of the raw downstream model, such as repeatability and performance over replacement. Developing better understanding of these traits would allow for more principled construction of better datasets.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DCLM 1 opoch
DCLM 12.68 2528 WD
DCLM 1D epach WD
DCLM 12.68 2528
DCLM 10 epoch
RW 12.6 2528
RW epoch
254
Centered Care Metric
9.1
9.9
10
Compute (Chinchilla Optimal)
Epochs (Chinchilla Optimal)
Figure 1: Repeating the aggressively filtered dataset DCI.M-aaseline for up epochs consistently outperforms training on a single epoch of the ten times larger RefinedWeb supserset (cyan)-provided) that we adapt the weight decay for the high-repetitinn runs. On the left, we show that this result appears consistent across compute budgets (model sizes of IB, 3B, 7B, and 12.6B). On the right we show that, the largest compute budget tested (12.6B parameters, 252B total tokens seen), adapting the weight decay as a function of repetition allows for a significantly better result versus training on the superset. Results are evaluated on the centered core metric from DCLM, which normalized average over 22 tasks. We also include a MMLU version of the right side in Appendix C
In what follows, in Section we study how datasets of different quality behave under repetition and explore how to improve multi-epoch performance. We compare DCLM-baseline filtering [15] with superset comprising the strictly looser RefinedWeb filtering 21). well the commonly studied C4 dataset 23 We find that while better datasets are not necessarily more repeatable, the interaction between dataset performance and repeatability depends the compute allocation used to train the model. Furthermore, we study practical considerations for scaling up aggressively filtered datasets by repeating them for multiple epochs account for the reduced number of tokens available. Datasets can be made more repeatable through regularizing faccorr, such as increasing weight decay and increasing tokens per parameter. We find that when training on the same number of total tokens, using ten epochs of the aggressively filtered DCLM- DCLM-baseline can outperform a single epoch of its ten times larger superset- superset-Refinndee RebinedWeb.
Next, in Section 4 we investigate how to create better datasets by analyzing duplicate documents within the dataset. We find that web data filtering results in bias towards documents that are more often partially duplicated, which was also reported in Penedo et al. [22]. Deduplicating the resulting dataset will therefore reduce the precision of the overall filtering step. We suggest simple methods that use quality classifier to manipulate the individual counts of documents to improve dataset quality given a desired token budget, and show that this outperforms using only duplicate counts as in standard deduplication methods.
Lastly, reflect on the role of data filtering for LLMs, and argue that it is key ingredient for improving LLMs no matter the compute budget. We hope that this work not only clarifics some of the limitations of existing data filtering methods, but also demonstrates the value and practicality of data filtering, thus encouraging further research in this important direction.
Related work
Data Filtering We investigate whether data filtering is justifiable when it requires repeating to compensate for the decrease in tokens. Recent works such as DataComp-LMM (DCLM) [5] and FineWeb-edu (22) have shown how data filtering can greatly improve performance. However, these datascts only retain a few trillion tokens, which may not be enough to train state-of-the-art models

---

DCLM 288 on 18 params
DCLM 206 30
2308
14038
2.40
DCLM 568 18 params
DCLM 568 on 38 params
params
DCLM 2308 30 parems
235
Ca Vel Perplexity
2.70
2.65
25
2.40
2.55
318
2018
Total Tokens
FLOPs (6ND)
Figure 2: C4 perplexity function of training dataset and repetition count. Over-repeating data leads to diminishing returns irrespective of the training dataset chosen. For each dataset, we vary the number of unique tokens available, and then vary the training token budget. In the left plot we train IB parameter models to show that heavy repetition results in similar degradation in validation loss for all datasets tested. The right hand side shows that this effect holds at larger compute budgets for similarly over-trained models on the DCLM dataset.
unless trained for multiple epochs. Our study thus takes into account this scenario, where aggressively filtered datasets must be repeated, and compares them with larger but lower quality datasets.
Repeatability We build on the work in Muennighoff et al. [19], which studies training models that are data-constrained. They find that for a fixed compute budget, web-crawl data can be repeated four times before encountering significant diminishing returns compared to training on unique tokens. We will refer to this as repeatability, which is measured for fixed training token budget by comparing the performance of repeating a subset for multiple epochs with that of a single epoch. Unlike this work, we examine the repeatability of crawl-data from two new angles: 1. whether dataset quality affects repetition resilience; if regularization techniques can dampen the impact of over-repetition.
Goyal et al. [9] studies data filtering for CLIP models and argues that data filtering methods must take into account the computational budget. They find that training a ViT-B/32 for 640M samples on 12.8M LAION filtered samples is worse than 128M unfiltered samples, and that CLIP models trained on large amounts of compute should use less aggressive filtering. Although this differs from our findings, there are significant experimental differences due to modality and filtering techniques.
Compute Allocation Explicitly repeating tokens can lead different optimal compute allocations. Hoffmaan et al. [II] studies optimal allocation of parameters and tokens for a training computational budget. Their suggested ratio for parameters to tokens is called Chinchilla optimal, while training for more or less tokens can be considered overtraining or undertraining relative to that model size. Muennighoff et al. [19 find that for repeated data, overtraining is more efficient than staying at Chinchilla optimal.
Deduplication Prior work has shown that training on deduplicated data is desirable because improves performance [13. reduces privacy risks [12]. and reduces memorization [3] However, more recent works like DCLM and FibeWeb-edu use sharded deduplication for engineering and performance reasons. TxT360 [7 takes slightly different approach, using global deduplication followed by upsampling by the natural distribution. We build on existing deduplication methods, and move toward upsampling based quality metrics document level.
3

---

Repeating Filtered Datasets
In parallel to LLMs training on larger datasets, researchers are also focusing on filtering data to improve quality. Together, these factors accelerate concerns about rumning out of training data. In the earlier days of training LLMs, common practice was to train LLMs on a single epoch of data. This is in stark contrast state-of-the-art vision models, where practitioners would repeat data for many epochs. Though single epoch training may have benefits such as preventing memorization in this work we study data repetition across datasets from a purely performance perspective. Muennighoff et al. [19] studied the repeatability of datasets in a data-constrained setting and found that datasets can be repeated for four epochs before noticing diminishing returns compared training on fresh data. Building off this, we examine datasets of varying quality created through data filtering understand whether these data filtering approaches affect repeatability. In this section we study datasets "as is" and treat them as single unit rather than collection of documents. We use original C4, DCLM's reproduction of RefinedWeb (we will refer to this as RefinedWeb and RW v2 interchangeably), and are deduplicated.
13 3.
See Appendix to compare datasets "as is" with datasets that
288
DCLM 288 on 38 params
DCLM 568
0.34
DCLM 568 on params
DCLM 2308 18
params
0.35
Centeree Core Metric
9.42
0.34
0:30
2.58
0.30
0.29
3.34
0.24
225
356
1400
2000
2010
1021 FLOPs (GND)
Tatal Tokons Soon
Figure 3: Downstream accuracy average 22 tasks as different datasets are repeated and overtrained. The left side only contains IB models across variety of datasets and unique tokens, while the right side only trains on DCLM-baseline while varying in model size and unique tokens.
In Figure 2 we study a scenario similar to the findings in Muennighoff et al. [9 but instead start with Chinchilla optimal unique tokens for IB parameter models. On the left side, we observe that the C4 validation perplexity eventually sees diminishing returns when training on C4 after repeating, In addition, we also train on higher quality datasets M-C----aeleeee and RefinedWeb, and find that they have similar repeatability behavior when evaluating on C4 validation perplexity. In this setting we start with a Chinchilla optimal compute allocation and then repeat epochs by fixing the model size while increasing the token count, and compare this to datasets with more available tokens. On the right side we vary model size, but maintain the same dataset sizes that we repeat. This figure suggests that increasing model size while maintaining dataset size decreases repeatability, but there is not enough information to make a conclusion so we will explore this further later in the section. When move to evaluating downstream accuracy across 22 task subset from LLM Foundry (referred to as core centered metric in DCLM), we see in Figure that quantifying repeatability is even less clear. It appears that when comparing using downstream accuracy, repeatability more difficult measure, and it unclear repeating meets the same diminishing returns seen when evaluating C4 perplexity. What is common between the two evaluation settings is that datasets that achieve better performance do not appear to be more repeatable than inferior datasets when examining diminishing returns.
Although better performing datasets are not more repeatable by preventing diminishing returns, in the overtraining setting (Figure 3) better datasets repeated multiple times clearly maintain better performance than inferior dataset that is not repeated. However, it is important to note that

---

overtraining is practical only in certain settings with priorities such as minimizing the inference cost, and there is an assumption Figure there are least enough tokens to train Chinchilla optimal. This assumption does not always hold when LLMss are scaled up, especially when data filtering is involved.
2.00
DCLM 1b_2x
DCLM
2375
DCLM 3b,1x
DCLM 3b 2.5x
Centered Core Metric
2.50
spoch Brsseline
epech
3.255
epoch
36, 2.5x
Epochs
Figure 4: Downstream accuracy average of 22 tasks DCLM- DCLM-baseline baseline dataset as we fix total tokens trained and vary epochs and unique tokens seen. Dotted horizontal line represents training one epoch on the RefinedWeb baseline, which we compare against because DDL.Mebaseline is additional filtering on top of RefinedWeb. Legend denotes model size and multiplier to scale tokens seen at Chinchilla optimal.
In contrast repeating datasets by training for longer, we next examine fixing the token budget and repeating by decreasing the number of unique tokens. This scenario forces us to examine cases where there are fewer unique tokens available than Chinchilla optimal. In Figure we compare for a fixed model size and total token budget how performance varies as you reduce the number of unique tokens available, causing an inerease in the number of repeats for which the model sees the dataset. When there is no longer at least the Chinchilla optimal number of tokens available, repeating the DCLM-baseline dataset with too few unique tokens can lead to worse performance than training for single epoch on its superset of RefinedWeb filtered data. On the other hand, compute allocations with high tokens per parameter seem to benefit more from higher quality datasets, since DCLM can be repeated more times to equalize performance with unrepeated RefineWWeb, Tokens per parameter is the main factor, as increasing model size while maintaining token count decreases DCLM repeatability to equalize with RefinedWeb, and increasing token count while maintaining tokens per parameter does not consistently increase DCLM repeatability to equalize performance with unrepeated WefinedWeb [Figurc] left). This would also explain why unrepeated RefinedWeb does worse than heavily repeated DCLM in Figurq But there are increasing diminishing returns to performance with respect to compute when increasing tokens per parameter, there still needs to be a balance when doing data filtering.
Table 1: Weight decay reduce performance degradation from repeating data. All models 12.6B parameters trained for 252B total tokens. We report results using the centered core metric. The default weight decay 0.0316. Note that confirm WD Ix optimal for single epoch of RefinedWeb, as well as for single epoch of DCLM.
Repeats
WD Ix
WD 2x
WD 3x
10 epochs
0.489
0.523
0.513
5 epochs
0.532
0.539
0.528
2 epochs
0.541
0.546
0.545
1 epochs
0.550
0.547
0.546
5

---

We can mitigate performance degradation caused by repeating data with regularization in the form of increasing weight decay. In Table Il we see that the optimal weight decay value increases as we increase the number of repeats. A reasonable schedule is to scale weight decay by roughly the square root of the number of repeats. With the optimal weight decay values, we see in Figure [ that training outperforms training on RefinedWeb filtered data for a single epoch.
for ten epochs of This is notable because DCLM- takes the top 10% of documents of Refined as scored by a FastText classifier. However, it is important to note that this does not mean DCLM-baseline is strictly better dataset than RefinedWeb filtering. When data is a much more constrained resource than compute, it is likely that diminishing returns from heavily repeating DCLM-baseling will perform worse than repeating RefinedWed filtered data a couple of times.
Repeating Documents To Create Better Datasets
Table 2: 7B models trained on DCLM-baseline for 138B tokens (Chinchilla optimal) using different subsampling methods.
Subsampling Method
CORE (22 task average)
MMLU
Duplicate-aware subsample
34.1
25.
Global deduplication then subsample
41.7
28.5
Uniform subsample
44.4
42.2
Keep docs with duplicates 2 7, then dedup
45.8
40.0
We have established that, even as compute budgets increase, models can remain robust to dataset repetition given the right training recipe. We now investigate whether high quality data repetition may be best implemented at a document level, as opposed to increased epochs on the entire dataset. To better understand this question, we dive into
which contains many duplicates (fuzzy and exact) because it uses sharded deduplication due to engineering constraints. Penedo et al. 22 also uses sharded deduplication, observing that global deduplication does not always improve performance. This is in contrast to prior standard practices of deduplication, which recommended global deduplication and training for single epoch on the resulting dataset.
The performance gains from data filtering clearly demonstrate that not all documents are of equal quality. We can also observe this by comparing subsampling strategies. The standard strategy is uniform subsampling, which results in a dataset that is diverse with very few duplicates, while more likely keeping documents that have more duplicates. A second strategy is to run minhash- based global fuzzy deduplication before subsampling down to the desired token count. Compared to uniform subsampling, incorporating global fuzzy deduplication treats documents with different duplicate counts equally, but both strategies should have few or no duplicates the resulting subset if subsampling significantly. Lastly, using duplicate-awar awaee subsampling. For each group of duplicates, this strategy keeps or removes the entire group with probability equivalent to the duplicate count normalized by the desired dataset reduction. This strategy results in a subset with a significant number of duplicates compared to the other subsampling strategies.
mimic the duplication profile of the original dataset by
In Table [ we observe that duplicate-awas aware subsampling performs the worst, suggesting that keeping significant number of duplicates hurts performance. Uniform subsampling does significantly better than global deduplication before subsampling, which shows that documents in DCLM-baseline with high duplicate count are likely to be of higher quality. This is further supported by the fact that keeping unique copics for documents with greater than seven duplicates achieves competitive performance to uniform; however, this removes over 96% of the documents in DCLM-bascline.
Given token budget and metric correlated with document quality, we can take advantage of these findings by manipulating counts of individual documents through resampling. This is analogous to mixing but document level instead of forming buckets. We use a function for assigning the counts of deduplicated documents within the dataset, adjust the range of the function by estimating how many more times you should repeat the best document in the dataset before including the worst, and adjust the domain of the function based on the number of documents desired. examples of such functions, which include greedily selecting the best documents with a constant function equal to certain count (y=count), and a linear function which would increase the diversity

---

Doc Multiplicity
Doc Multiplicity
Doc Multiplicity
Available
Available
Available
Used from available
Used from available
Used from available
by oversampling
Used by oversampling
Used by eversampling
Doc Quality
Doc Quality
Doc Quality
(a) Greedy copy
(b) Greedy copies
(c) Linear up to copies
Figure 5: Examples of strategies for count manipulation. Note that Greedy copy is similar global deduplication, and can result in worse performance if document quality varies significantly. For example, if the dataset is split evenly between high quality documents and low quality documents, may be better to repeat the high quality documents.
Listing 1: Pseudocode to sample documents using count manipulation with a simple greedy function.
def greedy_ foncttoou exaelooeeeaappeeeeeee (score)
given score, return how many copies of the accopanying document want.
threshold pre-calculated from all scores. in this example, threshold keeps copy of top # documents to reach desired output document count. if score >- threshold:
return
else:
return 0
def sumple_count_manpplle output []
(documents, counts, scores, function);
for doc, count, score in zip (dpccuumsnss, counts, score) Conction(ccorr) for in range (target, if up. random. rand() output. append (doc)
target_coume
count:
return output
of documents when compared to the equivalent greedy strategy with the same maximum count value. Key to this approach is having metric correlated with document quality. Here use the worst ranking within the dataset between FastText score and pre-deduplication document count. In Table subsample from the 3.8T RCLM-bascline down to 138B tokens for training 7B model, and our results show that doing basic document count manipulation can outperform the various baseline subsampling approaches. In this setting of extreme subsampling that reduces trillions of tokens to billions of tokens, the best strategy greedily selecting high quality documents according to our metric without repeating.
Table 3: 7B models trained for 138B tokens DCLM-baseline with different document count manipulation strategies. Figure 5 these strategies. Ensemble uses the worst rank between FastText score and duplication count.
Subsampling Method
CORE (22 task average)
MMLU
DCLM- baseline uniform subsample
44.4
42.2
Greedy copy (dup count)
41.7
28.5
Greedy copies (ensemble)
44.6
38.5
Greedy copy (ensemble)
46.1
46.0
Linear up to 4 copies (ensemble)
46.0
44.2

---

Table 4: 7B models trained for 138B tokens by doing count manipulation duplicate-aware subsampled 280B tokens from DCLM-baseline. In this setting, the greedy yet diverse strategy sees each document approximately 3 times. Metric used is FastText score. Note that using duplication count as the metric would be ineffective here because there are not enough unique documents.
Subsampling Method
CORE (22 task average)
MMLU
DCLM-baseline DA subsample
34.1
25.1
Greedy Diversity
40.4
28.3
Greedy 5 copies
40.4
29.4
Linear up to 5 copies
41.2
30.4
The results in Table are unusual that we are taking an already highly filtered dataset and subsampling even further. Naturally, there enough high quality tokens to train without repeats, hence the success of the greedy yet diverse strategy. However, this becomes impractical as we scale and there are longer enough unique tokens to support this strategy. Therefore the setting where the dataset we are trying to manipulate is closer in size to the desired token count. We duplicate-awan subsample DCLMMbaseline down 280B tokens and apply count manipulation to produce 138B tokens for training a 7B model. As we scale back up. this would allow us to use the strategy on the entire DCLM-baseline to create dataset close two trillion tokens. In Table 4 we see that the best strategy is applying a linear function to balance the tradeoff between exploring diverse documents and exploiting high quality documents. Contrasting the best strategy in the two settings suggests that using a good count manipulation strategy is more important in the setting where data constrained.
now investigate
DCLM-BASELINE
1.0
Exact Dedup
Fuzzy Dedup
0.8
Removal Rate
0.6
0.0
0.5
1.0
1.5
2.0
2.5
3,0
Documents Seen
1e9
Figure 6: In DCLM-baseline, the proportion of duplicates inside the dataset increases as the number we this trend also holds for RefinedWeb, but with lower
of documents increases. Appendix B removal rates.
Combining duplicate-aware subsampling with count manipulation powerful tool for studying datasets. Increasing dataset size typically refers to obtaining new tokens the same quality. DCLM does this by increasing the number of Common Crawl WARC files the same rate increasing the training token budget. However, as seen in Figure 6, when the number of WARC files increases, so does the number of duplicates. We find that DCLM-baseline constructed from the DCLM 7B 2x pool contains 33% fuzzy duplicates, but M-LM--aseline constructed from 20 times more WARC files contains 83% fuzzy duplicates. By uniformly subsampling run smaller scale experiments, this experiment design uses both new data and repetitions when scaling back up. Additionally. it is impractical to deduplicate unfiltered data due to engineering constraints caused by the amount of data, as well as design decisions like what type of deduplication to use. Duplicate- subsampling gives us the tools to design experiments where data scaling happens through providing new data of equal quality.

---

Count manipulation differs from filtering because filtering is restricted by the amount of documents available in the pool. Even with the perfect data quality metric, filtering alone may not create the best dataset because high quality documents do not necessarily appear many times in the dataset. Count manipulation allows for oversampling and is thus complementary to filterin,, depending on traditional filtering techniques like deduplication and filtering based on quality metric. Furthermore, while the experiments presented have built off of DCLM-baseling is aggressively filtered dataset. Count manipulation should be just as if not more impactful on less filtered datasets because there is a wider range in quality of documents within the dataset, making it more likely that repetition of high quality documents improves performance more than seeing low quality documents for the first time.
it is important to remember that
Revisiting the ideas in Section 3 we can replace repeating entire datasets with count manipulation. Though it may appear that datasets as a whole can only repeat set number of times before significant diminishing returns, when examining individual documents it is likely that repeating high quality document many times is still better than seeing a low quality document for the first time. This tradeoff can be captured by the function applied to count manipulation.
5 How to Use Better Datasets
We would like to conclude by arguing that creating better datasets still an important area of research. Our findings in Section suggest that an aggressively filtered dataset like DCLM-baseline can be used as pre-training data at scale by repeating the dataset for multiple epochs with the right recipe adaptations, but this is just one of the many use cases for data filtering. Additionally, our findings in Section I can be used improve existing datasets, and can be especially useful in situations where there is constraint on the availability of unique data.
Better datasets are especially beneficial for training smaller scale LLMs. This can happen when practitioners want to reduce inference cost by training a smaller model or when there is limit on the amount of compute available. In this setting, the amount of data available is no longer a key constraint, which places greater importance on the tokens that are actually seen. Recent works such as DCLM [15 and Fine Web-edu (22) have shown how data filtering can greatly improve performance across models that only vary by the training data used.
High quality data is often used during continual pre-training. Works like DCLM and Llama 3 6 use this technique to introduce higher quality data at the end of pre-training. This technique is separate and often complementary to fine-tuning techniques like instruction tuning. However, we find in Appendix D that for 7B models trained on 138B tokens, using DCLM-baseline for the last 10% or 20% of training performs the same as mixing that same proportion into RefinedWeb. Nonetheless, this technique can be used to introduce high quality data that was not ready at the beginning of the training run, and may help more for smaller amounts of data or different domains. Feng et al. 7 have shown that curriculum learning can be effective when targeting particular domains like math and code.
Lastly, while PCLM-baseling may not contain sufficient tokens for all use cases, it is important to remember that almost all pre-training datasets have filtered component and we can change the filtering ratio based on needs. Throughout this work we have compared DCLM to RefinedWeb because DCLM is created through additional filtering on top of RefinedWeb filtering. As compute budgets for LLM pre-training increase, more emphasis will be placed on looser and more efficient data filtering that achieves high recall.
Limitations Data filtering techniques have their limitations as well. Our understanding of models is only as good as the evaluations we rely on. It is possible that data filtering reduces capabilities or removes knowledge that we desire in our models, but we can only hope to quantify this through evaluations that measure the effects that we care about. Additionally, we focus strictly on web-scraped datasets created for general language modeling. Dataset properties may differ when studying different domains such as code and math. Finally, our experiments are smaller scale than frontier level models, so our trends must continue hold as we scale in order for these findings to apply to frontier level models.

---

Impact Statement
This paper studies data filtering and its role in improving Large Language Models. The societal consequences are similar to those typically discussed in Machine Learning and Large Language Models. One additional consequence we would like to highlight is the potential for bias amplification in dataset creation. Although we are not releasing additional datasets and models, repeating or resampling data can contribute toward certain biases, so researchers should take proper care to evaluate for such when developing and releasing datasets and models.
References
[1] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
[2] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[3] Nicholas Carlini, Daphne Ippolito, Malthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net 2023. URL attps://openreviie www.:nneeetoomdddeeeeeeeee
Christopher Clark, Kenton Lee, Ming-Way Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044, 2019.
[5] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:I803..0 80..00057..55.. 2018.
[6] Abhimanyu Dubey, Abhinav Jauhri. Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman. Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnacve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Iharra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak. Jongsoo Park. Joseph Rocca, Joshua Johnstun. Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Ilama 3 herd of models. CoRR, abs/2407 21783, 2024. doi: URL httpsst/::///o org/10. 40w.0/4880888888 arxiv. 2ww.ii.220..... 2407.21783
00.48550/RXXI..240.222
[7] Steven Feng. Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Maximize your data's 's potential: Enhancing Ilm accuracy with two two-phase pretraining. arXiv preprint arXiv:2412.15285 2024.
[8] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Mitchell Wortsman Ryan Marten, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Mehdi Cherti Richard Vencu, Ranjay Krishna, Pang Wei Koh. Olga Saukh, Alexander Ratner, Shuran Song. Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https: //apxiv///rxv....// org/ aba/ 2304 14108
[9] Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, and J. Zico Kolter. Scaling laws for data filtering- data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22702-222 June 2024.
10

---

[10] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 20Xx:2000,03300, 2020.
[11] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute- optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
https://arxiv owgg/bs/////ooooooooo.ooo...
15556
[12] Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697-10707. PMLR, 2022.
[13] Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck. Chris Callison Burch, and Nicholas Carlini. Deduplicating training data makes language models better. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( /Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 1424-84445 Association for Computational I. Linguistics, 2022. doi: 10.18653/V1/20222.CC0 577. httpp:ttt///// http::::://oo..o/////// wrg/10 10. 18653/v1/2022. wttp:///.2222....1..o2. acw-long..5.555
[14] Hector Levesque, Ernest Davis, and Leora Morgenstemn. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
[15] Jeffrey Li, Alex Fang, Georgios Smymiss, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Erash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-Im; In search of the next generation of training sets for language models. arXiv preprint arXiv:2466. 1794, 2024.
[16] Aixin Liu, Bei Feng, Bing Xuc, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseck-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[17] Zhengzhong Liu, Bowen Tan, Hongyi Wang. Willie Neiswanger, Tianhua Tao, Haonan Li, Fajri Koto, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360 k2: Building a 65b 360-open-souree source large language model from scratch. arXiv preprint arXiv:2501. 10014444,......... 2025.
[18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint 1809.01809.02 2018
[19] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel Scaling dataa-onstraiiee language models. Advances in Neural Information Processing Systems, 36:50358-5033 2023.
[20] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint 16X1v:1606.06031,, 2016.
[21] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.011 2023.
[22] Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. CORR, abs/2406. 2024. doi: 00-48550/ARXX..2406.1 17557. URL httpss //doi.org/100 wtt..l...//./////......
48550/ http://wwvvv........ 17557
[23] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-66 66,, 2020. http:h/tpmmmmllll org arg/papers/v21/ 20-074. 074.7thtml
[24] P Rajpurkar. Squad: 100,000+ questions for machine comprehension text. arXiv preprint arXiv: 16066.0200.0 2016.
[25] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249 22249-266,777 2019.
[26] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series, 2011.
11

---

[27] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications the ACM, 640))99-106, 2021.
[28] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abuhakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating capabilities of language models. arXiv preprint arXiv:2206.04615. 2022.
[29] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:181..0933, 2018.
[30] Gemini Team, Rohan Anil, Sebastian Borgeaud, lean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et Gemini: a family highly capable multimodal models. arXiv preprint 1rXiv:2312. 18805 2023.
[31] Yury Tokpanov, Paolo Glorioso, Quentin Authony, and Beren Millidge. Zyda-2: trillion token high-quality dataset. arXiv preprint arXiv:241 2.002641.00068, 2024.
[32] Rowan Zellers, Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:/905.0000 2019.
[33] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint 33i::2304.06344 2023.
12

---

A Training Setup
We use OpenLM (MIT license) for training. Standard model configurations and hyperparameters can be found in the DCLM code base. OpenIM is used for all 1B and 3B models, as well as 7B models in Section 4 These models are trained on Nvidia H100s.
We use AXLeam for 7B and 12B models in Section These models are trained on TPUs.
The main datasets trained on are DCLM (CC-by-4.0), RefinedWeb (v2, from DCLM) (CC-by-4.0), and C4 (CC-by-4.0 or odc-by depending on source).
B
Evaluation
We evaluate the centered metric from DCLM, which consists of the following 22 tasks: AGI
Eval LSAT Analytical Reasoning [33 ARC Challenge [ ARC Easy [5]. BIG-bench es algorithms
28
BIG-hench dyck languages 28 BIG bench language identification 28). BIG-bench operators
28
BIG-bench: wikidata [28], BIG-bench repeat copy logic [28). BoolQ [4] Commonsense QA
29
COPA 226]. CoQA [25], HellaSwag (zero-shot and few-shot) 32 Jeopardy (MosaicML),
LAMBADA
20
OpenBook QA [8]] PIQA (2). SQuAD (24). Winograd Schema Challenge (14).
Winogrande
We also evaluate 7B and larger models on MMLU (few-shot) 10
C MMLU Repeatability
MMLU (few-shot)
DCLM 252B
0.52
DCLM 12.68 252B
RW v2 12.6B 252B
0.50
0.48
Epochs (Chinchilla Optimal)
Figure 7: Right side of Figure [ except measuring MMLU (few-shot) on the y-axis. We see that DCLM even with repeats significantly outperforms RefinedWeb on MMLU, even more when comparing against centered core metric.
D
Continual Pre-training
We train 7B models for 138B tokens to investigate the effect of curriculum learning on general pre-training data of different quality.
Dataset
Core
MMLU
Extended
Shuffle 90% RW and 10% DCLM
41.3
25.7
22.4
90% RW followed by 10% DCLM
41.0
26.2
21.9
Shuffle 80% RW and 20% DCLM
41.6
27.3
22.5
80% RW followed by 20% DCLM
41.6
26.1
22.2
13

---

E Deduplication Trends as Datasets Scale
DCLM-BASELIIN
1,0
Exact Dedup
Fuzzy Dedup
0.8
Removal Rate
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Documents Seen
Figure 8: In DCLM-baseline, the proportion of duplicates inside the dataset increases the number of documents increases. In Figure we see this trend also holds for RefinedWeb, but with lower removal rates.
RW Global Shard
Exact Dedup
Fuzzy Dedup
0.8
Removal Rate
0.6
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Documents Seen
1e9
Figure 9: RefinedWeb, the proportion of duplicates inside the dataset increases as the number of documents increases. Notice that these removal rates lower than those of DCLM-baseline
14

---

Fuzzy vs Exact Duplicates
Some initial 7B parameter (Chinchilla optimal) experiments explore the differences between fuzzy and exact duplicates. For the second pair of experiments, floor f refers to keeping documents that have least fuzzy copies, while ceil refers keeping no more than e copies. Additional work is required to better understand the differences between fuzzy and exact duplicates.
Dataset
Core
MMLU
Extended
DCLM-baseline duplicate-awand subsample (fuzzy)
34.1
25.1
17.5
DCLM-baseling duplicate-aware subsample (exact)
33.8
25.3
17.7
DCLM-baseline floor 21 ceil 4 (fuzzy)
42.7
37.4
25.6
M-baellslline floor 21 ceil 1, epochs (exact)
43.3
32.4
24.0
G DCLM-baseline Count Manipulation Metric Statisties
Number of Documents
Number of Documents
109
10a
x 10'
107
10"
20
40
60
80
100
10-1
109
Multiplicity
FastText Score
DCLM-baseline distribution of fuzzy du-(b) DCLM-baseline distribution of Fast plicates
scores
1e7
Number of Unique Documents
21
Averaged Multippicitty piiciiyyyy
20
19
2-
18
17
16
10-1
100
10-1
10"
Fast Score
FastText Score
(c) MCLM--bsellie unique documents by (d) DCLM-bascling average fuzzy duplicate FastText score
count by FastText score
15

---

H Deduplication Methods Details and Comparisons
We minhash-base based fuzzy deduplication written Rust. Our hyperparameters are 5-gram tokens and 14 bands of size 9.
Here compare fuzzy dedup results against the implementation from Tokpanov al. 31 (Zyda-2 DCLM-dedup), as well as DCLM-baseline (no additional global dedup).
Table 5: 7B Chinchilla optimal deduplication comparison
Dataset
Core
MMLU
DCLM Our Fuzzy Dedup
41.7
28.5
Zyda-2 DCLM-dedup
43.6
29.4
DCLM-baseling
44.4
42.2
Table 12B Chinchilla optimal deduplication comparison
Dataset
Core
MMLU
Zyda-2 DCLM-dedup
53.3
56.8
DCLM-baseline
55.0
58.3
I Deduplication Interaction with Repeats
In Section B] we study datasets as is, leaving the possibility of duplicates already existing in the datasets. Here compare repeating randomly sampled subset with repeating global fuzzy deduplicated (Zyda-2 DCIMM-eddup) subset. Note that randomly subsampling produces datasets where documents either have low repeats because the subset is much smaller than the pool that we subsample from, and the datasets studied in Section 3 already go through some sort of deduplication.
The results below suggest that low repeat counts favor DCLM-baseline, while high repeat counts favor the deduplicated version. At the compute regimes we study our findings should apply to both scenarios, but we hypothesize that deduplication or resampling becomes more important as we use higher repeat counts larger scales.
Epochs
DCLM Core
Dedup Core
DCLM MMLU
Dedup MMLU
epoch
55.0
53.3
58.3
56.8
2 epoch
54.0
53.5
58.6
57.9
5 epoch
53.2
51.9
56.9
55.5
10 epoch
49.0
49.5
52.9
51.4
16

---

Additional Count Manipulation Details
J.1
Count Manipulation Strategies
One way to determine the count function corresponding to each of the strategies in Figure 5 is as follows:
1. Greedy copy: Find the threshold such that the desired number documents after deduplication above the threshold. Then sample all documents probabilistically by keeping each document with probability duplicate_coont if the document's s score over the threshold.
Greedy copies: Find the threshold such that the desired number of documents after deduplication above the threshold. Then sample all documents probabilistically by looping 4 times per document over the threshold, with each trial attempting to keep that document with probability Zaspiiaate_cnn _ooont"
Linear up copies: First, determine the maximum number of copies (in this case max aax__copees 4) you would like to keep for the best document. Next, determine the goal number of document (goal_ gooll_doss) you would like to keep. The number of unique documents for each copy count is pool_docs bucket_size size xmaa=_capa " Using this, calculate the top mar__opics thresholds using the dedupli-
cated documents. Lastly, for each document (non-deduplicatee)) use the thresholds to identify which copy count bucket (i) it belongs to, and loop times, with each trial attempting to keep that document
with probability daplicate_conv comnt"
J.2 Ensembling
The ensembling strategy in Table is calculated by first finding each document's duplication count and DCLM FastText classifier score, then assigning each document its ranking (e.g. best score is 1, worst score number of documents) for each metric, and finally taking the maximum of those values. In this new metric, lower is now better. It is important for each metric used in the ensemble to not only be correlated with data quality, but also be fine-graing are-graiddd. Otherwise, in sable(dduplication count is no longer a useful metric because there are not enough unique documents.
J.3 Metric Comprison Between FastText Score and Ensemble
Table 7: Ensemble results from Table compared with using DCLM FastText score as metric.
Subsampling Method
MMLU CORE (22 task average)
DCLM-baseling uniform subsample
44.4
42.2
Greedy copy (dup count)
41.7
28.5
Greedy copies (ensemble)
44.6
38.5
Greedy copy (ensemble)
46.1
46.0
Linear up to 4 copies (ensemble)
46.0
44.2
Greedy copies (FastTexx)
41.6
45.3
Greedy copy (FastText)
45.5
45.4
Linear up to 4 copies (FastText)
42.1
46.7
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract includes our main findings, which are repeated and expanded upon in the introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include limitations section at the end of our main paper, well as discuss potential problems with data filtering.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experiment details (e.g- pseudocode, descriptions, hyperparameters) can be found throughout the paper and the appendix, and code needed to reproduce experiments are publicly available unless specifically omitted for now maintain anonymity.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The datasets we work with are available online, mainly through Common- Crawl's bucket and HuggingFace. One of the training frameworks used publicly available, while the other one will be available once anonymity is no longer necessary.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.cc/pulli/guidiee htapb//ii/gguidee//ooeieeeeeeeee
swwsssssicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details either specified in the paper, available through the publicly available training framework.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Experiments are large scale training runs, which means multiple runs to establish error bars is impractical.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We specify in the appendix the type of computer resources used for our training jobs; however, do not give specific computational costs total hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpppp//eaariig...cpppbb.t/ttta
Answer: [Yes]
Justification:
We have read the code of ethics and we believe that research follows the
code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include an impact section the end of the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release new data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the datasets and other assets used, and mention their licenses in the appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We train LLMs and then evaluate on them, but do not use as a tool in other parts of the research process.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hbtpap//hearipssccc//offrrrreeee for what should or should not be described.
24


Reviews:
[{"review_id": "tyCNOTvF4Y", "reviewer": "Reviewer_t4Kk", "summary": "This paper investigates the practical trade-offs between data quality and data quantity in the pre-training of large language models (LLMs), particularly in the context of scaling compute budgets where the availability of unique, high-quality data is becoming a constraint. The authors challenge the conventional wisdom of single-epoch training on unique data by exploring the efficacy of repeating smaller, more aggressively filtered datasets. The paper's core findings are twofold: \n+ Dataset-level Repetition: The authors demonstrate that repeating a high-quality, aggressively filtered dataset (DCLM-baseline) for up to ten epochs can yield superior model performance compared to training for a single epoch on its ten-times-larger, less-filtered superset (RefinedWeb). This surprising result is shown to hold across various compute scales, provided that the training recipe is adapted—specifically, by increasing weight decay to counteract the effects of repetition.\n+ Document-level Repetition: Moving beyond dataset-level epochs, the paper explores the idea that not all documents are created equal. It introduces \"count manipulation,\" a more granular approach to dataset construction where the number of times each individual document appears in the training set is explicitly controlled based on a quality metric (e.g., a classifier score or the document's original duplication frequency). Experiments show that intelligently upsampling high-quality documents and downsampling low-quality ones can create better datasets for a fixed token budget than standard uniform subsampling or simple deduplication.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n+ A Counter-intuitive and Foundational Insight on Dataset Repetition: The paper's most impactful contribution is the robust demonstration that repeating a smaller, aggressively filtered dataset for multiple epochs can outperform training for a single epoch on a significantly larger, but lower-quality, superset. This finding provides a clear, evidence-based strategy for navigating the increasingly critical issue of data scarcity and directly challenges the prevailing \"single-epoch\" training orthodoxy in the LLM community.\n\n+ A Novel, Granular Approach to Data Composition: The work moves beyond coarse, dataset-level repetition to explore the more nuanced concept of document-level repetition. By introducing \"count manipulation,\" the authors reframe dataset creation as a principled optimization problem, where the frequency of each document is explicitly controlled based on quality metrics. This represents a more powerful and flexible paradigm than simple filtering or uniform sampling.\n\n+ A Practical Roadmap for Data-Constrained Scaling: Together, these insights offer a practical roadmap for training high-performance models when the volume of unique, high-quality data is a limiting factor. The paper forces a critical re-evaluation of long-held assumptions and suggests that the community may have prematurely dismissed the utility of multi-epoch training, a standard practice in other deep learning domains.\n\n# Weaknesses\n+ Uncertain Extrapolation to Frontier-Scale Training Regimes: The primary weakness of the study lies in the scale of its experiments. While substantial for academic research, the models are trained on token budgets in the hundreds of billions. This is approximately an order of magnitude smaller than the over 30 trillion token datasets now commonly used to train state-of-the-art frontier models.\n\n+ The Question of Shifting Scaling Laws: The core uncertainty is whether the observed phenomena will extrapolate linearly to these much larger compute scales. It is plausible that scaling laws may shift, and the optimal trade-off between data quality, quantity, and repetition could be qualitatively different in a multi-trillion token regime. Therefore, it remains an important open question whether the benefits of heavily repeating a 1T token high-quality dataset would still hold against a single pass through a 15T token moderately-filtered corpus.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper.\n\nWe agree that our models do not scale to the same levels of data and compute as state-of-the-art frontier models. We also agree that a more extensive scaling analysis of repetition vs decreased-filtering is a useful future extension to this work.\n\nHowever, we also believe that, at a minimum, our existing results demonstrate that increased regularization and taking into account data filtering can support higher data repetition than was previously demonstrated (by Mueninghoff et al) at similar scales. We believe that these results are worth sharing more broadly with the community.\n\nWe would guess that in the particular case of DCLM it would perform poorly when repeated to 10T+ tokens because it contains less than 1T unique tokens. A more effective study at that scale would require new datasets, which may lead to slightly different behaviors and comparisons. We hope researchers continue looking into data filtering in relation to data repetition, as well as adjacent topics like improving token yield.\n\nWe would also be curious of the reviewer’s opinion on whether these things can be clarified with better scaling laws or improved metrics instead of training larger models with larger token budgets."}, {"review_id": "OIRwoexZTt", "reviewer": "Reviewer_fPkP", "summary": "This work aims to study how best to use heavily filtered and deduplicated datasets at larger training scales. The first half of the paper considers repetition at the dataset level, building on works such as (Muennighoff et al. [19]). The second half considers repetition at the individual document level with the goal of creating better datasets given a token budget. The authors end by reflecting on the scenarios where building better datasets is likely to have the most impact.", "questions": "1. In Table 3 of Section 4, duplication count is used as or as part of the quality metric. Can the authors explain the motivation behind this?\n\n2. Can the authors expand on the motivation behind the duplicate-aware sampling. The paper says this is done \"to mimic the duplicate profile of the original dataset.\" Is this intended mainly as a comparison to the count manipulation methods which also have a high number of duplicates but perform much better (presumably because the documents were chosen according to quality).\n\n3. In duplication-aware sampling, all or none of the duplicates are included probabilistically while for count manipulation the number of an duplicate is capped at some user chosen quality (the highest used in this paper is 5). Could this also be part of the difference between duplicate-aware subsampling and your count manipulation methods? \n\n4. Line 102-103 says the following:\n\n```\nIn Figure 2, we study a scenario similar to the findings in Muennighoff et al. [19], but instead start\nwith Chinchilla optimal unique tokens for 1B parameter models.\n```\n\nAre these tokens then excluded for the dateset subsets that are constructed for the rest of the experiments?", "limitations": "Yes, the authors have adequately discussed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-written and explores the interaction between data quality and repetition in a series of generally well-designed experiments. The results provide useful insights for practitioners when limited curated data becomes a constraint.\n\nThe first group of experiments in Section 3 mainly explore how the effects of repetition varies on 3 datasets (C4, a replication of RefinedWeb, and DCLM-baseline) across different scales:\n\n1. The first experiment (Figures 2 and 3) explores how repetition affects performance for each of the 3 datasets at the 1B scale. Each run starts with training the model to the Chinchilla optimal duration with unique tokens from the datasets. Then 3 different separate runs are done with different size pools of the data; for each run the perplexity on the C4 validation set is reported as well as the averaged performance on a set of downstream tasks. For DCLM, the same experiment was performed at the 3B scale. \n\n2. The second experiment (Figure 4) is similar except it fixes the total training budget across runs and then subsets the data to achieve the desired number of repetitions. This is done for different model sizes and durations on the DCLM-baseline data; each model size/duration also includes the baseline performance on one epoch of RW v2.\n\n3. The third experiment (Table 1) explores using weight decay to mitigate the performance degradation from repeating tokens. Here the experiments are fixed at model size 12.6B parameters, duration 252B tokens. The number of repeats and magnitude of the weight decay is then varied. Figure 1 then emphasizes a scenario where weight decay can be used to meaningfully improve performance at the high repeat scale.\n\nSection 4 then turns to how to \"create better datasets relative to a token budget by explicitly manipulating the counts of individual document,\" focusing on repeats at the document rather than dataset level. Two main techniques are explored: subsampling methods and count manipulation. Count manipulation fills the token budget with documents in descending order according to some quality metric and utilizes different rules for how many documents to keep at each level of quality. The authors emphasize that count manipulation becomes fundamentally different from filtering in that high quality documents can be upsampled so that they appear more times in the new dataset than they do in the original. The performance of these strategies is explored in Tables 2-4.\n\n**Weaknesses**\n\nSee the questions section for some concerns regarding Section 4 where I think the motivation behind some of the design choices could be made clearer.\n\nOverall, some of the figure legends and captions could be improved to make the experiments easier to parse for the reader. A few specific examples (primarily to aid in improving the paper in revisions):\n\nThe caption for Figure 4 says \"Legend denotes model size and multiplier to scale tokens seen at Chinchilla optimal.\" I presume this means that this is setting the total duration for each run in terms of a multiplier on the Chinchilla optimal duration, i.e. the duration for each run is the multiplier times 20x the number of parameters. However, because you are also changing the number of repeats this notation is not immediately clear. My suggestion would be to label the 6 colors with a more verbose description of the model size/duration settings and then have 2 more labels that explain that the solid and dashed lines correspond to the 2 datasets.\n\nOn the left hand side of Figure 1, since there are only 4 compute settings it would be useful to include the model size/training duration in the figure or caption. I assume from the $x$-axis label that the duration is the Chinchilla optimal for each model size, so just the model size could be sufficient. On the right hand side, it would help to clarify that 12.6B 252B refers to the model size and duration (later you have things like DCLM 230B where the number refers to the dataset size and not the duration). Also in Figures 1 and 2 you use the abbreviation RW v2 which isn't explained until page 4.\n\nOn the right hand side of Figure 1, it looks like \"DCLM 12.6B 252B\" is the WD 1x column from Table 1 and \"DCLM 12.6B 252B WD\" is the max performance for each row, which is mostly WD 2x. Some explanation of this would be helpful to make it clear why the curves match at 1 epoch. Also I presume that the largest compute scale in the left panel is the 12.6B parameter, 252B duration scale explored in the right panel. Using some marker to indicate that the 1 epoch and 10 epoch points have matching values on the left panel could be helpful given that the current shared colors are confusing (the 1 epoch point is coming from the purple line on the left).", "was_revised": true, "final_justification": "The paper is well-written and explores the interaction between data quality and repetition in a series of generally well-designed experiments. The results provide useful insights for practitioners when limited curated data becomes a constraint. In the rebuttal, the authors have updated the paper to better outline the flow of experiments and the clarity of the plots. I thus recommend acceptance.", "rebuttal": "Thank you for your suggestions for improving our paper.\n\nWe have updated our draft to take into account your suggestions regarding presentation.\n\nRegarding your questions:\n1. We started with deduplication count because our exploration in this section is motivated by the observation that deduplication can hurt dataset quality. This implies that data points are of different quality and may need to be weighted in a different manner. We then combine it with FastText score because it is considered a better quality indicator than deduplication count.\n2. Duplicate-aware sampling is most useful for running small scale experiments. Scaling up experiments can often assume new unique tokens of the same quality, but in works (e.g. in the DCLM paper) without an initial global deduplication, scaling back up is now expanding our dataset by adding additional repeats. Duplicate-aware sampling would be one way to design data pools for filtering experiments.\n3. Yes, this is why duplicate-aware subsampling performs worse. Another way to view this is that there are less unique documents after duplicate-aware subsampling when compared to naive subsampling.\n4. No, in figure 2, when we increase the token count (e.g. go from 28B to 56B unique tokens) we re-sample from the dataset, so there may be overlap between the 28B dataset and the 56B dataset. Here, “start” refers to the smallest dataset that we repeat with, and in contrast to Muennighoff et al. have much higher unique tokens per parameter."}, {"review_id": "FwcHtMORG8", "reviewer": "Reviewer_w2eL", "summary": "This paper tackles the tension between data quality and data quantity in training large language models under a fixed compute budget. The authors show that, once you’ve exhausted “fresh” high-quality data (DCLM), you can squeeze extra performance by repeating that subset—provided you adjust regularization (e.g., increase weight decay). They then push this further with a document-level count manipulation scheme: assigning repeat counts to each doc based on a quality score, so that the constraint of compute budget focuses only on better corpus.", "questions": "1. Could you provide a more detailed sensitivity analysis of weight decay across repeat counts?\n    \n2. You use FastText-based scores to rank documents. Have you tried other proxies (e.g., language-model perplexity, diversity metrics, QuRater)? Does the choice materially affect outcomes?\n    \n3. Document-level count manipulation introduces bookkeeping overhead. Can you quantify the additional preprocessing time and memory compared to uniform sampling?\n    \n4. A single “pipeline” figure showing: (a) scoring, (b) count assignment function, (c) repeat schedule, would greatly improve clarity. Consider adding this to the camera-ready.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper spans multiple model sizes, a wide range of tasks, and detailed ablations (perplexity, core metrics, MMLU), which makes the findings very robust.\n    \n- Beyond just “here’s a trick,” the authors provide hyperparameter settings, pseudo-code for count manipulation, and appendices with implementation details—bonus points for reproducibility.\n    \n- While prior work has touched on data filtering and sampling, the explicit pairing of multi-epoch repeats, plus the doc-level repeat allocation, limited-resource budget regimes.\n\n\nWeaknesses:\n\n- Key insights (e.g., that boosting weight decay is required to make repeats helpful) are buried. Figure 1’s default curves actually *mislead* unless you already know to jump to Table 1. There’s no high-level roadmap or a guiding “flow” diagram, so it reads like disjointed blog posts stitched together.\n    \n- The “why” behind count manipulation isn’t motivated with simple analogies or visuals. Readers unfamiliar with corpus building jargon may struggle to grasp why document-level repeats beat global strategies.\n    \n- Juggling four different model scales × three evaluation metrics × multiple sampling schemes makes it hard to see the forest for the trees. A more focused presentation (or a distilled summary table) would help.", "was_revised": true, "final_justification": "The authors have replied to my question and address my major concern. I have raise my score accordingly.", "rebuttal": "Thank you for your helpful comments towards improving our work.\n\nWe have improved our draft by adding an explicit roadmap at the end of the introduction, as well as additional transitions and clear lead sentences for paragraphs introducing new experiments. Additionally, we motivate count manipulation by connecting Figure 5a with standard deduplication, and expanding the caption.\n\nNew Figure 5 caption: Examples of strategies for count manipulation. Note that Greedy 1 copy is similar to global deduplication, and can result in worse performance if document quality varies significantly. For example, if the dataset is split evenly between high quality documents and low quality documents, it may be better to repeat the high quality documents rather than each document a single time.\n\nFor figure 1, would adding an arrow between the purple and pink lines labeled “improvement from weight decay” improve clarity?\n\nRegarding your questions:\n\n1. Table 1 contains results where we vary weight decay and repeat counts. Here is also an additional table that contains the same but for 3B models trained for chinchilla optimal number of tokens:\n| Repeats | WD 0.033 | WD 0.06 | WD 0.1 |\n| ----- | ----- | ----- | ----- |\n| 10 epochs | 0.334 | 0.332 | 0.347 |\n| 5 epochs |  ​​0.362 | 0.367 | 0.367 |\n| 2 epochs | 0.365 | 0.372 | 0.366 |\n| 1 epoch | 0.371 | 0.373 | 0.372 |\n \n2. The main scores we used were FastText and duplication count, and then ensembling the two. We find that FastText performs slightly better than duplication count alone, and ensembling the two provides some additional improvement. This ordering is similar to that of using these scores for standard data filtering, and we would expect some of the other proxies you’ve mentioned to be ordered similarly.\n3. The additional preprocessing time is a linear pass over the dataset, and the additional memory required is minimal. This is because we can pre-calculate thresholds, and probabilistically sample, as suggested in our pseudocode (Listing 1). Typically the metrics used to rank documents have already been calculated; in our case, FastText scores are calculated during the creation of DCLM, while counts are calculated during deduplication.\n4. Thank you for the suggestion."}, {"review_id": "Xmt6BKnAkJ", "reviewer": "Reviewer_Ta7J", "summary": "This paper investigates how to duplicate data samples under many different practical data settings. This is a timely topic since as LLMs scale, the compute need for data, as the chinchilla optimal rule suggested, also grows. The paper examines model performance across various compute budgets and pre-training datasets that have undergone filtering and deduplication.\n\nThe paper provides several findings, mostly are intuitive and are aligned with community beliefs. For example, higher quality datasets may be affordable to be repeated more times, and that not all documents are equal, and better datasets can be created by manipulating individual document counts considering document quality.", "questions": "- Given the paper's crucial observation that adjusting weight decay is key for effective data repetition, I am curious to understand the authors view if the findings align with, or diverge from, the theoretical insights presented in papers like [1].\n\n\n[1] https://arxiv.org/abs/2502.15938", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n\n## Practical considerations and scenarios\n\nIt is commendable the authors for thoroughly exploring a diverse range of distinct and practical data constraints and scenarios. Unlike many dataset papers that make a particular assumption of constraint and conduct experiments solely around that, which can sometimes limit practical applicability, this paper investigates scenarios such as:\n\n- Reproducing Muennighoff et al.'s work on diminishing return of repated data\n- Simulating scenarios where \"the dataset we are trying to manipulate is closer in size to the desired token count\"\n- \"fix the token budget and the number of unique tokens used for training\"\n\nThis broad practical scope significantly enhances the utility and relevance of the findings for real-world LLM training. These results will be valuable to different training goals, such as small models, large models, inference-optimal models, etc.\n\n## Practical findings and training methods\n\nThe paper finds that adjusting weight decay will help reduce the diminishing return of repetition, and also provides a practical data sampling method. The weight decay method remind me of [1], where the authors theoretically study the interplay of weight decay and data set size, by considering the model weights as a EMA of the weight updates. This finding here highlights that dataset study should not be considered simply as a \"preprocessing step\" before training.\n\nThe study offers actionable strategies for optimizing LLM training with constrained resources, whether due to limited unique data volume or specific compute budgets.\n\n# Weakness\n\n## Writing\nThe paper presents a list of distinct experimental settings and comparisons. However, the overall organization and presentation could be improved to make it easier for readers to find and consistently follow the flow of these numerous experiments, sometimes lacking clear initial roadmaps or explicit transitions between different scenarios.\n\n\n[1] https://arxiv.org/abs/2502.15938", "was_revised": true, "final_justification": "The writing of this paper is probably one of the biggest concern, as pointed out by two reviewers. \n\nWhile I am writing this final justification, I am still waiting the authors to provide more updates. As the state of this, even though the authors agree to update the paper. There are significant rewriting needed and I am not convinced that we will get a manuscript up to the standard of the venue. I am not leaning towards acceptance, and may even consider lower the scores.", "rebuttal": "Thank you for your helpful suggestions.\n\nAlthough PDF uploading is disabled, we have improved our draft by adding an explicit roadmap at the end of the introduction, as well as additional transitions and clear lead sentences for paragraphs introducing new experiments.\n\nRegarding your question, figure 4 of Bergsma et al. suggests that weight decay is more important as you increase tokens per parameter, but their data is deduplicated and is not repeated. Our paper explores weight decay in relation to data repetition, so perhaps these findings are orthogonal. One potential contradictory point is that with repetition in mind, weight decay does matter at chinchilla optimal, which they would consider in the lower TPP regime."}]

Author rebuttal:

Meta-review: The paper shows the aggressive data filtering and repetition can outperform training for an epoch with unfiltered larger dataset, when the training process, in particular, regularization, is adjusted. The authors also study document-level repetition.

The reviewers noted that the writing and presentation could be improved, and some key details, such as around regularization, should be more highlighted. The motivation behind some design choices could also be made clearer. Most of these concerns were addressed during the rebuttal. The reviewers also requested sensitivity analysis, which was provided during the rebuttal but only covers weight decay.

One of the primary concerns is that the experiments were conducted on token budgets in the hundreds of billions, an order of magnitude smaller than the datasets used for state-of-the-art frontier models. It remains unclear whether the observed benefits of data repetition will extrapolate to larger scales. However, the findings may still be relevant at smaller scale models used in academic research.

A review of older related work on data pruning, data difficulty is missing (e.g., Sorscher et al., Paul et al., Toneva et al., and many many others, some of which may not be applicable in the LLM training, but are readily extendible and deserve a discussion). Similarly, discussion of work on the effects of various direct and indirect regularization methods is missing.

Another concern of mine is that the natural baseline is missing: training for longer on unfiltered data with regularization, especially were the model scale is larger than the "optimal" data regime let's say, as suggested by scaling laws. From what I can tell, the experiments on the larger unfiltered data are with one epoch training.
Decision: Accept (poster)