Title: Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
Venue: NeurIPS 2025
Accepted: yes
Keywords: datasets

llm

repetition

deduplication

scaling

pre-training

language models

data filtering
Abstract: Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, as large language model compute budgets continue to grow, the limited data volume provided by heavily filtered and deduplicated datasets will become a practical constraint. In efforts to better understand how to proceed, we study model performance at various compute budgets and across multiple pre-training datasets created through data filtering and deduplication. We find that, given appropriate modifications to the training recipe, repeating existing aggressively filtered datasets for up to ten epochs can outperform training on the ten times larger superset for a single epoch across multiple compute budget orders of magnitude. While this finding relies on repeating the dataset for many epochs, we also investigate repeats within these datasets at the document level. We find that not all documents within a dataset are equal, and we can create better datasets relative to a token budget by explicitly manipulating the counts of individual documents. We conclude by arguing that even as large language models scale, data filtering remains an important direction of research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how to duplicate data samples under many different practical data settings. This is a timely topic since as LLMs scale, the compute need for data, as the chinchilla optimal rule suggested, also grows. The paper examines model performance across various compute budgets and pre-training datasets that have undergone filtering and deduplication.\n\nThe paper provides several findings, mostly are intuitive and are aligned with community beliefs. For example, higher quality datasets may be affordable to be repeated more times, and that not all documents are equal, and better datasets can be created by manipulating individual document counts considering document quality.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n## Practical considerations and scenarios\n\nIt is commendable the authors for thoroughly exploring a diverse range of distinct and practical data constraints and scenarios. Unlike many dataset papers that make a particular assumption of constraint and conduct experiments solely around that, which can sometimes limit practical applicability, this paper investigates scenarios such as:\n\n- Reproducing Muennighoff et al.'s work on diminishing return of repated data\n- Simulating scenarios where \"the dataset we are trying to manipulate is closer in size to the desired token count\"\n- \"fix the token budget and the number of unique tokens used for training\"\n\nThis broad practical scope significantly enhances the utility and relevance of the findings for real-world LLM training. These results will be valuable to different training goals, such as small models, large models, inference-optimal models, etc.\n\n## Practical findings and training methods\n\nThe paper finds that adjusting weight decay will help reduce the diminishing return of repetition, and also provides a practical data sampling method. The weight decay method remind me of [1], where the authors theoretically study the interplay of weight decay and data set size, by considering the model weights as a EMA of the weight updates. This finding here highlights that dataset study should not be considered simply as a \"preprocessing step\" before training.\n\nThe study offers actionable strategies for optimizing LLM training with constrained resources, whether due to limited unique data volume or specific compute budgets.\n\n# Weakness\n\n## Writing\nThe paper presents a list of distinct experimental settings and comparisons. However, the overall organization and presentation could be improved to make it easier for readers to find and consistently follow the flow of these numerous experiments, sometimes lacking clear initial roadmaps or explicit transitions between different scenarios.\n\n\n[1] https://arxiv.org/abs/2502.15938\n\nQUESTIONS:\n- Given the paper's crucial observation that adjusting weight decay is key for effective data repetition, I am curious to understand the authors view if the findings align with, or diverge from, the theoretical insights presented in papers like [1].\n\n\n[1] https://arxiv.org/abs/2502.15938\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the tension between data quality and data quantity in training large language models under a fixed compute budget. The authors show that, once you’ve exhausted “fresh” high-quality data (DCLM), you can squeeze extra performance by repeating that subset—provided you adjust regularization (e.g., increase weight decay). They then push this further with a document-level count manipulation scheme: assigning repeat counts to each doc based on a quality score, so that the constraint of compute budget focuses only on better corpus.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper spans multiple model sizes, a wide range of tasks, and detailed ablations (perplexity, core metrics, MMLU), which makes the findings very robust.\n    \n- Beyond just “here’s a trick,” the authors provide hyperparameter settings, pseudo-code for count manipulation, and appendices with implementation details—bonus points for reproducibility.\n    \n- While prior work has touched on data filtering and sampling, the explicit pairing of multi-epoch repeats, plus the doc-level repeat allocation, limited-resource budget regimes.\n\n\nWeaknesses:\n\n- Key insights (e.g., that boosting weight decay is required to make repeats helpful) are buried. Figure 1’s default curves actually *mislead* unless you already know to jump to Table 1. There’s no high-level roadmap or a guiding “flow” diagram, so it reads like disjointed blog posts stitched together.\n    \n- The “why” behind count manipulation isn’t motivated with simple analogies or visuals. Readers unfamiliar with corpus building jargon may struggle to grasp why document-level repeats beat global strategies.\n    \n- Juggling four different model scales × three evaluation metrics × multiple sampling schemes makes it hard to see the forest for the trees. A more focused presentation (or a distilled summary table) would help.\n\nQUESTIONS:\n1. Could you provide a more detailed sensitivity analysis of weight decay across repeat counts?\n    \n2. You use FastText-based scores to rank documents. Have you tried other proxies (e.g., language-model perplexity, diversity metrics, QuRater)? Does the choice materially affect outcomes?\n    \n3. Document-level count manipulation introduces bookkeeping overhead. Can you quantify the additional preprocessing time and memory compared to uniform sampling?\n    \n4. A single “pipeline” figure showing: (a) scoring, (b) count assignment function, (c) repeat schedule, would greatly improve clarity. Consider adding this to the camera-ready.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to study how best to use heavily filtered and deduplicated datasets at larger training scales. The first half of the paper considers repetition at the dataset level, building on works such as (Muennighoff et al. [19]). The second half considers repetition at the individual document level with the goal of creating better datasets given a token budget. The authors end by reflecting on the scenarios where building better datasets is likely to have the most impact.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper is well-written and explores the interaction between data quality and repetition in a series of generally well-designed experiments. The results provide useful insights for practitioners when limited curated data becomes a constraint.\n\nThe first group of experiments in Section 3 mainly explore how the effects of repetition varies on 3 datasets (C4, a replication of RefinedWeb, and DCLM-baseline) across different scales:\n\n1. The first experiment (Figures 2 and 3) explores how repetition affects performance for each of the 3 datasets at the 1B scale. Each run starts with training the model to the Chinchilla optimal duration with unique tokens from the datasets. Then 3 different separate runs are done with different size pools of the data; for each run the perplexity on the C4 validation set is reported as well as the averaged performance on a set of downstream tasks. For DCLM, the same experiment was performed at the 3B scale. \n\n2. The second experiment (Figure 4) is similar except it fixes the total training budget across runs and then subsets the data to achieve the desired number of repetitions. This is done for different model sizes and durations on the DCLM-baseline data; each model size/duration also includes the baseline performance on one epoch of RW v2.\n\n3. The third experiment (Table 1) explores using weight decay to mitigate the performance degradation from repeating tokens. Here the experiments are fixed at model size 12.6B parameters, duration 252B tokens. The number of repeats and magnitude of the weight decay is then varied. Figure 1 then emphasizes a scenario where weight decay can be used to meaningfully improve performance at the high repeat scale.\n\nSection 4 then turns to how to \"create better datasets relative to a token budget by explicitly manipulating the counts of individual document,\" focusing on repeats at the document rather than dataset level. Two main techniques are explored: subsampling methods and count manipulation. Count manipulation fills the token budget with documents in descending order according to some quality metric and utilizes different rules for how many documents to keep at each level of quality. The authors emphasize that count manipulation becomes fundamentally different from filtering in that high quality documents can be upsampled so that they appear more times in the new dataset than they do in the original. The performance of these strategies is explored in Tables 2-4.\n\n**Weaknesses**\n\nSee the questions section for some concerns regarding Section 4 where I think the motivation behind some of the design choices could be made clearer.\n\nOverall, some of the figure legends and captions could be improved to make the experiments easier to parse for the reader. A few specific examples (primarily to aid in improving the paper in revisions):\n\nThe caption for Figure 4 says \"Legend denotes model size and multiplier to scale tokens seen at Chinchilla optimal.\" I presume this means that this is setting the total duration for each run in terms of a multiplier on the Chinchilla optimal duration, i.e. the duration for each run is the multiplier times 20x the number of parameters. However, because you are also changing the number of repeats this notation is not immediately clear. My suggestion would be to label the 6 colors with a more verbose description of the model size/duration settings and then have 2 more labels that explain that the solid and dashed lines correspond to the 2 datasets.\n\nOn the left hand side of Figure 1, since there are only 4 compute settings it would be useful to include the model size/training duration in the figure or caption. I assume from the $x$-axis label that the duration is the Chinchilla optimal for each model size, so just the model size could be sufficient. On the right hand side, it would help to clarify that 12.6B 252B refers to the model size and duration (later you have things like DCLM 230B where the number refers to the dataset size and not the duration). Also in Figures 1 and 2 you use the abbreviation RW v2 which isn't explained until page 4.\n\nOn the right hand side of Figure 1, it looks like \"DCLM 12.6B 252B\" is the WD 1x column from Table 1 and \"DCLM 12.6B 252B WD\" is the max performance for each row, which is mostly WD 2x. Some explanation of this would be helpful to make it clear why the curves match at 1 epoch. Also I presume that the largest compute scale in the left panel is the 12.6B parameter, 252B duration scale explored in the right panel. Using some marker to indicate that the 1 epoch and 10 epoch points have matching values on the left panel could be helpful given that the current shared colors are confusing (the 1 epoch point is coming from the purple line on the left).\n\nQUESTIONS:\n1. In Table 3 of Section 4, duplication count is used as or as part of the quality metric. Can the authors explain the motivation behind this?\n\n2. Can the authors expand on the motivation behind the duplicate-aware sampling. The paper says this is done \"to mimic the duplicate profile of the original dataset.\" Is this intended mainly as a comparison to the count manipulation methods which also have a high number of duplicates but perform much better (presumably because the documents were chosen according to quality).\n\n3. In duplication-aware sampling, all or none of the duplicates are included probabilistically while for count manipulation the number of an duplicate is capped at some user chosen quality (the highest used in this paper is 5). Could this also be part of the difference between duplicate-aware subsampling and your count manipulation methods? \n\n4. Line 102-103 says the following:\n\n```\nIn Figure 2, we study a scenario similar to the findings in Muennighoff et al. [19], but instead start\nwith Chinchilla optimal unique tokens for 1B parameter models.\n```\n\nAre these tokens then excluded for the dateset subsets that are constructed for the rest of the experiments?\n\nLIMITATIONS:\nYes, the authors have adequately discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the practical trade-offs between data quality and data quantity in the pre-training of large language models (LLMs), particularly in the context of scaling compute budgets where the availability of unique, high-quality data is becoming a constraint. The authors challenge the conventional wisdom of single-epoch training on unique data by exploring the efficacy of repeating smaller, more aggressively filtered datasets. The paper's core findings are twofold: \n+ Dataset-level Repetition: The authors demonstrate that repeating a high-quality, aggressively filtered dataset (DCLM-baseline) for up to ten epochs can yield superior model performance compared to training for a single epoch on its ten-times-larger, less-filtered superset (RefinedWeb). This surprising result is shown to hold across various compute scales, provided that the training recipe is adapted—specifically, by increasing weight decay to counteract the effects of repetition.\n+ Document-level Repetition: Moving beyond dataset-level epochs, the paper explores the idea that not all documents are created equal. It introduces \"count manipulation,\" a more granular approach to dataset construction where the number of times each individual document appears in the training set is explicitly controlled based on a quality metric (e.g., a classifier score or the document's original duplication frequency). Experiments show that intelligently upsampling high-quality documents and downsampling low-quality ones can create better datasets for a fixed token budget than standard uniform subsampling or simple deduplication.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n+ A Counter-intuitive and Foundational Insight on Dataset Repetition: The paper's most impactful contribution is the robust demonstration that repeating a smaller, aggressively filtered dataset for multiple epochs can outperform training for a single epoch on a significantly larger, but lower-quality, superset. This finding provides a clear, evidence-based strategy for navigating the increasingly critical issue of data scarcity and directly challenges the prevailing \"single-epoch\" training orthodoxy in the LLM community.\n\n+ A Novel, Granular Approach to Data Composition: The work moves beyond coarse, dataset-level repetition to explore the more nuanced concept of document-level repetition. By introducing \"count manipulation,\" the authors reframe dataset creation as a principled optimization problem, where the frequency of each document is explicitly controlled based on quality metrics. This represents a more powerful and flexible paradigm than simple filtering or uniform sampling.\n\n+ A Practical Roadmap for Data-Constrained Scaling: Together, these insights offer a practical roadmap for training high-performance models when the volume of unique, high-quality data is a limiting factor. The paper forces a critical re-evaluation of long-held assumptions and suggests that the community may have prematurely dismissed the utility of multi-epoch training, a standard practice in other deep learning domains.\n\n# Weaknesses\n+ Uncertain Extrapolation to Frontier-Scale Training Regimes: The primary weakness of the study lies in the scale of its experiments. While substantial for academic research, the models are trained on token budgets in the hundreds of billions. This is approximately an order of magnitude smaller than the over 30 trillion token datasets now commonly used to train state-of-the-art frontier models.\n\n+ The Question of Shifting Scaling Laws: The core uncertainty is whether the observed phenomena will extrapolate linearly to these much larger compute scales. It is plausible that scaling laws may shift, and the optimal trade-off between data quality, quantity, and repetition could be qualitatively different in a multi-trillion token regime. Therefore, it remains an important open question whether the benefits of heavily repeating a 1T token high-quality dataset would still hold against a single pass through a 15T token moderately-filtered corpus.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}