Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Authors: Alexander Wettig
Kyle Lo
Sewon Min
Hannaneh Hajishirzi
Danqi Chen
Luca Soldaini
Venue: ICML 2025 poster (2025)
Keywords: pre-training
data-centric
data curation
language models
Primary area: deep_learning->large_language_models
TL;DR: We construct novel domains for unstructured web data and demonstrate how it leads to better data curation

Abstract: Modern language models are trained on large, unstructured datasets consisting of trillions of tokens and obtained by crawling the web. The unstructured nature makes it difficult to reason about their contents and develop systematic approaches to data curation. In this paper, we unpack monolithic web corpora by developing taxonomies of their contents and organizing them into domains. We introduce WebOrganizer, a framework for organizing web pages in terms of both their topic and format. Using these two complementary notions of domains, we automatically annotate pre-training data by distilling annotations from a large language model into efficient classifiers. This allows us to study how data from different domains should be mixed to improve models on downstream tasks, and we show that we can combine insights about effective topics and formats to further boost performance. We demonstrate that our domain mixing also improves existing methods that select data based on quality. Furthermore, we study and compare how quality-based methods will implicitly change the domain mixture. Overall, our work demonstrates that constructing and mixing domains provides a valuable complement to quality-based data curation methods, opening new avenues for effective and insightful pre-training data curation.

Organize the Web: Constructing Domains Enhances Pre- Pre-Training Data Curation
Alexander Wettig" 12
Kyle Lo
Sewon Min
Hannaneh Hajishirzi" 24
Danqi Chen' Luca Soldaini
Abstract
1. Introduction
Modern language models are trained large, unstructured datasets consisting of trillions of to- kens and obtained by crawling the web. The unstructured nature makes it difficult to reason about their contents and develop systematic ap- proaches data curation. In this paper, we pack monolithic web corpora by developing tax- onomies of their contents and organizing them into domains. We introduce WebOrganizer, a framework for organizing web pages in terms of both their topic and format. Using these two complementary notions of domains, we automat- ically annotate pre-tree vre-training data by distilling an- notations from large language model into effi- cient classifiers. This allows us study how data from different domains should be mixed to im- prove models on downstream tasks, and we show that we can combine insights about effective top- ies and formats to further boost performance. We demonstrate that our domain mixing also im- proves existing methods that select data based on quality. Furthermore, we study and compare how quality-based methods will implicitly change the domain mixture. Overall, our work demonstrates that constructing and mixing domains provides a valuable complement to quality- tion methods, opening new avenues for effective and insightful pre-training data curation.
Curating good training data is crucial for enhancing the ca- pabilities of language models. Early pre-training datasets, like the Pile (Gao et al., 2020) or RedPajama (Togeth- erAI, 2023), were created by curating data from multiple sources-sule as Wikipedia, Reddit, or BookCorpus (Zhu et al., 2015) giving rise to the research problem of how to balance these domains (Xie et al., 2023a). However, as the demand for data has grown trillions tokens, the major- ity of data is obtained from crawling the web, and the importance of curating domains has diminished.
Recent efforts in data curation, such as Fen (Penedo et al., 2024) and DCLM (Li et al., 2024), produce multi- trillion-token datasets with CommonCrawl as the singular source, offering no summary of their contents. In the ab- sence of domains, the focus has shifted to cleaning corpora using heuristic rules (Raffel al., 2020; Rae et al., 2021; Penedo et al., 2023) and quality filters (Wettig al., 2024; Sachdeva et al., 2024; Penedo et al., 2024; Li et al., 2024).
In this paper, we propose WebOrganizer, a framework to construct meaningful domains for monolithic web cor- pora. Our approach consists of designing taxonomies for unstructured web content, and scaling automatic labeling of documents according to these taxonomies by distill- ing large language model classifier (Llama-3.1-40SB Instruct) to small and efficient models (140M parameters). WebOrganizer establishes a rich, two-dimensional struc- ture for pre-training data by introducing two complemen- tary domain taxonomies-tople and format-which clas- sify web pages into 24 categories based on subject matter and style, respectively. This paper, for instance, would fall under the Science & Technology topic and the Academic Writing format. Figure 1 provides an overview of these do- mains and demonstrates how WebOrganizer shines a light on the composition of different types of internet content in cleaned pre-training corpus derived from Common- Crawl. We also compare our domains k-means clus- tering of document embeddings, and find that the clusters
data cura-
Website
weborganizer.co alwen ht co ceb/ebbrgggnieerrr CodeCreator//eeborgaal
Artifacts
Code
'Princeton Language and Intelligence, Princeton University Allen Institute for Artificial Intelligence 'University of Califor- nia, Berkeley *Paul G. Allen School of Computer Science & Engi- necring, University of Washington. Correspondence to: Alexan- der Wettig pr.petettot..tt..ddu.....
'Throughout the paper, we use the term domain denote dataset partitions, rather than conventional web domains, which we will refer to URL domains.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Audio Transcript
Academic Writing
Legal
History
Science Tech.
Fashion
Entertainment
Politics
Social Life
Structured
Support
Product Page
Data
& Beauty
Content Listing
Creative Writing
FAQ
Software Dev.
News (Org.)
About (Org.)
Docs
Food & Dining
Knowledge Article
Adult
Religion
Hardware
Sports & Fitness
News Article
Listicle
Home & Hobbies
Travel
Games
Tutorial
Spam Ads
Nonfiction Writing
Trunc.
Finance & Business
Comment Section
User About Review
Literature
Health
Art Design
Q&A
(Pers.)
Personal
Crime Law
Education & Jobs
Transp.
Software
Blog
Industrial
We construct topic domains (left) and format domains (right) to organize pre-training corpora. The areas visualize the number tokens per domain a cleaned pre-training corpus based on CommonCrawl. See Appendix A for detailed definitions of the categories. We provide an interactive explorer of the domains at weborganizer allen. ai.
Figure 1:
mostly align with topies and do not reveal different formats.
Finally, observe that data selection with quality filters implicitly changes the domain proportions of the dataset, and we quantify how much their performance gain can be accomplished from this domain mixing alone. We ob- serve that the Fine Web-Edu quality filter has similar do- main preferences to the mixtures optimized for MMLU and the implicit topic and format mixture retains up to 84% of the performance gains from quality filtering. Meanwhile, while the DCLM-fasttext quality filter (Li al., 2024) am- plifies certain formats, we find that its implicit data mix- tures perform considerably worse, suggesting that it utilizes aspects of quality beyond broad domain effects.
How effective are these domains for data curation? Par- titioning a corpus into domains provides rich affordances for data curation, as we can flexibly up or down-sample domains. More importantly, it enables principled meth- ods that can explore possible data mixtures in systematic ways and optimize the domain proportions meet the ob- jectives of data curation. We adapt the RegMix framework (Liu et al., 2024) to predict which domains should be up- sampled to improve two downstream tasks, MMLU and HellaSwag, which are commonly used for measuring the quality of pre-training data. For example, we find that up- sampling documents from the Science & Technology topic favors MMLU while the Tutorial format suits HellaSwag.
We open-source WebOrganizer as a tool for understanding. documenting and curating pre-traine pre-training data. To encourage future work, include the code for constructing domains and training domain classifiers, well as the annotated pre-training corpus.
Our experiments show that constructing domains and op- timizing their mixture towards specific tasks is effective. The reweighted topics, formats, and k-means clusters all improve downstream performance across a range of down- stream tasks. Furthermore, we show that, since the topic and format domains capture different aspects of web pages, can combine their data mixtures, which boosts per- formance considerably-matteee selecting documents with the FineWeb-Fdu quality filter (Penedo et al., 2024). Even more compelling, we find that our optimized domain mixtures work well with quality fil- ters and further enhance their performance. Notably, the average accuracy FibeWee-Edd increases from 54.2% to 56.2% when adding domain almost doubling the gain of Fine Web-Edu over 51. 1.6% baseline accuracy.
2. Constructing Domains for Web-Scale Data
State-of-the-art language models rely overwhelmingly on web-crawled training data (Baack, 2024; Raffel al., 2020; Brown et al., 2020; Rae et al., 2021: Penedo et al., 2023; Soldaini et al., 2024; Penedo et al., 2024; Li et al., with open-source research typically resorting to data provided by the CommonCraw/ foundation Unlike the large-scale ImageNet dataset (Deng et al., 2009), which was collected according to an explicit conceptual hierarchy,
the overall results of
commoncraw.. .org
2

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
these web corpora simply contain all web pages adhering to certain filtering rules, resulting in trillions of tokens of text without an inherent structure. While it is common practice to include specially curated domains, such as Wikipedia or StackOverflow (Touvron et al., 2023; TogetherAI, 2023; Soldaini et al., 2024), these additions are comparatively small and do not demystify the vast amount of data within CommonCrawl.
occurring topies into topic clusters-for example, our In- dustrial topic spans manufacturing, mining, agriculture, and utilities, mathematics is subsumed in Science & Tech- nology; in terms of formats, cooking recipes become part of Tutorials. We also adjust the categories to match the abil- ities of language models and what they can deduce from seeing only the URL and text contents of web page. For example, we observe that models are uncertain when choosing between comment sections and discussion fo- rums, motivating us merge these formats. In other in- stances, we add guidelines for resolving ambiguous cases. We eventually settle on 24 categories per taxonomy (see definitions and prompts in Appendix A).
Our practices of data curation are opaque and uninformed without a firm understanding of how these large-scale cor- pora are internally composed. In this paper, our approach is to design domain taxonomies to address this short-coming. We first lay out the desirable properties of such domains, and then we describe our method for creating laxonomies and annotating pre-training datasets, and finally compare our taxonomy-driven domains to a baseline that partitions a corpus via k-means clusters.
Our approach of proposing taxonomies in natural language and refining them based on model annotations is flexible and can be used for other purposes- for instance, to anno- tate corpus of scientific papers with detailed subject ar- eas, or to taxonomize the data within each of our domains to build a nested hierarchy. Unlike recent techniques for automatically constructing taxonomies with large- lagg-lan models (Chen et al., 2021a; Mishra et al., 2024; Pham et al., 2024), our approach requires human effort, but also bene- fits from human oversight and domain expertise.
Desiderata Since a corpus can be partitioned in exponen- tially many ways, we seek domains that produce human insights into pre-training corpora and our domains should align with meaningful human categories. To facilitate hu- man exploration, we also aim for compact number of do- mains that capture high-level trends and allow for con- cise representation of the corpus. Therefore, each domain should also have a reasonable amount of presence in the corpus. For example, URL. domains would be too granular, as there are 18.5k URL domain names with more than Ik documents in a 200B token subset of CommonCrawl and 14.7M URL domain names with fewer documents (see Fig- ure 5 in appendix). A smaller set of domains also decreases the chance of domain conflicts and ambiguities, and makes it easier to learn how to rebalance these domains.
2.2. Training domain classifiers for scaling annotations
While useful during taxonomy development, it would be extremely expensive to annotate web-scale corpus with large language model. Therefore, we enable WebOrga- nizer by fine-tuning small classifier model to imitate the an- notations of Llama-3. (lama-3.1-405B-B 1-40sB-Innttruct using a soft knowl- edge distillation loss (Hinton et al., 2015). We initialize the classifiers with gte-base-en-v1.5 (Li et al., 2023b)- 140M parameter embedding model with 8192 token context window-and train them in two stages improve their coverage over diverse web content. In the first stage, we train with 1M annotations from the cheaper Llama-3. 1- 8B- Instruct model, followed by 80K high-quality annota- tions from Llama-3.1 1-1005-B-nstruuc In Appendix B, we discuss the setup in more detail and show how the two- stage training is useful for improving the classifier accu- racy. We use the topic and format classifiers to anno- tate 200B pre-training corpus, which is based Com- monCrawl and cleaned using heuristic rules (Penedo et al., 2023) and deduplication (Soldaini et al., 2024).
2.1. Human-in-the-logg design of domain taxonomies
We design two domain taxonomies for WehOrganizer to capture the topic and format of web pages, respectively. These are meant to capture complementary characteristics: The topic should describe the subject matter of content, whereas the format concerns its style, intent and venue.
We start by reviewing existing fine-grained web tax- onomies, specifically the crowd-sourced curlic.org web directory, Google Adsense, the Wikipedia ontology, and the most frequent URL domains. We identify com- mon themes and propose coarse-grained topic and for- mat definitions, which we iteratively refine by prompting Llama-3.1-4055 -05BB- (Dubey et al., 2024) to classify CommonCrawl samples and reviewing these annotations.
2.3. Domain statistics
Figure I gives an overview of the topic and format domains provided by WebOrganizer and visualizes their proportions in the pre-training corpus. Figure 2 shows the highest val- ues of the normalized pointwise- mutual information be-
Following our desiderata, we consolidate less frequently
*This distinction has also heen made by van der Wees al.
(2015) in terms of topic and genre.
3

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
3. Optimizing Domain Mixtures for Downstream Tasks
-1.0
-0.5
0.0
0.5
1.0
Sofware Dev.
The promise of organizing a corpus with WebOrganizer is that we can learn the importance of each domain in a principled way. In this section, we study how to rebalance these domains to align with the needs of downstream tasks. This reflects the typical goal of data curation, which is to improve task performance when using a dataset for train- ing language models (Wettig et al., 2024; Penedo et al., 2024)-for instance, this is the protocol of the DataComps- LM competition (Li et al., 2024).
Literature
Adult
Sdence & Tech.
History
Finance & Business
Food & Dining
Sports& & Fitness
Home& & Hobbies
Software
Fashion Beauty
Creative Wmingg
Acadenic Wimling Knowldge Anicle Legal Notlees Tulbriaa/
Spam Ads
Q&A Forum
News Artile
Product Page
Sinuctured Data
Custoner Support User Review
Personal Blog
Documentation
Mixture prediction While many methods have been de- veloped optimize domain mixtures (Xie et al., 2023a; Chen et al., 2023; Albalak al., 2023; Fan et al., 2024; Chen et al., 2024; Jiang et al., 2024), most focus on min- imizing the in-distribution loss. We decide to use Reg- Mix (Liu et al., 2024) due to its simplicity and adapt it to optimize the mixture distribution for downstream tasks. For each set of domains-topic clusters-we train 512 models of 50M parameters for IB tokens and fit gradient-booster tree regression model (Ke et al., 2017). We make a mixture prediction by search- ing for the lowest loss in the input space of the regression model, restricting our search to mixtures which upsample domains at most 6,5x, which ensures that we do not ex- haust all documents when selecting training data in Sec- tion 4. We use an iterative search method, deviating from RegMix. Appendix C discusses our implementation in de- tail.
Figure 2: We visualize the 15 highest co-occurrences in the normalized pointwise mutual information (NPMI) ma- trix between topics (y-axis) and formats (x-axis). Figure 6 shows the full matrix, where most entries are close to zero.
formats, and k-mmaan
tween topic annotations T and formats F.
F) NPMI(T;F)) log p(T) / llog p(T, F))
where value of 0 suggests independence and implies complete co-occurrence. The majority of entries are close to zero with reasonable exceptions for pairs such as Doc- umentation and Software Development. The normalized mutual information measures the overall level of redun- 21(1: F)
dancy, NMI(T; F)= = H[T)+H(F) R 0.10, which close to zero and suggests that an independence assumption ap- proximates the domain product well.
Target tasks Whereas RegMix (Liu et al., 2024) uses the C4 loss as a proxy loss for task performance, we directly focus on two popular question-answering tasks, MMLU (Hendrycks al., 2021) and HellaSwag (Zellers et al., 2019), as well as their average. MMLU requires diverse world knowledge and problem solving abilities, whereas HellaSwag is an adversarially filtered dataset for common- sense reasoning. To avoid contamination, we use the train- ing and validation set of these two tasks, respectively. We seek a mixture that minimizes the next-token prediction loss over the correct response normalized by the response length (bits-per-byte) given a 5-shot prompt. This loss has also been used for extrapolating model task performance (Bhagia et al., 2024).
2.4. Comparison to k-means clustering
Clustering is a natural baseline for partitioning a corpus and has previously been used for training expert models (Guru- rangan et al.. 2023). We use the ste-base-e-vvl. model (Li et al., 2023b) to compute document embeddings for the 200B pre-training corpus and run k-means using dis- tributed implementation by Vo et al. (2024). We obtain 24 clusters that are more evenly balanced than our domains, but lack inherent natural language descriptions. Interest- ingly, we find that the k-means cluster assignments tend to reflect the web site topic more strongly than its format (since NMI(C; 7') & 0.46 vs. NMI F') 0.13, also see Figure 7 in the appendix). Furthermore, the trend is similar even with more fine-grained k-means domains of 576 clus- ters (NMI statistics remain within + 0.03). The orthogonal nature of the format domains suggests that careful human- in loop taxonomies can provide richer data annotations than clustering document embeddings alone.
Predicted mixtures Figure visualizes the training dis- tributions predicted by RegMix across the topic and format domains constructed by WebOrganizer. We observe that the two target tasks call for remarkably different data mix- tures. The MMLU mixture heavily upsamples Science & Technology, followed by History and Health, and in terms of formats, promotes Academic Writing and Q&A Forums.
4

---

Organize the Web: Constructing Domains Enhances Pre- Pre-Training Data Curation
Corpus
MMLU
HellaSwag
Both
Corpus
MMLU
HellaSwag
Both
Entertainment
Personal Blog
Politics
Product Page
Finance Business
News Article
Sports Fitness Health
Comment Section
Content Listing
Home Hobbies
Nonfiction Writng
Education & Jobs
Knowledge Article
Literature
Tutorial
Social Life
News (Org.)
Religion
Listicle
Science Tech.
Academic Writing
Food Dining
Audio Transcript
Travel
Spam Ads
Crime
Structured Data
Games
Creative Writing
Transportation
User Review
Software
About (Org.)
Art Design
About (Pers.)
Fashion & Beauty
Truncated
History
Q&A Forum
Software Dev.
Customer Support
Hardware
Legal Notices Documentation
Industrial
Adult
FAQ
0
10
20
0 10 20
10 20
10 20
10
20
10 20
0
10 20
0
10
20
% Mixture Weight
% Mixture Weight
Figure 3: The corpus proportions of our topic domains (left) and formats (right), and the training mixtures predicted by RegMix for targeting MMLU, HellaSwag, and both tasks. Numerical values can be found in Table 9 in the appendix.
HellaSwag exhibits smoother mixtures, notably amplify- ing Home Hobbies and Fashion & Beauty, and strongly boosting Tutorials. Meanwhile, the mixtures tailored to- wards the average of both tasks tend to combine the promi- nent components of each task mixture. We provide pre- dicted mixtures for additional downstream tasks, includ- ing HumanEval (Chen et al., 2021b) and Natural Ques- tions (Kwiatkowski al., 2019) in Appendix D.
shot setting on a suite tasks: MMLU (Hendrycks al., 2021), HellaSwag (HSwag) (Zellers al., 2019), PIQA (Bisk al., 2020), WinoGrande (WinoG) (Sakaguchi al., 2021), CommonSenseQA (CSQA) (Talmor al., 2019),
Social IQa (SIQA) (Sap et al., 2019), ARC-easy/challenge (ARC-e/ARC-e) (Clark et al., 2018), and OpenBookQA (OBQA) (Mihaylov et al., 2018). OLMES measures task performance in both the multiple-choice format and a cloze formulation, well as curating few-shot examples, pro- ducing more reliable evaluation for smaller models.
4. Evaluating Pre-Training Data Curation with WebOrganizer
4.2. Topic Format selection
We demonstrate the practical value constructing do- mains with WebOrganizer by training models with the do- main mixtures produced by RegMix in Section 3. We show how to combine data mixing for topies and formats, and how domains can be used together with quality filters.
We construct new taxonomy, consisting of all 576 pairs of topic and format domains. Finding training mixture of this cardinality would be expensive and sensitive to noise. Here, make the assumption that we select topics and formats independently, and select data according to
4.1. Experimental Setting
(topie format) (topiic Pr
All our experiments are implemented in the DataComps- LM (DCLM) framework (Li et al., 2024), using the 1b-1x competition pool. We follow best practices and use heuris- tie filters, followed by deduplication to reduce the 1.6T raw token pool to base corpus of 200B tokens. From this dataset, we select 29B tokens by sampling according domain mixture and train IB parameter model. Full de- tails of our experimental setup can be found in Appendix E.
where Pr (topic) and Ps (f the predictions from separate RegMix pipelines. In practice, there are cases where Pr(toiic) (format) can exceed the amount of data available for that pair. In such cases, we take all available documents and up-sample everything else to compensate.
4.3. Combining quality filters and domain mixing
Quality filters assign scores to individual documents (Xie et al., 2023b; Wettig et al., 2024; Sachdeva et al., 2024), and select data more granular level than possible
Evaluation suite We use OLMES (Gu et al., 2024) to evaluate models and their domain mixtures. We use
5

---

Organize the Web: Constructing Domains Enhances Pre- Pre-Training Data Curation
Table 1: Evaluating the benefits of domain mixing, where the domain mixtures are tailored towards MMLU and HellaSwag (Figure 3). All models are trained in the 1b-1x setting from DCLM (Li et al., 2024) The baseline corpus is pre pre-processed with heuristic filtering and deduplication and forms the basis for the other data curation methods.
Data Curation
MMLU
HSwag
PIQA
WinoG CSQA
SIQA
ARC-e
ARC-c
OBQA
Avg
Baseline corpus
30.3
57.5
71.3
56.1
59.0
49.9
62.2
34.0
44.0
51,6
+ Clusters
31.8
59.4
73.4
58.2
58.7
50.7
66.1
35.2
44.8
53.2
+ Topic
31.4
56.2
72.1
54.8
61.3
47.8
70.3
40.6
49.0
53.7
+ Format
31.7
60.9
74.1
56.9
60.1
47.4
65.8
35.9
47.6
53.4
+ +Topic x Format
32.7
60.1
73.4
56.5
62.3
49.3
69.7
38.8
49.0
54.6
12.4
12.6
12.1
10.4
13.3
50.6
17.5
14.8
15.0
13.0
FineWeh-Fdu
34.3
56.0
69.9
57.7
60.0
47.9
71.9
42.3
48.2
54.2
Topic X Format
34.2
62.5
73.3
57.1
63.0
49.4
72.2
43.3
50.8
56.2
40.1
16.5
13.4
40.6
13.0
10.3
11.0
12.6
12.0
DCLM-fasttex
33.4
59.0
70.5
58.8
63.2
50.7
71.4
39.8
48.8
55.1
Topic Format
33.8
63.1
74.3
57.6
62.7
49.8
73.4
42.2
47.8
56.1
10.4
14.1
73.8
01.2
40.5
12.4
41.0
with domain rebalancing. Therefore, they are a powerful baseline. We compare to two state-of-the-ar quality fil- ters: FineWeh-Edu (Penedo et al., 2024), 110M param- eter model distilled from prompting Llama-3-700 to rate the educational value of web pages, and DCLMM-fatttxt (Li et al., 2024), bigram model trained to identify text resembling reference corpus consisting mostly of GPT-4 conversations. For both methods, we select all the highest- ranking documents until the token budget is reached.
target task accuracy, the topic mixture produces the best overall accuracy with 2.1% absolute gain over the ran- dom sampling baseline, with excellent transfer to ARC- easylchallenge and OpenBookQA. We note that reweight- ing k-means clusters performs well with an overall 1.6% point improvement, and Table 10 shows that they the most well-suited for targeting only HellaSwag.
Topic and format mixtures can be combined Our do- mains offer the advantage that topic and format mixtures can be combined. Our experiment (row 5) demonstrates that this is effective, improving performance in 8 out of 9 tasks and achieving a 3.0% absolute gain overall, which narrowly beats the FineWeb-B quality classifier. It also consistently improves performance when only aiming for one of the two downstream tasks in Table 10, notably at- taining an MMLU score of 33.2%. This illustrates that both topic and format are important axes for data curation.
We explore a simple strategy for composing quality filters and domain mixtures: We use the domain mixture de- termine the desired number of tokens from each domain subset. Then we perform the data selection with the qual- varying the
ity filter separately for each mubset-effecting quality threshold per domain, depending on the mixture.
4.4. Results
Table shows the results of our main experiments with mixtures optimized for both MMLU and HellaSwag. In the first setting, we consider how domain mixing improves upon the inherent data mixture of the baseline corpus. Then, we show how domain mixing also improves the per- formance of quality filtering. Results for individual task mixtures reported in the appendix (Table 10).
Domain mixtures improve quality filters Finally, show that our domain mixtures can also boost the overall performance of two state-of-the-a quality filters, improv- ing the average performance of Fine Feb-Eeu and DCLM- fasttext by 2.0% and 1.0%, points respectively (rows 6- 9). We highlight that our domain mixing addresses the weaknesses of the quality classifiers-foo instance on Hel- Saag Fine
Domain mixing is broadly effective We observe that reweighting the domain proportions of the pre-training pus improves downstream performance across all three of the topic, format, and k-means cluster domains (rows 1-4 in Table 1). Rebalancing formats achieves the best trade- off between MMLU and HellaSwag, and improves per- formance on out of the transfer tasks. Despite the
underperforms the random sam- pling baseline by 1.5% points, which tailored mix- ture converts to 5% absolute gain. This reflects the fact our domain mixtures can be subtly calibrated to meet the demands of the downstream tasks, whereas would hard to encode exactly the right preference for certain sub-distributions by changing the prompt for the Fine Web-
6

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Edu classifier or the reference corpus for the DCLM- fasttext classifier. With the same domain reweighting, FineWeb-Edu and DCLM-fasttext achieve similar perfor- mance across tasks.
Table 2:
Approximating quality filters by their implicit mixtures over topics and formats. Numbers in parenthe- ses show how much of the gain of the quality classifier is achieved by mixing alone.
5. Quality Filters as Implicit Domain Mixers
Data Curation
PPL.
MMLU
Task Avg
In Section 4, we combine domain mixing and quality filter- ing by using the mixture to specify how many tokens to se- leet per subset. Without an explicit domain mixture, a qual- ity filter will naturally upsample certain domains, which is equivalent to applying an implicit domain mixture and sub- sequently selecting the top documents within each domain. This process offers insights on two quality classifiers con- sidered in this work, and presents a richer way to describe differences between them.
Baseline corpus
12.1
30.3
51.6
FineWeb-Edu
14.7
34.3
54.2
as Topic
12.6
32.5 (55%)
52.4 (29%)
as Format
12.3
32.8 (63%)
52.5 (33%)
as Topic x Format
12.9
33.2 (73%)
53.8 (84%)
fastefa
14.0
33.4
55.1
as Topic as Format
12.2
31.5 (41%)
51.8 (9%)
12.2
31.4 (35%)
52.0 (16%)
We reconstruct the implicit domain mixture by computing the domain statistics of the quality filtered training datasets. Figure 4 visualizes these distributions for Fine Web-Edu and fasttxa.. We observe that FineWeb-BB Web-Edu devi- ates more strongly from the corpus than DCLM-fastter in terms of topics, while faLM-fasttext amplifies a larger number of categories in terms of formats. Their mixtures also share notable similarities with the RegMix predictions in Figure 3-all amplifying Politics, Health, Science & Tech., and History, to varying degrees, as well as Knowl- edge Articles, Tutorials, Academic Writing, and Q&A Fo- rums. However, the exact proportions differ substantially and their behaviors also diverge. For example, DCLM- fasttext retains by far the most documents from Entertain- ment and Games topics, as well as from Comment Sections and Creative Writing formats.
as Topic x Format
12.5
32.0 (56%)
52.8 (35%)
Nature of data quality It has become common to claim that datasets which produce better benchmark scores have "higher quality" (Li et al., 2023a; Wettig et al., 2024; Sachdeva et al., 2024; Penedo et al., 2024; Li et al., 2024). In domain mixing, the "quality" of domain is reflected by how much it should be upsampled, but our findings in Section 3 suggest that MMLU and HellaSwag exhibit very different domain preferences, and optimizing for both tasks requires making trade-offs. This highlights how "data qual- ity" is sensitive to the choice of downstream tasks, and we observe that the notion of "quality" by FineWWebEER Edub-Edu is particularly biased to specific domains that benefit down- stream tasks. However, there are many aspects of web con- tent that are not captured by WebOrganizer, e.g., the preva- lence of misspellings or factual errors, and these might be better modeled by scoring individual documents. Such ef- fects may explain why mated by domain effects, and why both quality filters sub- stantially outperform random sampling when imposing the same Topic x Format mixture (Table 2).
Approximating quality filters by domains We adopt only the implicit domain mixtures of quality classifiers for pre-training data curation, replacing the "local" selection within each domain with random sampling. The results of training IB parameter models are shown in Table 2. Both topic and format domains help to approximate the perfor- mance of quality classifiers. Of the two quality classifiers under study, we find FineWeb-Edu to be better approxi- mated by domain effects. In this case, implicit Topic x Format mixture recovers its performance gains by 73% on MMLU and 84% on average. However, a substantial gap remains for approximating DCLM OCLteffattext, suggesting that this classifier relies more on selecting the "right" docu- ments within each domain.
is not well approxi-
6. Related Work
Data selection Many methods have been developed se- lecting pre-training data for training large language mod- els. It has become common practice to remove noisy web sites using heuristic filtering rules (Raffel al., 2020; Rae et al., 2021; Penedo et al., 2023), focusing on surface statis- tics such mean word length word repetitions. This is typically followed by deduplication (Lee et al., 2022; Jiang et al., 2023; Abbas et al., 2023; Tirumala et al., 2023; Sol- daini et al., 2024). Additional data selection techniques include measuring n-gram similarity to high-quality refer- ence corpora (Brown al., 2020; Xie et al., 2023b; Li et al.,
Finally, we report the held-out perplexity of the models and observe that the values for domain mixing and substantially lower than using the quality filtering and close to the base- line corpus. This suggests that document-level quality fil- tering is a far stronger intervention on the pre-training dis- tribution than rebalancing domains or topics.
7

---

Organize the Web: Constructing Domains Enhances Pre- Pre-Training Data Curation
Corpus
RegMix
FineWebedu DCLMastter
Corpus
RegMix
FineWebedu
DCLM PCaMiesttext
Entertainment
Personal Blog
Politics
Product Page
Finance Business
News Article
Sports Fitness Health
Comment Section
Content Listing
Home Hobbies
Nonfiction Writing
Education & Jobs
Knowledge Article
Literature
Tutorial
Social Life
News (Org.)
Religion
Listicle
Science Tech.
Academic Writing
Food Dining
Audio Transcript
Travel
Spam / Ads
Crime
Structured Data
Games
Creative Writing
Transportation
User Review
Software
About (Org.)
Art & Design
About (Pers.)
Fashion & Beauty
Truncated
History
Q&A Forum
Software Dev. Hardware
Customer Support
Legal Notices
Industrial
Documentation
Adult
FAQ
0
10
20
0
10
20
0
10 20
10 20
10
20
10 20
0
10
20
0
10
20
% Mixture Weight
% Mixture Weight
Figure 4: The implicit domain compositions from quality filtering compared to the corpus distribution for topic domains (left) and format domains (right). We include the RegMix prediction tailored to both MMLU and HellaSwag from Figure to facilitate comparison. Numerical values can be found Table in the appendix.
2024; Brandfonbrener et al., 2024), using perplexity of ex- isting language models (Wenzek et al., 2020; Muennighoff et al., 2023; Marion et al., 2023; Ankner et al., 2025), and prompting large language models to rate documents based on qualities such as factuality or educational value (Gu- nasekar et al., 2023; Wettig et al., 2024; Sachdeva et al., 2024: Penedo et al., 2024). An alternative approach data curation focuses generating synthetic training data from large language models (Gunasckar et al., 2023; Li et al., 2023a), which may involve designing an explicit taxonomy of skills and concepts (Ding et al., 2023; Ben Allal et al., 2024) or re-writing existing data (Maini et al., 2024) the latter would reflect the original domain proportions and our approach could be used to improve the data mixture.
into 13 topics, and combine this with FineWeb-EEu quality buckets and SlimPajama sources, resulting in large set of composite domains. Rather than learning the relationship between these domain weights and downstream tasks with RegMix, they propose a selection strategy that samples from the domains based on gradient influence scores. Simi- larly, (Zhang et al., 2025) define fine- mne-grained k-means clus- ters (k=10,000) with the motivation to increase diversity during influence-based data selection. SemDeDup (Abbas et al., 2023) also utilizes large set of k-means clusters to define prototypicality scores and diversify the pre-traie pre-training distribution. While all these works have the shared goal of improving data quality, our work also contributes a two- dimensional structuree for organizing web data and pro- vides comparative study of fine-grained quality selection and coarse coarse-grained domain mixing.
Data curation with domains Several language models add specially curated domains to their pre-training data (Touvron et al., 2023; Soldaini et al., 2024; OLMo et al., 2024), but CommonCraw/ data forms typically the majority of data and has also been shown to outperform domain cu- ration (Penedo et al., 2023; Li et al., 2024). Several works have investigated the specific impact of varying portion of code in the pre-training data (Ma al., 2024: Petty et al., 2024; Aryabumi al., 2025; Chen al., 2025). Dubey et al. (2024) briefly mention using knowledge clas- sifiers to downsample "over-represented" data for training Llama-3. Instead of using domains for rebalancing data, Gao et al. (2025) observe performance improvements when conditioning on domain metadata during pre-training. Sim- ilar to our work, (Bai et al., 2024) propose to classify data
Data mixture optimization Many techniques have been developed for tuning domains proportions while training language models. These seek to minimize the validation losses across the domains (Xie et al., 2023a; Albalak et al., 2023; Jiang et al., 2024; Chen al., 2024), although some methods apply out-of-domain settings (Chen al., 2023; Fan et al., 2024). Most methods adjust mixtures dynam- ically during training, and some make predictions from many static mixtures (Liu et al., 2024; Ye et al., 2024; Kang et al., 2024). In concurrent work, Held et al. (2025) use large language model to predict the utility of subsets to downstream tasks. Due to the lack of meaningful do- mains, CommonCrawl is partitioned into "head" and "tail"
pro-
8

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
domains based on perplexity scores. Thrush et al. (2025) partition web corpus into almost 10k domains based frequent URL domains and rank them based on correlations between domain perplexities and benchmark scores from 90 existing open language models. Hayase et al. (2024) ex- tracts the data mixture of a private pre-training corpus from tokenization rules. Instead of optimizing domain mixtures, researchers have also developed approximations for the im- pact of individual training examples on loss of validation set (Engstrom et al., 2024; Yu et al., 2024; Wang et al., 2024). However, (Zhang et al., 2025) demonstrate that par- titioning the dataset into domains (in their work, k-means clusters) is beneficial for increasing data diversity when se- lecting data with gradient- based influence approximations.
two high-level domain taxonomies. This enables analyz- ing the internal composition of web-crawled pre-training corpora (as in Figure 1) and examining how it changes af- ter quality filtering (in Figure 4). There is wide scope for refining these data representations in future work, includ- ing hierarchical taxonomies, e.g., breaking down Science & Technology into the various scientific disciplines; or multi- label classification which could better account for ambigu- ous cases where a document covers multiple topics does not fit cleanly to any one label.
Impact Statement
Our work advances data curation for language models, and thus carries the broader societal implications associated with improving the capabilities of these models. By taxon- omizing web data, we develop tool that aims to enhance the transparency of pre-training corpora- corpora-potentialy potentially help- ing both researchers and the broader public develop a better grasp of the available pre-training data for language mod- cls. At the same time, we acknowledge the risks inherent in this process: Reducing the rich diversity of online content to a limited set of discrete domains can obscure important phenomena and may lead to errors, biases, or misrepresen- tations. We highlight that there are many valid ways to de- fine web taxonomies, and our efforts do not represent a def- inite "ground truth". Similarly, the predictions of how to re- balance the domains are sensitive to noise, as they are based on relatively few small model runs. Furthermore, is uncer- tain how well they transfer across model scales. Despite these challenges, in the absence of other meaningful meta- data, we believe that our domain annotations contribute to more informed understanding of web-scale training data.
Analysis of pre-training data WebOrganizer can serve as a tool for analyzing the contents of web corpora and the effects of quality filtering. In related work, Long- pre et al. (2024b) study data curation in terms of toxicity, source composition, and dataset age, and Longpre et al. (2024a) analyze licensing issues in web corpora. Elazar et al. (2024) provide a scalable tool for searching web-scale corpora and study the prevalence of toxicity, duplicates, and personally identifiable information. Lucy et al. (2024) use self-descriptions of website creators to measure how quality filters amplify and suppress speech across topics, regions, and occupations. Ruis et al. (2024) employ influ- ence functions to find pre-training documents important for learning factual knowledge and mathematical reasoning re- spectively. In a separate line of work, large language mod- els have been used for clustering large corpora (Wang et al., 2023; Zhang et al., 2023; Pham al., 2024) and describing clusters post-hoc (Zhong et al., 2022; Tamkin et al., 2024).
Acknowledgments
7. Conclusions
We thank Pang Wei Koh, Tyler Murray, and Mayee Chen for helpful discussion. We also thank Mengzhou Xia, Dan Friedman, Tianyu Gao, Alex Fang. Maria Antoniak, Ben Lee, and Catherine Chen for feedback on the draft. This research is partially funded by the National Science Foun- dation (11-2211779).
We introduce WebOrganizer-aa tool for organizing un- structured web corpora into topic and format domains. By annotating a 200B token pre-training corpus, we demon- strate how WebOrganizer documents the internal contents of the pre-training data, and that can re-balance these subsets to increase the performance of downstream tasks. Importantly, we show that topic and format selection can be combined, and that domain mixing can be integrated with quality filtering, which combines the benefits of document- level selection with well calibrated domain ratios.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Gan- guli, and Ari S Morcos. SemDeDup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.095.. 2023.
Increasing the transparency of data curation is an inter- esting avenue for future work. Better documentation of pre-training the data can inform model developers about potential strengths and weaknesses of the model and also improve the understanding of other stakeholders such as policy makers or end users. In this work, we make initial progress in this direction by introducing WebOrganizer and
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Models, 2023.
Thomas Wolf, and Mathhis Gallé (eds.), Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, virtual+Dublin, May 2022. Association for Computational Linguisties. doi: 10.18653/v1/2222 bigscience- 1.9. URL org/ wig.oi22.isgiccccee........ 1.9/.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Per- plexed by perplexity: Perplexity-hasee data pruning with small reference models. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL
hwwp///ppnreeeeeeeeeee neetiieeweee//eeeeeeeee fese/ffore forum? id=1GTARJhxtq
David Brandfonbrener, Hanlin Zhang. Andreas Kirsch, Jonathan Richard Schwarz, and Sham M. Kakade. Color-filter: Conditional loss reduction filtering for tar- geted language model pre-training. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https::/t////ett /ttps////epeeeeeeeeeeeome net./ forum?1d==GUccmMMER
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzich Fadace, Ahmet Üstün, and Sara Hooker. To code not code? exploring impact of code in pre-training. In The Thir- teenth International Conference on Learning Represen- tations, 2025. URL https:/ ptctpp:://ppereeee /openreview.w... net/ forum? d= feNluAcx.
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complex- ity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21-29. IEEE, 1997.
Stefan Baack. A critical analysis of the largest source for generative ai training data: Common crawl. In Proceed- ings of the 2024 ACM Conference on Fairness, Account- ability, and Transparency, FAceT '24, pp. 2199-2208, New York, NY, USA, 2024. Association for Comput- ing Machinery. ISBN 9798400704505.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpt: /ttp:os::///rrrceeiings/s neurips. cc / pappaaperpppppm files 1 055Tcc66bfc244077111fff0aa222223 pdf.
doi: 10.1145/
3630106.3659033. URL https:// /ttp:///oo.o.o/...////// 1145/3630106 3659033.
Tianyi Bai, Ling Yang. Zhen Hao Wong, Jiahui Peng, Xin- lin Zhuang. Chi Zhang, Lijun Wu, Jiantao Qiu, Wentao Zhang, Binhang Yuan, al. Multi-agent collaborative data selection for efficient Ilm pretraining. arXiv preprint arXiv:2410.08102. 2024.
Loubna Ben Allal, Anton Lozhkov Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm- corpus, 2024. URL https: / http://hhggggggac..e.e / datasets s Htt.g/ggggggggee////g/eee http://ggaaaeee//mmmmmmmm-oom- comp://oommmmmmmmss......
Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A Smith, Dirk Groeneveld, Pang Wei Koh, et al. Establishing task scaling laws via compute-efficient model ladders. arXiv preprint arXiv:2412.04403, 2024.
Catherine Chen, Kevin Lin, and Dan Klein. Construct- ing taxonomies from pretrained language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4687- 4700, Online, June 2021a. Association for Computa- tional Linguistics. doi: 10. 1 373. URL https: hatp:/:///aaaaatthoo/ naac
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pp. 7432- 7439, 2020.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang. Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Wein- bach. GPT-NeoX-208 200B: An open-source autoregres- sive language model. In Angela Fan, Suzana Ilie,
373//.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2 03374, 2021b.
10

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Re. Skill-it! a data-driven skills framework for understand- ing and training language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: /ttps//////eeeiww./////mm net forum? id=IcizwOINLE.
al. The Llama 3 herd of models. arXiv preprint 24X72:240..11783, 2024.
Yanai Elazar, Akshita Bhagia, lan Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Conference on Learning Representations, 2024. URL https: opwnreviee.e..eeooo wwwperevieeeeeeeeeeeeeeeeeeeeeee met//oruu
Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. Aioli: A unified optimization framework for language model data mix- ing. arXiv preprint arXiv:2411.05735. 2024.
id=RvfPnOXPV4.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamod- els. In Forty-first International Conference on Machine Learning, 2024. URL https: ftttps://operreeeee /pppreveeeeeeeeeeee net/forue BCB-GGBKK.........
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang. and Heng Ji. Scaling laws for predicting downstream performance in LLMs, 2025. URL https: htt////ppnnreewereeeeeeee wwwpreeiiiwweweeeeeeeeeeeeeeeeee wwt.forunt id=BDi sxnHzRL.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. DOGE: Domain reweighting with generalization esti- mation. In Ruslan Salakhutdinov, Zico Kolter, Kather- inc Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 12895-12915. PMLR, 21-27 Jul 2024. URL https: http:://////ocecengsss/.. mml. press/v233 matsss//33//aa22eehhmm wt/.//aa224ee.... html.
Christopher Clark, Kenton Lee, Ming-Wot Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019, Asso- ciation for Computational Linguistics. doi: 10. N19-1300. URL https: aocclanhooooooo /rg/ 419-1300.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 1803.05455 2018
Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sad- hika Malladi, and Danqi Chen. Metadata conditioning accelerates language model pre-training arXiv preprint arXiv:2501.01956, 2025.
J. Deng, W. Dong, R. Socher, Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Scale Hierarchical Image Database. In CVPRO9, 2009.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A stan- dard for language model evaluations. arXiv preprint arXiv:2406.08 08446, 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high- quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, pp. 3029-3051, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10.18553 wwt.://....mmmmmmmm.m. main. 183. URL https:: /tatps::/aclasee ant ogy. wtt.llggy..o223333 o2233023. emnlp-main. 1aomn..8///... 183/.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harki- rat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised do- main discovery. arXiv preprint arXiv:2303. /4177, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan,

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A. Smith. Data mixture inference attack: BPE tokenizers reveal training data compositions. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024. URL https:// openreview httprviwwwweeeeee.m wwwt/eeuunnt worum Edxy
Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:://proceeddngggg neurips. cc/ /cc/apeee..iilee/eeer//2//////// 04452451000aee336666d82286667622 pdf.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguisties, 7:452-466, 2019. doi: 10.1162/aac a aclanthology wotg///-1//0//111
William Held, Bhargavi Paranjape, Punit Singh Koura, Mike Lewis, Frank Zhang, and Todor Mihaylov. Opti- mizing pretraining data mixtures with llm-estimated util- ity. Xrviv preprint arXiv:2501.11744 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representa- tions, 2021.
URL https://
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424-8445, Dublin, Ireland, May 2022. Association for Compu- tational Linguisties. doi: 10. 10..1665//110022.ac-looo 577. URL https:://aaaaa https///aaaaatttooy...ooo org/2022.r.220022 acl-lone .57.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015.
URL https: /uaps://app /api. 72003477.
Tao Jiang. Xu Yuan, Yuan Chen, Ke Cheng, Liangmin Wang, Xiaofeng Chen, and Jianfeng Ma. Fuzzyd- edup: Secure fuzzy deduplication for cloud storage. IEEE Transactions on Dependable and Secure Comput- ing. 20(3):4466-244 2483, 2023. doi: 10. 1021 3185313.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg. Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yo Yu Hsich, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El- Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang. Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev. Thomas Kollar, Alexandros G. Di- makis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11 11794, 2024.
Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and Zico Kolter. Adaptive data optimization: Dy- namic sample selection with scaling laws. arXiv preprint arXiv:2410.1 18200 2024.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classi- fication. In Mirella Lapata, Phil Blunsom, and Alexan- der Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics: Volume 2, Short Papers, pp. 427-431, Valencia, Spain, April 2017. Association for Computa- tional Linguistics. URL https: /aclannhooooo..... org/E17-2065
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Au- tomatic prediction of compute-optimal data composi- tion for training 1lms. arXiv preprint arXiv:2407.20177, 2024.
Yuanzhi Li, Séhastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: 1.5 technical report, 2023a.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. Von Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning,
12

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
2023b. URL https://arxiv.orge
lations, 2024. URL https: https::/tppeeeeeeeeeeeeeee http://pppnreeeeereeeee net/ forum?id==TPJJKTT44
03281.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang. Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training, arXiv preprint 7Xii:2407.01492) 2024.
Pratyush Maini, Skyler Seto, Richard Bai, David Grang- ier, Yizhe Zhang, and Navdeep Jaidl. Rephrasing the web: A recipe for compute and data-efficient lan- guage modeling. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Pupers), pp. 14044-14072, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18665//1/2224444H-oggg
Shayne Longpre, Robert Mahari, Ariel N. Lee, Camp- bell S. Lund, Hamidah Oderinwale, William Bran- non, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole J Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ah- mad Mustafa Anis, An Dinh, Caroline Shamiso Chi- tongo, Da Yin, Damien Sileo, Deividas Mataciunas, Di- ganta Misra, Emad A. Alghamdi, Enrico Shippole, Jian- guo Zhang, Joanna Materzynska, Kun Qian, Kusha- gra Tiwary, Lester James Validad Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muen- nighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mo- hanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi LI, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Alex Pentland. Consent in crisis: The rapid decline of the Al data commons. In The Thirty-eight Conference on Neural Information P Processing Systems Datasets and Benchmarks Track, 2024a. URL https: // //openreview, wwt/foorun d=66PcE22F95.
757. URL https: httpa/////aaaannoooo..oooo acl -ong. 757/-
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining LLMs at scale, 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pp. 2381-2391, Brussels, Belgium, October- November 2018. Association for Computational Lin- guistics. doi: 10.18653/w1/D1--1111 //aclanthology .ww.gyy.orrg/oo org/ 800001/0-111200..
URL https:
Shayne Longpre, Gregory Yauney, Emily Reif, Kather- ine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Ja- son Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's guide to training data: Mea- suring the effects of data age, domain coverage, qual- ity, & toxicity. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pp. 3245-3276, Mex- ico City, Mexico, June 2024b. Association for Compu- tational Linguistics. doi: 10.18653/112002..maale 179. URL ht:ps:: http://aaaaatthhoee /ttpa/aaatttoooooorr//222... naac1-long. 179/.
Sahil Mishra, Ujjwal Sudev, and Tanmoy Chakraborty. Flame: Self-supervised low-resource taxonomy expan- sion using large language models. ACM Trans. In- tell. Syst. Technol.. December 2024. ISSN 2157-6904, doi: 10. 14517747/709007. URL https: //ts:://doo.ooo 10. 0..1145/370900 Just Accepted.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scal- ing data-constrained language models. In Thirty-eventth Conference on Neural Information Processing Systems, 2023. URL https: openreview wwt.ewww.eee/eeeeeeeeeeeeeeeeeee forum? id=]5BuTrE]35.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groen- eveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 OLMo 2 Furi- ous. arXiv preprint arXiv:2501.00656. 2024.
Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, and Jesse Dodge. AboutMe: Using self-deseriptions in webpages to document the effects of english pretraining data filters. arXiv preprint wwx.xx2401.0640000 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cap- pelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outper- forming curated corpora with web data only. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Represen-
13

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp- 79155-79172. Curran Associates, Inc., 2023.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pp. 4463-44339 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. /0.18653/w1/DD19-14444 URL https: / //aclann aelllantholllg ooy wrg//1 9000019-114 9-1454/.
Guilherme Penedo, Hynek Kydlřček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024.
Jackson Petty, Sjoerd van Steenkiste, and Tal Linzen. How does code pretraining affect language model task perfor- mance? arXiv preprint arXiv:2409.04556, 2024.
Noam M. Shazeer. GLU variants improve transformer. ArXiv, abs/2002.05202, 2020.
Chau Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit lyyer. TopicGPT: A prompt-based topic mod- eling framework. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume I: Long Papers), pp. 2956 2984,,994,,6 Mex- ico City, Mexico, June 2024. Association for Compu- tational Linguistics. doi: 10. 00.186553/11/2024maael/ 164. URL https:://///ttt http:///aaaaaatlaaa....o naacl-long. 164/.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richard- son, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groen- eveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pre- training research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pp. 15725- Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.116653/1112224acc-looo 840. URL ht acl cel-long, 84C.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Mil- lican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learn- ing with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):588-5551, 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neu- 2024. ISSN 0925-2312. rocomputing, https://tiiingggll..oooooooooooe 127063 doi: URL
Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580 2024.
com/
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answer- ing challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4158, Minncapolis, Min- nesota, June 2019. Association for Computational Lin- guistics. doi: 10.18653/w1/N19-1425 1421. URL https: //aclanntholooy.orggg..
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong. Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient 1lms. arXiv preprint arXiv:2402.09668, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
14

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Alex Tamkin, Miles McCain, Kunal Handa, Esin Dur- mus, Liane Lovitt, Ankur Rathi, Saffron Huang, AI- fred Mountfield, Jerry Hong, Stuart Ritchie, et al. Clio: Privacy-preserving insights into real-world ai use. arXiv preprint (tt::24/2.136755 2024.
The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024. URL https:// openreview net. cetrumiidd222vvNett.o
Zihan Wang. Jingbo Shang, and Ruiqi Zhong. Goal- driven explainable clustering via language descriptions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 10626-10649, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18663/..1200233mmmppp 657. URL https: httpss//aaaaatthooogogee emn 1 p- ..mm//
Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview. wwt./eeooommtteo forum? kuchuuuoVguu......
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving LLM pretraining via doe- ument de-duplication and diversification. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 53983-53995. Curran Associates, Inc., 2023.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4015 Marseille, France, May 2020. European Language Resources As- sociation. ISBN 979-10-95546-34-4, URL https: / //aclanntoloogy.org//2//
TogetherAI. RedPajama: An open source recipe to reproduce Ilama training dataset, 2023. URL https: //gi thw.. httbub/cm/mmogtteerrootttte RedPajama -ataaaama-Data
1rec-1.494
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint MrXiv:2302.1397/ 2023. URL https: /ttp///avvvv/.////////..//...ll. wttppdt/////oooooo.//tpppp...p.p 2302 ww......199..... 1ww.1. pdf.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high high-quality data for training language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 52915-52971, PMLR, 21-27 Jul 2024.
Marlies van der Wees, Arianna Bisazza, Wouter Weerkamp, and Christof Monz. What's in a domain? analyzing genre and topic differences in statistical machine translation. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 560-566, Beijing, China, July 2015. Association for Computa- tional Linguistics. doi: 10.3115/w1/P15-2097 URL https: http///aaaaanttooo..caaa. wrt/:15-222111
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Opti- mizing data mixtures speeds up language model pre- training. In Thirty-sevnnth Conference on Neural In- formation Processing Systems, 2023a. URL https: / // //oeerevieww net. forum? ioetmmmmd=lllmm XuRyUeHhd.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via im- portance resampling. Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Huy V. Vo, Vasil Khalidov, Timothée Darcet, Théo Moutakanni, Nikita Smetanin, Mare Szafraniec, Hugo Touvron, camille couprie, Maxime Oquab, Armand Joulin, Herve Jegou, Patrick Labatut, and Piotr Bo- janowski. Automatic data curation for self-supervised learning: A clustering- based approach. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https:/ http:///pppe/eeeeeee//ss 1d=G7p8dj2W01.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling perfor- mance. arXiv preprint arXiv:2403. 16952, 2024.
Zichun Yu, Spandan Das, and Chenyan Xiong. MATES; Model-aware data selection for efficient pretraining with data influence models. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems,
Jiachen T. Wang, Tong WU, Dawn Song, Prateek Mittal, and Ruoxi Jia. GREATS: Online selection of high- quality data for LLM training in every iteration. In
15

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
2024. URL https:t//////c hhctps://operreiieeetee wttp:eeviiieeeeefffrrrre.. net forum?? id=6gZPSMUAZ2.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ju Fan, Ye Yuan, Guoren Wang, and Conghui He. Harnessing diversity for important data selection in pretraining large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https: /ppppeeeeeeee....//////p / Spotlight.
Yuwei Zhang, Zihan Wang, and Jingho Shang. Cluster- LLM: Large language models as a guide for text clus- tering. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Em- pirical Methods Natural Language Processing, pp- 13903-13920, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/1/2023.
emnlp-main.855 858. URL. https: / ac accannholooyy org/2023. mamlp-main.855.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Chris- tos Kozyrakis, lon Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient exe- cution of structured language model programs. In The Tiihty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024. URL https:// openreview. net for.forrmiioo weerrumidd=Vgaae idvVkAAkibbg.
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Stein- hardt. Describing differences between text distribu- tions with natural language. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pp. 27099-27116 PMLR, 17- 23 Jul 2022. URL. https::///// / proceedinggs ngs. mlr. press vv1 http:/vvl22//eee 2e1161/2hhog22a...m html.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdi- nov, Raquel Urtasun, Antonio Torralba, and Sanja Fi- dler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international con- ference on computer vision, pp. 19-27, 2015. URL /awp:/..rgg/pa///555.....
https: / //aaxivvv.. wttp:///gg/p/oeoooooo 150.. 06724. pdf.
16

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
A. Domain Descriptions
Table 3: Detailed overview of our topic definitions. We mention common sub-topies for specific categories (e.g., Archi- tecture under Art) and discuss ambiguous cases to decrease the uncertainty when prompting model and arrive at sharper domain boundaries.
Topic
Notes
Adult
Art & Design
Includes: architecture
Crime & Law
Includes: law enforcement
Financial crime and litigation fall under 'Finance & Business"
Social issues and the legislative process fall under 'Polities'
Education Jobs
Includes: pedagogy. training & certification, academia Educational pages about a topic, e.g., food or mathematics, fall under that topic
Entertainment
Includes:
music, movies, TV shows, videos, celebrities, humor, nightlife Music film discussed art rather than entertainment falls under 'Art Design'
Fashion & Beauty
Includes:
clothing, accessories, cosmetics
Finance & Business
Includes: taxes, regulations, investments, insurance, credit cards, personal finance, corporate communi- cation, marketing, human resources
Food & Dining
Includes: recipes, groceries. beverages, restaurants
Nutritional sciences fall under 'Health'
Games
Includes: video games, board games, gambling
Hardware
Includes: computer hardware, phones, televisions, other consumer electronics
Health
Includes: medicine, wellness, mental health, veterinary science, nutritional science
Health insurance falls under 'Finance & Business"
History
Includes: geography, archacology
Home & Hobbies
Includes: real estate, renting, relocation, furniture, appliances, home improvement, DIY, gardening, pets. toys, collecting
Industrial General business topics business finance fall under 'Finance & Business'
Topics related to mining, agriculture, manufacturing, utilitics and construction
Includes: raw materials, industrial goods, chemicals, textiles
Literature
Includes: literary criticism, linguistics, philosophy, related subjects the humanities Text written literary style fall under the topic of its contents
Politics
Includes: social issues, political campaigns, the legislative process, geopolitics, protests, activism
Religion
Includes: spirituality
Science & Technology
Includes: physics, chemistry, biology, environmental science, mathematics, statistics, biotech, engineer- ing
Social Life
Includes: family, friends, relationships. community
Specific social activity (e.g., sports or board games) fall under those topics
Software
Topics related to the use of software and the internet
Software Development
Includes: algorithms, coding, and web development
Sports & Fitness
Includes: martial arts, motor sports, outdoor activities, sports equipment
Transportation
Includes: cars and other vehicles, taxis, public transportation, traffic, commuting, aviation, rail, shipping, logistics
Travel
Includes:
hospitality, hotels, sight-seeing, cruises
Detailed descriptions of tourist destinations fall under 'History"
17

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Table 4: Detailed overview of our format definitions. We mention typical features of these formats to help model without HTML access deduce the format from the text.
Format
Notes
About (Org.)
An organizational "About Page", typically containing a self-description or introduction by organization
such company, university, government agency, non-profit
Note that the content may appear similar to "Knowledge Article' in some cases, but not verified and may contain self-promotion
About (Personal)
An "About Page" on personal website or hobby website, lypically containing self-description, intro-
duction profile information
Academic Writing
Examples: a research paper, a paper abstract, a thesis, literature review
Audio Transcript
A written record of spoken language
Examples: interviews (e.g., in newspaper), the transcript of court hearing, movie, podcast, lecture, speech
Comment Section
A comment section or discussion forum with multiple posts or comments
Examples: Community sites like reddit, comment sections news article blogs
Content Listing
The page contains an overview content and is used for navigation Examples: sitemap, product catalog, search results, news listings with short snippets of articles Note that hyperlinks are not visible from the text content and have to be deduced
Creative Writing
The page consists of short story, chapters from novel, poem or song lyrics
Documentation
Examples: technical writing, API documentation, README files, source code
Unlike CCustomer Support', mcant for developers and experts, rather than end-users
FAQ
The page content in the Frequently Asked Questions format
Knowledge Article
Written in an objective and neutral style
Published on a moderated platform (like Wikipedia) or by reputable source
Legal Notices
Examples: terms of service, legal disclaimers, privacy policy, license agreement
Listicle
A blog or article that presents content in the form of list Examples: Buzzfeed-style articles, "Top 10" lists, "4 hest places to visit in X"
Lists showing the site contents and facilitate navigation fall under "Content Listing"
News (Org.)
Organizational news and announcements
Examples: a press release, a blog post by an organization such as a company, university, government agency, non- non-profit organization
News Article
Written by journalists current events and published by news organizations Long reads, profiles, editorials, and journalistic essays fall under 'Nonfiction Writing' Newspaper interviews fall under 'Audio Transcript'
Nonfiction Writing
Long reads, profiles, editorials, essays, obituaries, memoirs and other forms of nonfiction writing, written by journalists and other professional writers
Personal Blog
Written by an individual typically relating personal experiences and opinions
Product Page
Typically contains descriptions and promotions for a product service Also includes products in wider sense, for example university course descriptions
Q&A Forum
A user forum with an explicit question & answer format, e.g., Quora, Stack Exchange
Spam / Ads
The page consists primarily of spam content, SEO keyword stuffing, short online ads for other pages, products or services, or has no apparent purpose
Structured Data
Multiple data entries with a common structure Examples: table, datasheet, movie database, glossary, dictionary. json file, csv, xml
Customer Support
Content by organization and for general audience
Examples: troubleshooting guide
Truncated
The page contents are incomplete, e.g., truncated, pay-walled, or require a login
If the page has multiple snippets of truncated articles, choose 'Content Listing'
Also includes multimedia web pages where the web page text primarily describes and supplements the audiovisual content, e.g., video description or image gallery
Tutorial
Examples: cooking recipes, DIY instructions, WikiHow page, Khan Academy course
The page must contain the actual content of the tutorial / how-to guide
Guides specific products/scrvices from the website fall under 'Customer Support'
User Review
Reviews posted by users, e.g., on Yelp. TripAdvisor
18

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Full prompt We provide descriptions of the domains in Table 3 and Table 4. These domain descriptions given the model part of the prompt (with minor adjustments in phrasing). The prompt template is shown in Table and contains instructions, the text contents and URL of the web page, and the the list of domain descriptions. We randomly permute the order in which we list the domains for every new document, and enumerate the randomly shuffled choices as:
A: fdomain description }} B: {domain description 2}
The random order avoids spurious positional bias from the large language model, and the alphabetic IDs are useful for obtaining single-token outputs from the model, and obtain a soft prediction of domain categories that reflects model uncertainty. We truncate the text contents of web pages at 50K characters and add a truncation hint to the model. We also provide 5 few-shot examples to the model, formatted as previous conversation turns with the same prompt format. Each example is carefully curated to be an interesting case of potential domain conflict and provides an explanation of how the conflict should be resolved. The few-shot examples are also presented in random order for each annotation.
normalize the next-token probabilities of the characters A-X to
Table 5: The prompt template for classifying the topic and format of web page. The first two row shows the templates for system and user prompts, in which {domain) becomes either "topic" and "format" and consrructioss) are substituted with the content of the bottom two rows.
Prompt templates
System
Your task is to classify the {domain) of web pages into one of the following 24 categories: (choices)
(instructions)
User
Consider the following web page:
URL: Yfrl]'
Content: (text)
Your task is to classify the fdomain} of web pages into one of the following 24 categories: (choices)
(instructions)
Instructions
Topic
Choose which topic from the above 1ist is the best match for describing what the web page content is about. I= the content is about multiple topics, choose the one that is most prominent. Remember to focus on the topic, and not the format, e.g., book excerpt about first date is related 'Social Life' and not 'Literature'. The URL might help you understand the content. Avoid shortcuts such as word overlap between the page and the topic descriptions or simple patterns in the URL. Start your response with the single-letter ID of the correct tepic followed by explanation.
Format
Choose which format from the above list is the best match for describing the style, purpose and origin of the web page content. If the content has multiple formats, choose the one that is most prominent. Remember to focus on the format, and not the topic, e.g., a research paper about Legal issues does not count as 'Legal Notices'. The URL might help understand the content. Avoid shortcuts such word overlap between the page and the format descriptions or simple patterns in the URL, for example '.../blog/... comment sections, and other formats. Start your response with the single-letter ID of the correct format followed by an explanation.
may also occur for organizational announcements,
19

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
B. Training Domain Classifiers
Data annotation We obtain training data by prompting Llama models to annotate web pages using the prompts described in Appendix A. This includes randomizing the order in which domain descriptions and few-shot examples are presented to the model for each annotation. For all annotations, we leverage the SGLang inference framework (Zheng et al., 2024), and obtain soft probabilities over all category labels by normalizing the next-token probabilities over the alphabetical category labels. We sample web pages for annotations from the RefinedWeb reproduction released by DataComps-LLM (Li et al., 202c)-which undergoes similar pre-processing steps as our 200B token pre-training corpus (RefinedWeb filtering and deduplication). For the first stage of training, we annotate 1M web pages with 1--ma------B--nsttuue and for the second stage, a subset of 100K web pages is annotated with Llama-3. 1amma-3.1-405B 1--4055--nnsttre, using FP8 inference and 8x H100 NVIDIA GPUs. In both datasets, we reserve the same set 20K web pages as validation and test sets, therefore leaving 80K annotations for the second phase of training. We repeat the annotation process for both the topic and format taxonomies, and train two separate domain classifiers.
Fine-tuning setting We fine-tune gte-base-en-vl.5 embedding model, 140M parameter embedding model, which reports strong performance benchmarks for small model and also features a 8192 token context window (Li et al., 2023b), allowing us to process longer documents. In each training stage, we train for a total of 5 epochs with a total batch size of 512 sequences, a learning rate of le-4 which is warmed up for the first 10% of training steps and linearly decayed. Our main domain classifiers are shown the same web page features as the prompted Llama models, i.e. the text contents and web page URL,, using the template of ' ([url)nnnnfeee (ninttext)'' However, for the potential use case of annotating other documents without URL information, we also produce version of the domain classifiers trained with only the website test as input.
Classifier accuracy We consider how well the domain classifiers imitate the annotations by the (lama-3.1-405B-mme models on the validation set of 10K web pages and focus the subset where the large language model chooses a category with at least 75% confidence-whide is the case for 86% of topic annotations and 79% of format annotations. On this subset, we report both the overall accuracy and the worst-group accuracy, i.c., the worst accuracy when predicting certain label. The results are shown in Table 7. We make the following observations: (1) 2-stage training is particularly effective for improving the worst-group accuracy of the classifiers, and (2) it slightly helps to provide the web page URL to the domain classifiers. Despite these efforts, we note that there remains a gap between the 150M parameter domain classifiers and the 405B parameter Llama-3. -IInstruct model. However, we note that the ceiling for the domain classifier not 100%. Llama-3.1 1.1-nrtruutt-005B is sensitive to the order in which categories and few-shot examples are presented, and a different random seed produces an agreement of only 98% and 97% on this validation subset for topic and formats respectively, suggesting that the domain classifier introduces an additional 4.4%-5.1 % error into the annotation process.
Domain analysis Figure shows the full matrix of normalized PMI scores between topic and format annotations, com- puted across the 200B token annotated pre- pre-training corpus. Figure visualizes the normalized PMI values between k- means clusters and either the topic or the format domains. In Figure 5, we visualize the frequency of URL domains to highlight the need for meaningful graiied domains.
20

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
C. RegMix Implementation
Sampling training mixtures For each domain definition, we generate 512 random domain mixtures for training small models. The mixtures are sam- pled in similar fashion to the official RegMix implementation (Liu et al., 2024). We compute the domain proportions in the pre-training corpus and soften the distribution by applying a temperature of r 2 to obtain the prior distribution p. We then sample training mixtures π hicrarchically via log a ~ (log 0.1, I log 10) and π ~ Dirichlet(ap).
Table 6: Hyperparameters for small model training
Parameter
Value
Hidden size
512
Intermediate size
1536
Small model training We sample 1B tokens according to each training mixture and train small 50M parameter models on this data. The data is tokenized with the GPT-NeoX tokenizer (Black et al., 2022), as used by the DCLM model runs. The model architecture is based
Activation function
SwiGLU
Attention heads Num. blocks
8
RoPE base frequeney
10000
the Llama architec- ture Touvron et al. (2023), featuring SwiGLU activations (Shazeer, 2020) and RoPE positional embeddings (Su et al., 2024). The 512 model runs require approximately 360 NVIDIA H100 hours. The hyperparameters are given in Table
Peak learning rate
3e-3
Cosine cooldown
3e-4
Warmup ratio
10%
Adam ,3's
(0.9, 0.95)
Batch size
128
Simulation We follow Liu et al. (2024) and train a boosted tree regression model to predict downstream loss from the training mixture weights. However, our implementation diverges in the so-called "simulation phase" which seeks to predict the best performing mixture. Liu al. (2024) generate N 1M random mixtures according to π Dirichlet(p), where p is the prior domain distribution in the corpus (without applying temperature here). The regression model is used predict a loss for each mixture, and Liu et al. (2024) average K = 100 mixtures with the lowest loss to produce a prediction for the best mixture. In our exploration, we encountered the issue that the predicted mixture would be sensitive to the random seed and the byperparameters N and K, and it was also not clear how K should vary when increasing NN. Liu et al. (2024) do not provide a clear motivation for averaging, but likely reflects a prior towards smoother distributions. We found that was more convenient to express this by adding a soft KL constraint to the objective, encouraging the prediction to remain closer to the corpus distribution, (p || π), where the coefficient y independent of N. We also found that increasing N led to diminishing returns and developed a multi-step adaptive search method, which reliably identifies better mixtures under the regression mode. In each iteration, the algorithm updates the prior for generating mixtures with the best current candidate mixture. Algorithm 1 lays out the algorithm. We use the hyperparameters N 0.5M mixtures, T = 15 steps, 0.002, 0.2 and run the simulation with two random seeds, choosing the better mixture according to the objective. We make a final modification to RegMix when targeting two downstream tasks, i.e., HellaSwag and MMLU. In this case, we fit two separate regression models for the two tasks, and combine them by averaging their outputs.
Algorithm Adaptive search for RegMix
Input: corpus prior p. num. mixtures N, KL coefficient y, steps T, smoothing n, regression model f Output: predicted mixture q
I: q (f-p) p
Best mixture overall D Soft average of best mixtures in each iteration
2: w +
3: for in do
4: a Uniform (log 1, log 1000), i 1.NN
5: TT() ~ Da ()), w). s.t. x < (1)
No repetitions when selecting 30B out of 200B tokens.
6: arg min,.o ʃ (1) 7gKL p (1)
7:
n ( w + (1 - B,)
By E Linspace 1, 500)
Line search between w and w
8: w- + arg min,(s) ((5) f (1)))
+ X (p π ())
9: w - क + (1 - 7) w
> Update search prior with the best current mixture D Keep track of best mixture so far
10: q + arg min n E q) f (T) 7KL (p
11: end for
Analysis For our mixture predictions, we use all 512 mixtures to train the regression model. In an ablation, we reserve 50 mixtures for evaluations and compute the Spearman correlation between the RegMix predictions and small model
21

---

Organize the Web: Constructing Domains Enhances Pre- Pre-Training Data Curation
Documents Rank
Most Common
With 100-100K documents
blogspol.com wordpress com wikia com cbslocal.com waakkaxxaannge.com com..com urbandictionary.comm
seriouspuzzles
10'
ufostalkerrcom
scotland com
Bactorydireckcraff.coom
saveabunny.org
10
celiphonessco
w willpedia org yahoo com blogspol typepad
ecva/uuum com notcot
10"
harpercollins
veratadvvrriserr
factiiio
#Documents
nytimes,
simplycast.com com le-oonacccom
10
theguardian com Iieeournall com
fallout equastria.com metthithbaodrrwwngg com mednatdiract birevillechoods..
10
go.com mib.com
lalimes.com com reuters.com com blogspol.ca
slodksy.com neds com oblon.con
10
chicegotribuieecommm com
bashinglompost.com
lowerihestre
10
uk rih. rih.gov criel. com
hherainforestgarddn..
zainteriora
10
microsolt.com com
google.com com fool.com
amaaell-lbrarynnn
coach.ca butterscatch.com
16'
0
5,000,000 10,000,000
1000
Figure 5: Frequency statistics of URL domain names in our 200B CommonCrawl corpus. Left: Plotting log document frequency vs. the log rank of the domain name exhibits Zipfian long-tail behavior. Middle and right: We list the most common domain names (left) and a random sample of domains between 100-100K documents (right). We plot statistics after removing any sub-domains, i.e. en www.pediaa.ccmmmmm org + wikipedia org.
evaluations. Table shows the results. The correlation coefficient hovers around 0.90, despite the small size of the models and the out-of-distribution setting of few-shot downstream evaluation. We also explored predicting the held-out distributions using Data Mixing Laws (Ye al., 2024). However, this achieves worse Spearman correlations and seems overall less stable. We also note that predicting the average loss across MMLU and HellaSwag is slightly more accurate predictions when fitting two separate regression models.
The accuracies of domain classifiers to predict confident large language annotations (confidence > 75%).. We report both average accuracy and worst-group accuracy.
Table 7:
Topics
Formats
Avg
Worst
Avg
Worst
Domain classifiers
93.5
87.1
91.8
80.5
w/o 2-stage training
91.8
84.3
90.2
74.1
w/o URL features
92.1
86.0
88.9
80.2
D. Predicted Mixtures
Table 9 reports the numerical results the mixtures visualized Figure 3. addition to our main two tasks of focus, we also include predicted data mixtures for a wider range of downstream tasks in Figure 8. Note that we use bits-per-bytes of the correct solution across all tasks. We use 5 in-context examples for MMLU, HellaSwag, Natural Questions (NQ). 3 examples for HumanEval and MBPP, 0-shot for MATH. We note that both coding tasks upsample documents from the Software Engineering topic and Documentation format. Natural Questions also exhibits distinct patterns as being the only task to upsample the Entertainment topic category heavily.
22

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Software Dev.
Literature
Adult
Science & Tech.
History
Finance & Business
Food Dining
Sports & Fitness
Home & Hobbies
Software
Fashion & Beauty
Crime & Law
Social Life
Industrial
Health
Games
Education Jobs
Politics
Travel
Hardware
Art & Design
Religion
Entertainment
Transportation
Documentation
Creative Writing
Spam Ads
Acaowledge Artiole Legal Notices
Writing
Tutorial
Q&A Forum
News Article
Product Page
Sistmmer Support
Data
User Review
Personal Blog
About (Org.)
Nonfiction Writing
Audio Transcrip
About (Pout(Pa (Pars.)
News (Org.)
Comment Section
Listicle
Truncated
Content Listing
FAQ
Figure 6: The normalized pointwise-muten information matrix between all topics (y-axis) and formats (x-axis). score of 0 indicates independence and implies full co-occurrence.
Table 8: Spearman correlations coefficients when predicting the performance of 50 held-out mixtures using either the LGBM regression model (Ke et al., 2017) by fitting parametric Data Mixing Laws (Ye et al., 2024). When predicting the average performance of both tasks, we also ablate our approach of fitting separate regression models for each task with the default RegMix setting of having a single regression model predict their average (Liu et al., 2024). Clusters correspond to using k-means clusters as domains.
Domains
Target Task
Topics
Formats
Clusters
LGBM Regression
MMLU
0.89
0.86
0.87
HellaSwag
0.94
0.94
Both
0.91
0,91
0.91
w/ single model
0.89
0.89
0.88
Data Mixing Laws
MMLU
0.79
0.84
0.70
HellaSwag
0.80
0.91
0.82
Both
0.73
0.91
0.82
w/ single model
0.64
0.89
0.83
23

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
-1.00
-0.75
-0,50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Games
Documentation Product Page
Sports & Fitness
Travel
Tutorial
Religion
News (Org.)
Palitics
News Article
Fashion Beauly
Crealive Wriling
Literature
Personal Blog
Health
User Raview
Food & Dining
Academic Writing
Transportation
About (Org.)
Art & Design
Q&A Forum
Education Jobs-
Customer Support
Software
Entertainment
Structured Data
Finance & Business
Knowledge Article
Crime Law
Audio Transcript
Social Life
Spam Ads
Adult
About (Pers.)
Industrial
Nonfiction Writing
Hardware
Content Listing
Home Hobbies
Legal Notices
History
Truncated
Science Tech.
FAQ
17
21
18
6
20
13
0
23
18
4
12
2
17
23
18
22
20
7
11
21
Cluster
Cluster ID
Figure/ 7: The normalized pointwise-mutur information (NPMI) matrices between k-means cluster assignments and
the topic annotations (left) format annotations (right). A score of 0 indicates independence and implies full occurrence. We observe that k-means clustering based on document embeddings primarily aligns with topic information.
24

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Table 9: The domain proportions of the corpus, the mixture weights predicted by RegMix, and implicit mixtures used by quality filters. The numbers in parentheses indicate how much the domain is amplified (>> 1.0) or suppressed (<11.0) relative to the corpus.
RegMix
Implicit Mixtures
FineWeb- Edu
DCLM-
Corpus
MMLU
HellaSwag
Both
fasttext
Topics
Entertainment
(a2
(.6
(..
16
8.8
Politics
7.9
9.6
(.5(..7)
(.1
8.5
12.4
Finance & Business
7.5
3.3
6.8(0.9)
3.5
3.7
5.9 (.9(u.s)
Sports & Fitness
7.3
0.1
(1.7
4.5(0.6)
1.5(..2)
3.8
Health
6.5
14.0(2.2)
(..
7.1(.1)
14.3(2.2)
7.6(1.2)
Home Hobbies
5.9
1.1
(6.5(2.8)
10.9
3.0(0.5)
2.4(0.4)
Education & Jobs
5.5
4.4(a.8)
3.7
2.7
8.7(1.0)
3.5
I. Literature
4.9
1.5
(.7
1.3(0.3)
5.9
7.7
Social Life
4.8
(.1
(1.3)
5.1
1.3
4.4(0.9)
Religion
(.7
(.9(1.0)
3.2
8.1(1.7)
5.7
Science Tech.
4.1
26.3(6.4)
2.3(0.5)
25.5
15.8(3.8)
8.4(2.0)
Food & Dining
3.6
1.6(0.4)
(..90.9)
3.6(1.0)
1.(00.4)
2.2
Travel
3.3
3.4
1.6(0.5)
1.40044
1.3(0.4)
1.2
Crime & Law
3.1
5.7
(o.T
3.8
(.27
3.3
Games
(.9
1.8
1.6(0.5)
0.6(0.2)
(.7
Transportation
2.7
(.2
(.0
1.
1.7
1.8(u.6)
Software
2.7
(a.
3.2
1.3(u.s)
2.0(u.7)
2.3 (u.s)
Art & Design
2.4
2.0
(.
1.Qu.4)
2.3
1.4
Fashion Beauty
0.0
4.8(2.0)
1.
0.3
0.7
History
2.3
6.7(3.0)
14(0.6)
4.1(1.8)
9.0(4.0)
3.2
Software Dev.
2.2
4.1(1.9)
2.2(1.0)
1.1
3.6(1.6)
4,8(2.2)
Hardware
2.1
(.2
(.0
1.4(0.7)
1.0
1.7
Industrial
1.7
0.8
1.4(0.8)
0.9
2.4(144)
0.8
Adult
0.0
2.7
0.9
0.0
1.3
Formats
Personal Blog
22.9
26.4(1.))
(1.4
19.7
16.2
26.0
Product Page
6.0(0.5)
7.6
5.3
4.1(0.4)
2.8
News Article
9.0
4.5
7.0
3.1
6.7(0.7)
3,9
Comment Section
8.3
8.7(1.0)
4.8
6,2(0.7)
4.4(0.5)
12.4
Content Listing
7.9
6.9(0.9)
5.6(0.7)
4.1
5.2
1,6
Nonfiction Writing
6.6
S.4(a.8)
(.7
4.5
13.7
11.0
Knowledge Article
6.8(1.9)
2.5
6.2
15.2
6.9
Tutorial
3.6
5.3(1.5)
20.2
20.3
6.7(1.9)
5.1
News (Org.)
3.4
(.7
2.2
0.6(0.2)
2.6(0.8)
0.6
Listicle
3.1
2.0
(..
5.7
2.8(0.9)
3.1
Academic Writing
16.8
(.9
16.9
9.7(3.6)
4.9
Audio Transcript
0.2
1.2
0.1
1.7
5.1
Spam / Ads
2.2
0.0
1.3
0.0
0.5
1.4
Structured Data
2.1
1.2
1.5
1.4
2.8
1.8 (0.9)
Creative Writing
1.9
0.4
(.0
0.3
1.3
5.4
User Review
0.0
1.3
0.0
0.2
1.2
About (Org.)
1.7
2.5
1.3
0.9
1.4
0.3 (u.2)
About (Pers.)
1.1
0.6(0.6)
0.3
0.5
0.2
0.4
Truncated
0.9
0.9
0.2
0.1
0.6
0.2
Q&A Forum
0.8
2.4
(.5
2.8(3.5)
1.2
2.6
Customer Support
0.5
0.3
0.8
0.6(0.8)
0.3
Legal Notices
0.6
0.3
0.2
0.1
0.2
0.1
Documentation
0.6
1.0
0.5
0.1
1.6(2.7)
1.6(2.5)
FAQ
0.4
0.4(1.0)
0.1
0.1
0.5
0.9
25

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
MMLU
HellaSwag
HumanEval
MBPP
MATH
NQ
Entertainment Politics
Finance & Business
Sports & Fitness Health
Home & Hobbies
Education Jobs
Literature
Social Life
Religion
Science & Tech.
Food Dining
Travel
Crime Law
Games
Transportation
Software
Art & Design
Fashion & Beauty
History
Software Dev.
Hardware
Industrial
Adult
100
500
100
500
100
500
100
500
100
500
100
500
% Upsampling
MMLU
HellaSwag
HumanEval
MBPP
MATH
NQ
Personal Blog
Product Page
News Article
Comment Section
Content Listing
Nonfiction Writing
Knowledge Article
Tutorial
News (Org.)
Listicle
Academic Writing
Audio Transcript
Spam / Ads
Structured Data
Creative Writing
User Review
About (Org.)
About (Pers.)
Truncated Q&A Forum
Customer Support
Legal Notices
Documentation
FAQ
100
500
100
500
100
500
100
500
100
500
100
500
Upsampling
Figure 8: The predicted upsampling by RegMix of our topic domains (top) and formats (bottom), proportion of the corpus distributions. Note that the RegMix search constraints the upsampling to a maximum of 650%.
26

---

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
E. Experimental Details
Data pre-processing We use the 1b-1x data pool from DataComps-LM (Li et al., 2024) to facilitate comparisons with future work. The raw pool consists of 1.64T tokens, extracted from CommonCrawl with resilipaaae resiliparse. We follow the best practice established by (Li al., 2024) and run heuristic filtering eliminiate noisy web artifacts, specifically, the set of filters from the RefinedWeb dataset (Penedo et al., 2023). However, to the best of our knowledge, our reproduction differs slightly from Li et al. (2024), since we do not use the "high-quality URL filter", which was originally meant to exclude documents from high-quality domains such as Wikipedia and Github, such that they can be added to the data mix manually. In the next step, we perform deduplication using Bloom filter (Soldaini et al., 2024), while Li et al. (2024) use the MinHash algorithm in their 1b-1x baseline (Broder, 1997). The resulting corpus contains 200B tokens and constitutes the "token universe" for our data selection experiments and analyses. We annotate this corpus with quality scores from the Fine classifier (Penedo al., 2024) and the DCLM fasttext OH-2.5 ELI5 model (Joulin et al., 2017; Li et al., 2024), as well as with the top-1 prediction from the topic and format classifiers, and k-means cluster assignments.
Data selection From this 200B token base corpus, set apart approximately 1B token validation set and use the rest for selecting training data. For each training run, we include enough documents to amount to 30B tokens. This is slightly more than the 29B tokens required by DCLM for a 1b-1x 1x training run, but it ensures that there are enough tokens during model training, since some tokens are dropped in the subsequent tokenization and packing stage. For quality selection, we select the highest scoring documents until the token budget is reached. We speed up the tokenization process by allowing "imbalancedd" chunks, decreasing the chunk size to 2048 sequences, and scaling across many workers. The DCLM default choice of balancing chunks before writing was prohibitively slow on the slurm cluster we used.
Model training We use the 1b-1x reference setting from Li al. (2024). The models have 1,439,795,200 parameters and are trained for 28, 98,795,904,000 tokens with batch size of 256 and sequence length of 2048 tokens. We speed up training by adding torch. comp i le, making a single training run take 183 NVIDIA H100 hours.
Evaluation setting We use the OLMES evaluation framework (Gu et al., 2024) and evaluate 9 tasks with 5-shot in-context learning prompt: MMLU (Hendrycks et al., 2021), HellaSwag (HSwag) (Zellers et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (WinoG) (Sakaguchi et al., 2021), ConnonSenseQA (CSQA) (Talmor et al., 2019), Social IQa (SIQA) (Sap et al., 2019), ARC-easy/challenge (ARC-e/ARC-c) (Clark et al., 2018), and OpenBookQA (OBQA) (Mihaylov et al., 2018). The OLMES task suite also includes BoolQ (Clark al., 2019). However, we found that it produced unreliable results, e.g., the random sampling baseline would achieve 63.8%, and DCLM selection would 54.4%, which 9.4 percentage points lower and would have large impact the average performance.
We also used the DCLM evaluation framework to measure the Core score, normalized task average across 22 tasks (Li et al., 2024), which we report in Table 10. However, we find that OLMES routinely measures higher accuracies in common tasks (HellaSwag, PIQA, and WinoGrande), which useful for discriminating between models. We also observed that some Core tasks from BigBench and AGI eval are close to random performance at the 1 1x scale. Furthermore, given the symbolic nature of some tasks, c.g., dyck sequence completion, MMLU and HellaSwag are likely not good proxies for finding the best domain mixture. Note that we were unable to reproduce the exact Baseline and DCLM-fasttext performance by Li et al. (2024), likely due to small differences in the data pre-processing stage, discussed at the start of this section.
27

---

Organize the Web: Constructing Domains Enhances Pre- Pre-Training Data Curation
Table 10: Detailed results of our data mixing experiments, including the Core score from DCLM (Li et al., 2024) and held-out perplexity the baseline corpus. In each row, highlight the tasks used to optimize the domain mixture.
Data Curation
MMLU
HSwag
PIQA
WinoG
CSQA
SIQA
ARC.
ARC.
OBQA
Avg
Core
PPL
Baseline
30.3
57.5
71.3
56.1
59.0
49.9
62.2
34.0
44.0
51.6
26.1
12.1
Domain mixing: MMLLU
Clusters
32.0
57.0
70.2
55.4
59.4
50.7
64.2
36.1
43.4
52.0
26.0
12.7
Topic
32.3
52.7
68.5
56.0
57.1
70.2
38.7
44.4
52.1
26.7
12.8
Format
32.0
56.3
70.8
55.5
59.5
50.6
66.2
36.9
42.2
52.2
26.7
12.3
Topic Format
33.2
54.1
69.9
55.6
58.6
48.3
71.4
40.9
45.0
53.0
26.4
13.0
Domain mixing: HellaSwag
Clusters
30,5
61.0
74.1
57.1
61.0
49.7
64.4
34.6
42.2
52.7
25.4
12.3
Topic
30.1
60.1
72.8
56.7
57.8
47.6
63.3
32.8
39.4
51.2
24.0
12.3
Format
31.1
60.6
73.0
57.4
60.8
48.7
64.3
35.8
42,4
52.7
27.7
12.2
Topic Format
30.2
61.4
74.0
58.7
61.9
50.3
64.4
35.2
49.2
53.9
27.2
12.4
Domain mixing: MMLU and HellaSwag
Clusters
31.8
59.4
73.4
58.2
58.7
50.7
66.1
35.2
44.8
53.2
26.9
12.7
Topic
31.4
56.2
72.1
54.8
61.3
47.8
70.3
40.6
49.0
53.7
28.5
12.8
Format
31.7
60.9
74.1
56.9
60.
47.4
65.8
35.9
47.6
53.4
27.1
12.5
Topic Format
32.7
60.1
73.4
56.5
62.3
49.3
69.7
38.8
49.0
54.6
28.2
12.6
Quality filtering (+ domain mixing:
MMLU and HellaSwag)
Feb-Web-Edu
34.3
56.0
69.9
57.7
60.0
47.9
71.9
42.3
48.2
$4.2
29.1
14.7
Topic x Format
34.2
62.5
73.3
49.4
72.2
43.3
50.8
56.2
29.8
13.8
DCLM-fasttext
33.4
59.0
70.5
58.8
63.2
50.7
71.4
39.8
48.8
55.1
29.4
14.0
Topic x Format
33.8
63.1
74.3
57.6
62.7
49.8
73.4
42.2
47.8
56.1
30.2
13.7
28


Reviews:
[{"review_id": "25F5PO6YWW", "reviewer": "Reviewer_FJKu", "summary": "The authors propose a framework for organizing domains from unstructured web corpora along dimensions of topic and format. They use a human-in-the-loop procedure to design domain taxonomies and train a small language model to automatically categorize documents in a pre-training corpus, a 200B token subset of CommonCrawl. Empirical experiments show that using their proposed notions of domain along both topic and format helps improve downstream task performance when combined with quality filters and domain reweighting, though reweighting data domains alone is shown not to match the benefits of filtering alone.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for regarding our paper as a solid contribution to the field with strong empirical results. We are especially grateful for the detailed suggestions with respect to the typos and clarity in Section 5, and the many interesting and relevant citations. We will improve the paper with your feedback and discuss these references in the related works section. We will also use the extra page of the final version to add a clear limitations section before the impact statement!\n\n> Domain mixing results for only two downstream tasks\n\nWe focus on MMLU and HellaSwag, since these tasks exhibit interesting qualitative differences, are widely known in the community and are used by most prior works as stable and relevant metrics for data selection. However, we will include topic and format predictions for additional tasks in the appendix of the next version. Training additional models on many target task mixtures is unfortunately beyond our current computational resources and we believe that is equally important to hold out a set of tasks for measuring generalization to “unseen” tasks.\n\nQuestions:\n\n1. Applicable to more general LLM corpora?\n\nWhile we specifically designed our taxonomy with the goal of structuring many terabytes of web data, our domain classifiers can be applied to any text domain (we will also release a classifier that does not require an URL) and we would expect our models to produce reasonable results. Adapting the taxonomy is more complicated and computationally expensive — one would need to annotate additional data and retrain the domain classifiers — but our open-sourced scripts should make these operations run smoothly!\n\n2. GPU time for training domain classifier?\n\nGreat question! Fine-tuning the domain classifier is relatively cheap. The first stage of training takes less than 8 H100 hours and the second stage less than 1 H100 hour. By far the most extensive stage is annotating 100k web pages with Llama-3.1-405B, which takes approximately 600 H100 hours.\n\n3. Why gte-base-en-v1.5?\n\ngte-base-en-v1.5 is the best small-scale embedding model on MTEB at the time of writing with support for longer documents (up to 8192 tokens). Unlike bag-of-word representations (like tf-idf), the neural encoder can produce embeddings which also reflect semantic and conceptual aspects of the source document.\n\n4. Avoiding contamination\n\nThe task examples for fitting RegMix and evaluating our final models are from completely disjoint sets, and we never use final test examples for fitting domains. Since our domain mixtures are very low-dimensional, it is unlikely to leak substantial information from the test sets, but it is still best practice to use a different split for mixture optimization.\n\n5. Takeaways from Figure 4\n\nDCLM-fasttext and FineWeb-Edu are currently used to create some of the most widely used pre-training datasets and Figure 4 reflects the composition of these datasets. By comparing to the RegMix predictions, it also visualizes how the pre-training distributions change in the final four rows of Table 1, giving us some indication of which types of content are either “overlooked” or “overvalued” by the quality classifier. For example, we observe that FinWeb-edu has a strong inherent topic bias for Science, Health, and History, but overly filters  out more diverse topics. Meanwhile, DCLM-fasttext uniquely has strong preferences for the formats Comment Sections and Creative Writing. We will expand on this discussion in the next version of the paper.\n\n\n6. Perplexity scores in Table 2\n\nThese are computed on the same held-out data of the baseline corpus! We will add this for clarification.", "claims_and_evidence": "Claims supported", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Sound experimentation", "relation_to_broader_scientific_literature": "Well-grounded in broader scientific literature. Very solid contribution, substantial enough but not groundbreaking to the point of being controversial", "essential_references_not_discussed": "See other Comments for additional reference suggestions -- I would consider them relevant but not \"essential\"", "other_strengths_and_weaknesses": "Strengths:\n1. Well-written -- a sincere pleasure to read overall\n2. Strong empirical results\n3. Compelling proposed domain taxonomy that may be useful to a broad audience\n4. Interesting explorations across a broad set of ablations, including a k-means clustering baseline and a framing of quality filters as implicit domain reweighting\n5. Overall high quality reproducibility efforts for established dataset filtering\n\nWeaknesses:\n1. Some clarity issues. In particular it is at times difficult to follow the reasoning for including the Section 5 experiment in the paper\n2. Domain mixing results for only two downstream tasks are included, MMLU and HellaSwag (+ their combination), though the respective mixtures found are evaluated on a larger set of downstream tasks. I feel the paper would be stronger with more than just two domain mixing experiments, though I do not feel the current claims are necessarily undersupported as is. A skeptical reader may feel the results veer towards anecdotal\n3. I would have liked to see more explicit discussion of limitations of the present study", "other_comments_or_suggestions": "Clarity issues and typos:\n1. 2.2, \"fine-tuning *a* small classifier model\n2. I found the choice of wording in line 300 confusing (\"Domain mixing is broadly effective\" paragraph in 4.4). Unsure what exactly is meant by the \"Despite the target task accuracy\" sentence. In any case it does not seem essential to the paragraph, but I wonder if it might be more clear if the authors framed it as tasks being in-domain vs out-of-domain (if a matter of which tasks the domain mixture was geared towards), or simply mentioned the lower performance on HellaSwag specifically (if a matter of the underperformance on that single task) \n3. Clarify what the \"random sampling baseline\" is referring to in 4.4's \"Domain Mixtures improve quality filters\" paragraph\n4. I had trouble following the \"Approximating quality filters by domains\" paragraph in general, partially due to the density of typos (see below). More detail in the associated Table 2 caption could help. It is difficult to understand exactly what the \"Baseline corpus\" is out of context, and why perplexity score is expected to be lowest on it (see Q6 below). Explicitly framing the experiment as an ablation could help prevent misunderstandings as well: \"document-level quality filtering is a far stronger intervention on the pre-training distribution than rebalancing domains or topics\" feels possibly irrelevant to the key points of the paper. I would rather see a discussion of e.g. expected % values in Table 2 given particular hypotheses, or some kind of counterfactual reasoning about the results\n5. Line 365, \"In this case, *the* implicit Topic x Format mixture\"\n6. Line 370, \"this classifier could *be* better suited\"\n7. Line 372, \"values for domain mixing *are* substantially lower\n\nAdditional related work the authors may be interested in citing\n1. [Chronopoulou et al., 2022](https://aclanthology.org/2022.naacl-main.96/) and [Reid et al., 2022](https://aclanthology.org/2022.emnlp-main.63/) propose two different hierarchical domain taxonomies. [Chronopoulou et al., 2022](https://aclanthology.org/2022.naacl-main.96/) propose a provenance-based taxonomy where leaf domains are from the top 100 URL domains from C4, and resulting domains may span both topic and format. [Reid et al., 2022](https://aclanthology.org/2022.emnlp-main.63/) propose two hierarchical levels of fine-grained and higher level topical domains. Relevant since the authors mention potential future work further breaking down their proposed domains into hierarchical taxonomies.\n2. [Na et al., 2024](https://aclanthology.org/2024.emnlp-main.1176/) is another data mixing methodology paper, though \"optimizing\" data mixtures is not proposed as explicitly (no proportion reweighting experiments, only speculation of the method's extensibility towards reweighting). Relevant since the current submission's authors mention efficiency concerns from selecting a training mixture over the entire cross product of topic x format domains proposed and Na et al. propose a way to efficiently approximate fine-grained data ablations over many possible data mixtures with model merging instead of additional training. At the very least probably a Related Work citation (probably either Data mixture optimization or Analysis of pre-training data)\n3. [Held et al., 2025](https://arxiv.org/abs/2501.11747) is very recent and only a pre-print but also feels pretty relevant as another data mixing methodology paper. I am not familiar enough with RegMix or UtiliMax to know whether UtiliMax is expected to be a promising alternative over RegMix for the current submission and/or any follow-up experiments, but I felt it is at the very least a related works citation (Data mixture optimization)\n4. More influence functions literature for \"Analysis of pre-training data\": [Choe et al., 2024](https://arxiv.org/abs/2405.13954), [Yu et al., 2024](https://proceedings.neurips.cc/paper_files/paper/2024/hash/c4bec0d2fd217e6c2c3eafeced432582-Abstract-Conference.html)\n\nI do not consider any of these to be \"essential\" references, necessarily, but I do feel engagement with these works could improve the overall quality of the paper", "questions_for_authors": "1. Is the proposed taxonomy expected to be applicable out of the box to more general LLM corpora (including e.g. synthetic data, proprietary data, and other content not typically found readily on the web)? If there are limitations, how feasible or compute intensive would it be to adapt the proposed taxonomy to fit a new web corpus e.g. adding just one more topical domain or one more format domain?\n2. In general, I am curious to know what the work for 2.2 (Training domain classifiers for scaling annotations) and the related appendix section required in terms of GPU hours, and in particular how it compares to the projected cost of annotating documents with Llama 405b\n3. Why was `gte-base-en-v1.5` chosen to compute document embeddings (as opposed to e.g. tf-idf embeddings used by [Gururangan et al](https://arxiv.org/abs/2303.14177), whom the authors cite as an exemplary baseline for k-means clustering of documents?\n4. I had trouble following a detail of Target Tasks in Section 3: \"To avoid contamination, we use the training and validation set of [MMLU and HellaSwag], respectively.\" How is this expected to help avoid contamination?\n5. What are the key takeaways of Figure 4?\n6. Are perplexity scores reported in Table 2 with respect to the baseline corpus's held-out data, or is there a different held-out set for each row?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePnl7EYROq", "reviewer": "Reviewer_jae1", "summary": "This paper tackles two main challenges: first is to separate the huge pre-training corpus into groups of domains and styles, and the second part is to compute the optimum mixing/reweighting strategy to combine these datasets to achieve targeted improvements in few specific metrics like MMLU and HellsSwag. They also demonstrate that this approach is potentially to quality filtering with complementary gains using both. Experiments further validate the hypothesis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We are glad that you highlighted our extensive experiments and analyses, as well as the scalability and practical value of our proposed approach. We would like to briefly address two of your points.\n\n> Reliance on RegMix\n\nIn the paper, we clearly acknowledge that we use RegMix as an existing technique, since our focus is not on contributing another method for optimizing domain ratios given a set of data sources. Instead, we believe that the greater novelty of our work is to shift attention to defining the subsets for effective and large-scale data mixing. Nevertheless, we still extend RegMix to a novel setting, where we directly optimize the task loss of two downstream benchmarks (the bits-per-byte of the correct answers), instead of the perplexity on a corpus such as C4. In Appendix C, we also discuss several new modifications to RegMix that facilitate this, including fitting multiple per-task regression models and employing an iterative search method.\n\n> Scalability to improve multiple evaluation metrics is unclear\n\nWe will make predictions of the topic and format mixtures for additional target tasks (the existing RegMix checkpoint only need to be evaluated on new tasks) and include these in the appendix in the final version of the paper. In our current draft, we focus on only two target tasks, MMLU and HellaSwag, since (a) these tasks are well known in the community and are commonly used for evaluating data selection due to their relevance and low variance (b) we can more easily associate characteristics of the combined mixture with one of the two tasks (c) it was not clear to us how many tasks should be averaged (e.g., should we assign a higher weight to MMLU than one of multiple commonsense tasks?) (d) we reserve more tasks to be used for evaluating generalization to “unseen” tasks. In fact, we find that mixtures for MMLU and HellaSwag generalize well across many popular benchmark tasks. However, we agree with the reviewer that finding the “right” target tasks will remain a question for future research.", "claims_and_evidence": "Yes. both the issues claimed have supporting evidence in the form of ample experiments and analysis.", "methods_and_evaluation_criteria": "Yes. the paper targets improvement in evaluation datasets MMLU and HellaSwag, so the use of them and the resulting improvements are totally valid.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes. the paper targets improvement in evaluation datasets MMLU and HellaSwag, so the use of them and the resulting improvements are totally valid.", "relation_to_broader_scientific_literature": "The paper provides a thorough comparison with the existing literature. It also includes comparison and contrast with respect to the quality filter literature, making it comprehensively clear the advantages in the paper.", "essential_references_not_discussed": "Not an expert in the field, so not sure.", "other_strengths_and_weaknesses": "*Strengths*\n\n- The paper addresses a very important problem of making sense of billion-scale training data for LLMs, by suggesting to group them into domains and styles along with an effective mixture technique.\n- The method is easily scalable to bigger datasets, and the choice to run experiments with smaller models and extrapolate observations to bigger models is effective.\n- The gains are complementary to quality metrics, so there is room for further improvement. \n\n*Weaknesses*\n\n- The whole technique to extract the mixture using RegMix is directly derived from a prior work RegMix, so the new ideas in that part are limited. \n- The paper only focuses on improving two metrics, and scalability to improve multiple evaluation metrics seems unclear from the paper.", "other_comments_or_suggestions": "Please refer above.", "questions_for_authors": "Please refer to the weaknesses section above and answer the questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}, {"review_id": "ze6Hd3t2TI", "reviewer": "Reviewer_oDMK", "summary": "The paper introduces WebOrganizer, a framework to categorize pre-training data for LMs using topics and formats. The labels are annotated by a pretrained LM. Experiments show that (1) reorganizing data according to these domain classifications significantly improves downstream performance, (2) integrating WebOrganizer with existing quality-based data filtering methods further boosts model performance, (3) existing quality filters are implicitly doing filtering based on topics and formats.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind review! We are glad you found the claims in the paper clear and supported by convincing experiments!\n\n> Assumption of selecting topics and formats independently\n\nWe don’t mean to make an assumption in the theoretical sense, but we simply propose to form a mixture distribution of p(topic, format) = p(topic) p(format) given individual mixture distributions p(topic) and p(format). This is motivated by Section 2.3, where we find that topics and formats often have close to zero pointwise mutual information, i.e., they co-occur almost as though they were independently distributed.", "claims_and_evidence": "Most claims made in this paper are generally clear and supported by convincing experiments. One assumption I don't understand is Sec 4.2 \"we make the assumption that we can select topics and formats independently\", an analysis on a smaller subset of pretraining data could be interesting.", "methods_and_evaluation_criteria": "Most methods and evaluation are adopted from existing works (e.g., pretraining data, domain importance estimation methods, benchmarks). They are pretty standard and make sense for the problem.", "theoretical_claims": "The paper does not include significant theoretical claims or formal proofs. I don't fully understand the assumption in Sec 4.2 \"we make the assumption that we can select topics and formats independently\".", "experimental_designs_or_analyses": "The pretraining and evaluation designs (for both domain mixing and its combination with quality filters) sound valid to me.", "relation_to_broader_scientific_literature": "The paper is clearly related to existing literature on pre-training data selection and quality filtering. It shows that (1) a simple domain-based filtering is more effective than quality-based filtering, (2) domain-based selection can improve quality-based filtering.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9NABVZavY", "reviewer": "Reviewer_suAs", "summary": "This paper presents WebOrganizer, a method to construct, and annotate two sets of explainable and orthogonal domains (24 topics and 24 formats) with LLMs (e.g., Llama-3.1-405B-Instruct). The annotations are then approximated by a smaller (140M) model, making it scale efficiently to large corpora. It then employs the technique from RegMix to find the optimal domain mixture to optimize target performance by treating MMLU and HellaSwag as reference tasks. Experiment results show that the annotated domains can curate the pretraining data better than semantic clustering, and also augment quality-based filters to further improve the data curation outcome. Additionally, topic and format domains are basically orthogonal and complement each other. Analyses provide further insights into how quality-based filters implicitly change the domain mixture.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for engaging closely with our work! We are glad you found that our methodology is overall sound and that our claims are supported by clear evidence. However, you raised several points with respect to evaluation and novelty, which we hope to address below.\n\n> Comparing to greater number of clusters in SemDeDup and Quad\n\nThe central premise of our paper revolves around the question of how to partition a monolithic dataset into meaningful subsets for data mixing. Therefore, we consider different ways of structuring the data into an equal number of partitions and apply a fixed domain mixture optimization algorithm (for which we use RegMix). We believe this is a fair experimental setting given our paper’s goals. However, the reviewer correctly points out interesting future extensions, such as studying the effect of domain granularity.\n\nThe two mentioned papers, SemDeDup [1] and Quad [2], are fantastic related works, but neither of them perform k-means clustering with the goal of learning a data mixture. Instead, SemDeDup [1] performs clustering to compute “prototypicality” scores and select data points that are least similar to the cluster center — therefore performing sample-level filtering — and the DCLM paper finds that it performs worse than the RefinedWeb baseline (Table 4).  Quad [2] employs fine-grained clustering to obtain robust and efficient influence estimates across groups and then employs binary thresholding to select groups of similar data points from a corpus. Their proposed algorithm is designed for fine-grained group selection instead of data mixture optimization and requires careful tuning of the alpha and gamma hyperparameters. We agree with the reviewer that it would be interesting to add Quad to the experiments, but we would not think of it as a baseline for domain mixing, but rather compare it to FineWeb-Edu and DCLM-fasttext and see whether Quad would also benefit from reweighting domains. We hope to add these experiments in the final version of the paper.\n\n> DCLM Core is more informative and comprehensive than OLMES\n\nDCLM Core may contain a more comprehensive range of tasks, but in our experiments, we noticed substantial variance with the DCLM Core tasks at the 1B parameter scale (up to 1 point) and many tasks were close to random guessing performance. In fact, the DCLM paper acknowledges that other evaluation frameworks such as LightEval may be more suitable for evaluating small models. We also note that data mixing is a more subtle intervention on the training data than quality filtering, requiring more stable evaluations to reveal small differences between models. Ultimately, we found OLMES to be more reliable and informative. We also note that DCLM Core does not include MMLU, although MMLU is undoubtedly one of the most popular benchmarks for LLMs. We agree with the reviewer that it would be an interesting avenue for future work to investigate better and more diverse proxy data for domain optimization, but this was ultimately not the focus of our work. Instead, we chose MMLU and HellaSwag as they are widely known in the research community and used in almost all previous data selection works.\n\n> It would be better to make domain construction more automated with less human effort\n\nSince pre-training data curation is so crucial in shaping the behavior of a model that may potentially be used by millions of people, we argue that there is some value in retaining human oversight in the data curation process! Nevertheless, we expect future methods to automate the process of taxonomizing and structuring the web with much finer granularity.\n\n> Relation to Bai et al.\n\nThank you for bringing this interesting paper to our attention! We agree that there are similarities with our work and we will definitely highlight them in the next version of our paper. Most notably, Bai et al. [3] also contribute a topic classifier of 13 topics derived by summarizing k-means clusters, which we will compare to our 24 topics. From our analysis in Section 2.4, we would expect k-means clusters to reflect topic information, whereas it would be harder to obtain format domains this way. Besides contributing a high-quality set of format domains, we believe our work makes several independent contributions: (1) We show that topic and formats are approximately orthogonal and their distributions can be combined, (2) We propose a new, simple and effective way of combining data mixing with quality-based data selection, (3) We study the implicit domain biases of two popular quality filters.\n\nQuestions: Unfortunately, we have reached the rebuttal space limit and cannot respond to the interesting questions here\n\n[1] Abbas et al. \"SemDeDup: Data-efficient learning at web-scale through semantic deduplication\"\n\n[2] Zhang et al. \"Harnessing Diversity for Important Data Selection in Pretraining Large Language Models\"\n\n[3] Bai et al. \"Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining\"", "claims_and_evidence": "Claims are supported by clear evidence, but some experimental and evaluation drawbacks below may make the claims about the improvements over baseline methods less convincing:\n\n- The cluster baselines only group the data into 24 clusters, which is far away from the optimal cluster number (~10k) in previous works [1][2]. I don't think the number of clusters needs to be strictly the same as the number of domains in their comparison.\n- The improvements on OLMES (Table 1) may not smoothly transfer to DCLM Core (which is **more informative and comprehensive** from my perspective). For example, the topic × format one cannot outperform a single topic or format in DCLM Core.\n\n[1]: Abbas, Amro, et al. \"Semdedup: Data-efficient learning at web-scale through semantic deduplication.\" arXiv preprint arXiv:2303.09540 (2023).\n\n[2]: Zhang, Chi, et al. \"Harnessing Diversity for Important Data Selection in Pretraining Large Language Models.\" arXiv preprint arXiv:2409.16986 (2024).", "methods_and_evaluation_criteria": "Their methods to construct, and annotate two sets of explainable and orthogonal domains, and optimize their mixtures to improve target performance overall make sense. It would be better to make the domain construction more automated with fewer human efforts.\n\nIn terms of the evaluation criteria, I do have some different opinions. The authors mention in the Appendix that *given the symbolic nature of some tasks, e.g., dyck sequence completion, MMLU, and HellaSwag are likely not good proxies for finding the best domain mixture.*, while I think it is a better practice for this research to adopt better proxy data that can generalize better than MMLU and HellaSwag (e.g., some instruction data), rather than limit the main evaluation to the tasks more related to the chosen proxies. In other words, it is more convincing to choose DCLM Core as the main evaluation rather than OLMES since the main setup of this work is based on DCLM.", "theoretical_claims": "This paper does not make theoretical claims and focuses mostly on the empirical part.", "experimental_designs_or_analyses": "The experiment and analysis design is overall sound and self-consistent, except for the cluster baseline/evaluation issues I mentioned above, which I feel are important for my judgment.", "relation_to_broader_scientific_literature": "This paper is related to previous domain mixing work in pretraining scenarios but extends the concepts of domains from simple web URLs to more fine-grained and explainable topics/formats identified by LLMs, which can provide a new perspective to pretraining data curation. However, the key technical contributions seem a little bit limited due to the existence of similar previous research [1].\n\n[1] Bai, Tianyi, et al. \"Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining.\" arXiv preprint arXiv:2410.08102 (2024).", "essential_references_not_discussed": "I think [1] is a paper that has similar contributions to this one, which leverages GPT-4o as a topic annotator and trains a BERT-based model as the topic classifier. Its data selection pipeline also considers both domain, topic, and quality dimensions. I have not found the citation to [1] in this paper and I think the contribution of WebOrganizer seems a little bit limited given the similarity to prior works. **It also publicly releases its BERT topic classifier, which could serve as a direct comparison to the topic classifiers in this paper.**\n\n[1] Bai, Tianyi, et al. \"Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining.\" arXiv preprint arXiv:2410.08102 (2024).", "other_strengths_and_weaknesses": "**Strengths**\n\n- This paper provides an explainable technique to analyze the inherent composition of pretraining data and the behavior of quality filters. \n- WebOrganizer has the potential to improve over current individual quality-based filters by identifying the optimal domain weights.\n- The writing is clear.\n\n**Weaknesses**\n\n- The proposed method to identify and annotate topic domains, technically, does not significantly vary from prior works (please refer to Essential References Not Discussed).\n\n- The choice of the domain taxonomies (topic and format) and the number of used domains (24) are somehow intuitively set (or found) by humans. A more fundamental question would be how to identify the taxonomy and granularity of domain classifications more automatically.\n\n- The experiment setups can be polished as well. (1) the full potential of the clustering baseline has not been thoroughly presented; (2) the improvements over a more holistic evaluation set (DCLM Core in Table 10) are not as significant as in Table 1. DCLM is meant to be a standardized benchmark to evaluate the pretraining curation outcome and it is better to follow their evaluation setup. Specifically, the DCLM Core set consists of more than 20 tasks while the 9 evaluation tasks in OLMES seem to favor too much knowledge and commonsense abilities. Furthermore, the authors use the MMLU training sets as the target tasks, which already have the in-domain data from ARC-e, ARC-c, and OBQA.", "other_comments_or_suggestions": "I would also suggest the authors try out different (maybe larger) DCLM setups to show the generalization abilities of their method, but the feasibility definitely depends on the available computational resources the authors have because pretraining is always costly.", "questions_for_authors": "1. What is the clustering performance if we scale the number of clusters from 24 to 10k-level following the best practices in prior works [1][2]?\n2. What if we combine clusters x format in the data selection because *clusters mostly align with topics* in terms of both characteristics and evaluation results? Will it match the performance of the topic × format? Also, do you have some insights on why a single topic or format classifier cannot outperform clusters too much?\n3. Can you provide a comparison of your topic classifier with the one released by [3]?\n4. What is the detailed cost (preferred specific numbers) to adopt RegMix to find the optimal domain mixtures, i.e., training 512 models of 50M parameters for 1B tokens?\n5. In Lines 142-143, could you provide more details on how humans and LLM (Llama-3.1-405B-Instruct) work together to refine the topic and format definitions?\n6. How do you extract *75% confidence* from LLM annotation?\n7. This is just out of curiosity: will different annotation LLMs (like DeepSeek) have different behaviors in terms of classifying domains?\n\n[1]: Abbas, Amro, et al. \"Semdedup: Data-efficient learning at web-scale through semantic deduplication.\" arXiv preprint arXiv:2303.09540 (2023).\n\n[2]: Zhang, Chi, et al. \"Harnessing Diversity for Important Data Selection in Pretraining Large Language Models.\" arXiv preprint arXiv:2409.16986 (2024).\n\n[3] Bai, Tianyi, et al. \"Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining.\" arXiv preprint arXiv:2410.08102 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a data curation method that organize the web corpora by developing taxonomies of their contents and organize them into domains. The allows the study of mixing data to improve models on downstream tasks. The work is novel and solid, and the results are promising. One remaining concern is the compute cost of the optimization of domain mixtures, which is higher than the actual DCLM 1B-1x pretraining that needs only 240 H100 hours. This point needs to be clarified and discussed. Also, the related work [Bai, Tianyi, etal] needs further discussion. Overall, I recommend acceptance given the solid contribution.
Decision: Accept (poster)