ICML 2025

Organize the Web: Constructing Domains Enhances
Pre-Training Data Curation
Alexander Wettig 1 2 Kyle Lo 2 Sewon Min 3 2 Hannaneh Hajishirzi 2 4 Danqi Chen 1 Luca Soldaini 2
Abstract
Modern language models are trained on large,
unstructured datasets consisting of trillions of to-
kens and obtained by crawling the web.
The
unstructured nature makes it difficult to reason
about their contents and develop systematic ap-
proaches to data curation. In this paper, we un-
pack monolithic web corpora by developing tax-
onomies of their contents and organizing them
into domains. We introduce WebOrganizer, a
framework for organizing web pages in terms of
both their topic and format.
Using these two
complementary notions of domains, we automat-
ically annotate pre-training data by distilling an-
notations from a large language model into effi-
cient classifiers. This allows us to study how data
from different domains should be mixed to im-
prove models on downstream tasks, and we show
that we can combine insights about effective top-
ics and formats to further boost performance.
We demonstrate that our domain mixing also im-
proves existing methods that select data based on
quality. Furthermore, we study and compare how
quality-based methods will implicitly change the
domain mixture. Overall, our work demonstrates
that constructing and mixing domains provides a
valuable complement to quality-based data cura-
tion methods, opening new avenues for effective
and insightful pre-training data curation.
Website
weborganizer.allen.ai
Artifacts
hf.co/WebOrganizer
Code
CodeCreator/WebOrganizer
1Princeton Language and Intelligence, Princeton University
2Allen Institute for Artificial Intelligence 3University of Califor-
nia, Berkeley 4Paul G. Allen School of Computer Science & Engi-
neering, University of Washington. Correspondence to: Alexan-
der Wettig <awettig@cs.princeton.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. C...