Essential-Web v1.0: 24T tokens of organized web data
Authors: name: Essential AI; name: Andrew Hojel; name: Michael Pust; name: Tim Romanski; name: Yash Vanjani; name: Ritvik Kapila; name: Mohit Parmar; name: Adarsh Chaluvaraju; name: Alok Tripathy; name: Anil Thomas; name: Ashish Tanwer; name: Darsh J Shah; name: Ishaan Shah; name: Karl Stratos; name: Khoi Nguyen; name: Kurt Smith; name: Michael Callahan; name: Peter Rushton; name: Philip Monk; name: Platon Mazarakis; name: Saad Jamal; name: Saurabh Srivastava; name: Somanshu Singla; name: Ashish Vaswani
Published: 2025-06-17 02:03:36

Abstract: Data plays the most prominent role in how language models acquire skills and
knowledge. The lack of massive, well-organized pre-training datasets results in
costly and inaccessible data pipelines. We present Essential-Web v1.0, a
24-trillion-token dataset in which every document is annotated with a
twelve-category taxonomy covering topic, format, content complexity, and
quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned
0.5b-parameter model that achieves an annotator agreement within 3% of
Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain
competitive web-curated datasets in math (-8.0% relative to SOTA), web code
(+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on
HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0