BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale
  Pretraining
Authors: name: Pratyush Maini, name: Vineeth Dorna, name: Parth Doshi, name: Aldo Carranza, name: Fan Pan, name: Jack Urbanek, name: Paul Burstein, name: Alex Fang, name: Alvin Deng, name: Amro Abbas, name: Brett Larsen, name: Cody Blakeney, name: Charvi Bannur, name: Christina Baek, name: Darren Teh, name: David Schwab, name: Haakon Mongstad, name: Haoli Yin, name: Josh Wills, name: Kaleigh Mentzer, name: Luke Merrick, name: Ricardo Monti, name: Rishabh Adiga, name: Siddharth Joshi, name: Spandan Das, name: Zhengping Wang, name: Bogdan Gaza, name: Ari Morcos, name: Matthew Leavitt

Recent advances in large language model (LLM) pretraining have shown that
simply scaling data quantity eventually leads to diminishing returns, hitting a
data wall. In response, the use of synthetic data for pretraining has emerged
as a promising paradigm for pushing the frontier of performance. Despite this,
the factors affecting synthetic data quality remain poorly understood. In this
work, we introduce BeyondWeb, a synthetic data generation framework that
produces high-quality synthetic data for pretraining. BeyondWeb significantly
extends the capabilities of traditional web-scale datasets, outperforming
state-of-the-art synthetic pretraining datasets such as Cosmopedia and
Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1
percentage points (pp) and 2.6pp, respectively, when averaged across a suite of
14 benchmark evaluations. It delivers up to 7.7x faster training than open web
data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for
180B tokens on BeyondWeb outperforms an 8B model trained for the same token
budget on Cosmopedia. We also present several insights from BeyondWeb on
synthetic data for pretraining: what drives its benefits, which data to
rephrase and how, and the impact of model size and family on data quality.
Overall, our work shows that there's no silver bullet for generating
high-quality synthetic pretraining data. The best outcomes require jointly
optimizing many factors, a challenging task that requires rigorous science and
practical expertise. Naive approaches can yield modest improvements,
potentially at great cost, while well-executed methods can yield transformative
improvements, as exemplified by BeyondWeb.