The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly
  Licensed Text
Authors: name: Nikhil Kandpal, name: Brian Lester, name: Colin Raffel, name: Sebastian Majstorovic, name: Stella Biderman, name: Baber Abbasi, name: Luca Soldaini, name: Enrico Shippole, name: A. Feder Cooper, name: Aviya Skowron, name: John Kirchenbauer, name: Shayne Longpre, name: Lintang Sutawika, name: Alon Albalak, name: Zhenlin Xu, name: Guilherme Penedo, name: Loubna Ben Allal, name: Elie Bakouch, name: John David Pressman, name: Honglu Fan, name: Dashiell Stander, name: Guangyu Song, name: Aaron Gokaslan, name: Tom Goldstein, name: Brian R. Bartoldson, name: Bhavya Kailkhura, name: Tyler Murray

Large language models (LLMs) are typically trained on enormous quantities of
unlicensed text, a practice that has led to scrutiny due to possible
intellectual property infringement and ethical concerns. Training LLMs on
openly licensed text presents a first step towards addressing these issues, but
prior data collection efforts have yielded datasets too small or low-quality to
produce performant LLMs. To address this gap, we collect, curate, and release
the Common Pile v0.1, an eight terabyte collection of openly licensed text
designed for LLM pretraining. The Common Pile comprises content from 30 sources
that span diverse domains including research papers, code, books,
encyclopedias, educational materials, audio transcripts, and more. Crucially,
we validate our efforts by training two 7 billion parameter LLMs on text from
the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion
tokens respectively. Both models attain competitive performance to LLMs trained
on unlicensed text with similar computational budgets, such as Llama 1 and 2
7B. In addition to releasing the Common Pile v0.1 itself, we also release the
code used in its creation as well as the training mixture and checkpoints for
the Comma v0.1 models.