Towards Best Practices for Open Datasets for LLM Training
Authors: name: Stefan Baack, name: Stella Biderman, name: Kasia Odrozek, name: Aviya Skowron, name: Ayah Bdeir, name: Jillian Bommarito, name: Jennifer Ding, name: Maximilian Gahntz, name: Paul Keller, name: Pierre-Carl Langlais, name: Greg Lindahl, name: Sebastian Majstorovic, name: Nik Marda, name: Guilherme Penedo, name: Maarten Van Segbroeck, name: Jennifer Wang, name: Leandro von Werra, name: Mitchell Baker, name: Julie Belião, name: Kasia Chmielinski, name: Marzieh Fadaee, name: Lisa Gutermuth, name: Hynek Kydlíček, name: Greg Leppert, name: EM Lewis-Jong, name: Solana Larsen, name: Shayne Longpre, name: Angela Oduor Lungati, name: Cullen Miller, name: Victor Miller, name: Max Ryabinin, name: Kathleen Siminyu, name: Andrew Strait, name: Mark Surman, name: Anna Tumadóttir, name: Maurice Weber, name: Rebecca Weiss, name: Lee White, name: Thomas Wolf

Many AI companies are training their large language models (LLMs) on data
without the permission of the copyright owners. The permissibility of doing so
varies by jurisdiction: in countries like the EU and Japan, this is allowed
under certain restrictions, while in the United States, the legal landscape is
more ambiguous. Regardless of the legal status, concerns from creative
producers have led to several high-profile copyright lawsuits, and the threat
of litigation is commonly cited as a reason for the recent trend towards
minimizing the information shared about training datasets by both corporate and
public interest actors. This trend in limiting data information causes harm by
hindering transparency, accountability, and innovation in the broader ecosystem
by denying researchers, auditors, and impacted individuals access to the
information needed to understand AI models.
  While this could be mitigated by training language models on open access and
public domain data, at the time of writing, there are no such models (trained
at a meaningful scale) due to the substantial technical and sociological
challenges in assembling the necessary corpus. These challenges include
incomplete and unreliable metadata, the cost and complexity of digitizing
physical records, and the diverse set of legal and technical skills required to
ensure relevance and responsibility in a quickly changing landscape. Building
towards a future where AI systems can be trained on openly licensed data that
is responsibly curated and governed requires collaboration across legal,
technical, and policy domains, along with investments in metadata standards,
digitization, and fostering a culture of openness.