Institutional Books 1.0: A 242B token dataset from Harvard Library's
  collections, refined for accuracy and usability
Authors: name: Matteo Cargnelutti; name: Catherine Brobston; name: John Hess; name: Jack Cushman; name: Kristi Mukk; name: Aristana Scourtas; name: Kyle Courtney; name: Greg Leppert; name: Amanda Watson; name: Martha Whitehead; name: Jonathan Zittrain
Published: 2025-06-10 00:11:30

Abstract: Large language models (LLMs) use data to learn about the world in order to
produce meaningful correlations and predictions. As such, the nature, scale,
quality, and diversity of the datasets used to train these models, or to
support their work at inference time, have a direct impact on their quality.
The rapid development and adoption of LLMs of varying quality has brought into
focus the scarcity of publicly available, high-quality training data and
revealed an urgent need to ground the stewardship of these datasets in
sustainable practices with clear provenance chains. To that end, this technical
report introduces Institutional Books 1.0, a large collection of public domain
books originally digitized through Harvard Library's participation in the
Google Books project, beginning in 2006. Working with Harvard Library, we
extracted, analyzed, and processed these volumes into an extensively-documented
dataset of historic texts. This analysis covers the entirety of Harvard
Library's collection scanned as part of that project, originally spanning
1,075,899 volumes written in over 250 different languages for a total of
approximately 250 billion tokens. As part of this initial release, the
OCR-extracted text (original and post-processed) as well as the metadata
(bibliographic, source, and generated) of the 983,004 volumes, or 242B tokens,
identified as being in the public domain have been made available. This report
describes this project's goals and methods as well as the results of the
analyses we performed, all in service of making this historical collection more
accessible and easier for humans and machines alike to filter, read and use.