Toxicity of the Commons: Curating Open-Source Pre-Training Data
Authors: name: Catherine Arnett; name: Eliot Jones; name: Ivan P. Yamshchikov; name: Pierre-Carl Langlais
Published: 2024-10-29 23:00:05

Abstract: Open-source large language models are becoming increasingly available and
popular among researchers and practitioners. While significant progress has
been made on open-weight models, open training data is a practice yet to be
adopted by the leading open-weight models creators. At the same time, there
researchers are working to make language models safer. We propose a data
curation pipeline to reduce harmful outputs by models trained on public domain
data. There are unique challenges to working with public domain data, as these
sources differ from web text in both form and content. Many sources are
historical documents and are the result of Optical Character Recognition (OCR).
Consequently, current state-of-the-art approaches to toxicity filtering are
often infeasible or inappropriate for open data models. In this paper, we
introduce a new fully open-source pipeline for open-data toxicity filtering.
Our contributions are threefold. We create a custom training dataset,
ToxicCommons, which is composed of texts which have been classified across five
different dimensions (racial/origin-based, gender/sex-based, religious,
ability-based discrimination, and violence). We use this dataset to train a
custom classifier, Celadon, that can be used to detect toxic content in open
data more efficiently at a larger scale. Finally, we describe the balanced
approach to content filtration that optimizes safety filtering with respect to
the filtered data available for training.