Bridging the Data Provenance Gap Across Text, Speech and Video
Authors: name: Shayne Longpre, name: Nikhil Singh, name: Manuel Cherep, name: Kushagra Tiwary, name: Joanna Materzynska, name: William Brannon, name: Robert Mahari, name: Manan Dey, name: Mohammed Hamdy, name: Nayan Saxena, name: Ahmad Mustafa Anis, name: Emad A. Alghamdi, name: Vu Minh Chien, name: Naana Obeng-Marnu, name: Da Yin, name: Kun Qian, name: Yizhi Li, name: Minnie Liang, name: An Dinh, name: Shrestha Mohanty, name: Deividas Mataciunas, name: Tobin South, name: Jianguo Zhang, name: Ariel N. Lee, name: Campbell S. Lund, name: Christopher Klamm, name: Damien Sileo, name: Diganta Misra, name: Enrico Shippole, name: Kevin Klyman, name: Lester JV Miranda, name: Niklas Muennighoff, name: Seonghyeon Ye, name: Seungone Kim, name: Vipul Gupta, name: Vivek Sharma, name: Xuhui Zhou, name: Caiming Xiong, name: Luis Villa, name: Stella Biderman, name: Alex Pentland, name: Sara Hooker, name: Jad Kabbara

Progress in AI is driven largely by the scale and quality of training data.
Despite this, there is a deficit of empirical analysis examining the attributes
of well-established datasets beyond text. In this work we conduct the largest
and first-of-its-kind longitudinal audit across modalities--popular text,
speech, and video datasets--from their detailed sourcing trends and use
restrictions to their geographical and linguistic representation. Our manual
analysis covers nearly 4000 public datasets between 1990-2024, spanning 608
languages, 798 sources, 659 organizations, and 67 countries. We find that
multimodal machine learning applications have overwhelmingly turned to
web-crawled, synthetic, and social media platforms, such as YouTube, for their
training sets, eclipsing all other sources since 2019. Secondly, tracing the
chain of dataset derivations we find that while less than 33% of datasets are
restrictively licensed, over 80% of the source content in widely-used text,
speech, and video datasets, carry non-commercial restrictions. Finally, counter
to the rising number of languages and geographies represented in public AI
training datasets, our audit demonstrates measures of relative geographical and
multilingual representation have failed to significantly improve their coverage
since 2013. We believe the breadth of our audit enables us to empirically
examine trends in data sourcing, restrictions, and Western-centricity at an
ecosystem-level, and that visibility into these questions are essential to
progress in responsible AI. As a contribution to ongoing improvements in
dataset transparency and responsible use, we release our entire multimodal
audit, allowing practitioners to trace data provenance across text, speech, and
video.