Consent in Crisis: The Rapid Decline of the AI Data Commons
Authors: name: Shayne Longpre, name: Robert Mahari, name: Ariel Lee, name: Campbell Lund, name: Hamidah Oderinwale, name: William Brannon, name: Nayan Saxena, name: Naana Obeng-Marnu, name: Tobin South, name: Cole Hunter, name: Kevin Klyman, name: Christopher Klamm, name: Hailey Schoelkopf, name: Nikhil Singh, name: Manuel Cherep, name: Ahmad Anis, name: An Dinh, name: Caroline Chitongo, name: Da Yin, name: Damien Sileo, name: Deividas Mataciunas, name: Diganta Misra, name: Emad Alghamdi, name: Enrico Shippole, name: Jianguo Zhang, name: Joanna Materzynska, name: Kun Qian, name: Kush Tiwary, name: Lester Miranda, name: Manan Dey, name: Minnie Liang, name: Mohammed Hamdy, name: Niklas Muennighoff, name: Seonghyeon Ye, name: Seungone Kim, name: Shrestha Mohanty, name: Vipul Gupta, name: Vivek Sharma, name: Vu Minh Chien, name: Xuhui Zhou, name: Yizhi Li, name: Caiming Xiong, name: Luis Villa, name: Stella Biderman, name: Hanlin Li, name: Daphne Ippolito, name: Sara Hooker, name: Jad Kabbara, name: Sandy Pentland

General-purpose artificial intelligence (AI) systems are built on massive
swathes of public web data, assembled into corpora such as C4, RefinedWeb, and
Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit
of the consent protocols for the web domains underlying AI training corpora.
Our audit of 14,000 web domains provides an expansive view of crawlable web
data and how consent preferences to use it are changing over time. We observe a
proliferation of AI-specific clauses to limit use, acute differences in
restrictions on AI developers, as well as general inconsistencies between
websites' expressed intentions in their Terms of Service and their robots.txt.
We diagnose these as symptoms of ineffective web protocols, not designed to
cope with the widespread re-purposing of the internet for AI. Our longitudinal
analyses show that in a single year (2023-2024) there has been a rapid
crescendo of data restrictions from web sources, rendering ~5%+ of all tokens
in C4, or 28%+ of the most actively maintained, critical sources in C4, fully
restricted from use. For Terms of Service crawling restrictions, a full 45% of
C4 is now restricted. If respected or enforced, these restrictions are rapidly
biasing the diversity, freshness, and scaling laws for general-purpose AI
systems. We hope to illustrate the emerging crisis in data consent, foreclosing
much of the open web, not only for commercial AI, but non-commercial AI and
academic purposes.