BRIDGING THE DATA PROVENANCE GAP
ACROSS TEXT, SPEECH, AND VIDEO

Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, et al.

► To cite this version:

Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, et al. BRIDGING THE DATA PROVENANCE GAP ACROSS TEXT, SPEECH, AND VIDEO. ICLR 2025, Apr 2025, Singapore (SG), Singapore. hal-05300062

HAL Id: hal-05300062

https://hal.science/hal-05300062v1

Submitted on 6 Oct 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# BRIDGING THE DATA PROVENANCE GAP ACROSS TEXT, SPEECH, AND VIDEO

Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, Nayan Saxena, Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Da Yin, Kun Qian, Yizhi Li, Minnie Liang, An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang, Ariel N. Lee, Campbell S. Lund, Christopher Klamm, Damien Sileo, Diganta Misra, Enrico Shippole, Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Vipul Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong, Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, Jad Kabbara

The Data Provenance Initiative

## ABSTRACT

Progress in AI is driven largely by the scale and quality of training data. Despite this, there is a deficit of empirical analysis examining the attributes of well-established datasets beyond text. In this work we conduct the largest and first-of-its-kind longitudinal audit across modalities—popular text, speech, and video datasets—from their detailed sourcing trends and use restrictions to their geographical and linguistic representation. Our manual analysis covers nearly 4000 public datasets between 1990-2024, spanning 608 languages, 798 sources, 659 organizations, and 67 countries. We find that multimodal machine learning applications have overwhelmingly turned to web-crawled, synthetic, and social media platforms, such as YouTube, for their training sets, eclipsing all other sources since 2019. Secondly, tracing the chain of dataset derivations we find that while less than 33% of datasets are restrictively licensed, over 80% of the source content in widely-used text, speech, and video datasets, carry non-commercial restrictions. Finally, counter to the rising number of languages and geographies represented in public AI training datasets, our audit demonstrates measures of *relative* geographical and multilingual representation have failed to significantly improve their coverage since 2013. We believe the breadth of our audit enables us to empirically examine trends in data sourcing, restrictions, and Western-centricity at an ecosystem-level, and that visibility into these questions are essential to progress in responsible AI. As a contribution to ongoing improvements in dataset transparency and responsible use, we release our entire multimodal audit, allowing practitioners to trace data provenance across text, speech, and video.

## 1 INTRODUCTION

The capabilities and flaws of multimodal foundation models are often directly attributable to their training data (Carlini et al., 2023a; Rando et al., 2022; Carlini et al., 2023b; Parmar et al., 2024; Liu et al., 2023b;a; Dai et al., 2024). While the importance of *data measurement* has been widely established by prior work (Gadre et al., 2024), so has a prevailing absence of data documentation (Gebru et al., 2021; Bender & Friedman, 2018), transparency (Bommasani et al., 2023), and detailed understanding (Dodge et al., 2021; Bandy & Vincent, 2021; Sambasivan et al., 2021)—especially for modalities other than text. A lack of thorough data analysis has led to significant challenges, including privacy issues (Subramani et al., 2023), retracting datasets with harmful content (Birhane et al., 2021; David, 2023), adversarially bypassing safety filters (Rando et al., 2022), facial recognition bias with respect to gender and skin type (Buolamwini & Gebru, 2018a), gender bias in hiring (Chang, 2023), benchmark contamination from overlapping train and test sets (Lee et al., 2023a), and challenges in copyright (Henderson et al., 2023). Understanding data provenance can aid mitigation attempts to
<table><caption>Table 1: We quantify the breadth of our audit, including the total number of datasets (#), their size in tokens or hours, the sources, domains, creator organizations, countries, languages, tasks, and licenses. **In aggregate, we audited 3916 datasets from 659 organizations in 67 countries, spanning 2.1T tokens, and 1.9M hours. We cataloged nearly 798 unique sources, 443 tasks, and 55 licenses.**</caption><thead><tr><th rowspan="2"></th><th colspan="2">DATASETS</th><th colspan="2">SOURCES</th><th colspan="2">CREATOR ORGS</th><th colspan="2">LANGUAGES</th><th rowspan="2">TASKS</th><th rowspan="2">LICENSES</th></tr><tr><th>#</th><th>SIZE</th><th>#</th><th>DOMAINS</th><th>#</th><th>COUNTRIES</th><th>#</th><th>FAMILIES</th></tr></thead><tbody><tr><td>TEXT</td><td>3717</td><td>2.1T</td><td>713</td><td>23</td><td>534</td><td>60</td><td>502</td><td>21</td><td>395</td><td>50</td></tr><tr><td>SPEECH</td><td>95</td><td>775k</td><td>51</td><td>16</td><td>124</td><td>29</td><td>260</td><td>36</td><td>18</td><td>19</td></tr><tr><td>VIDEO</td><td>104</td><td>1.13M</td><td>44</td><td>24</td><td>101</td><td>23</td><td>-</td><td>-</td><td>33</td><td>11</td></tr><tr><td>TOTAL</td><td>3916</td><td>-</td><td>798</td><td>83</td><td>659</td><td>67</td><td>608</td><td>37</td><td>443</td><td>55</td></tr></tbody></table>

reduce model bias and toxicity (Welbl et al., 2021; Pozzobon et al., 2023) address representation in data (Xu et al., 2021), contamination (Elazar et al., 2023), and quality (Kreutzer et al., 2022; Marion et al., 2023), as well as practical challenges with identifying copyright-free and permissively licensed sets (Min et al., 2023).

Despite the urgent need for the provenance and characteristics of widely used datasets, the majority of attention to date has centered on text datasets (Elazar et al., 2023; Longpre et al., 2024b), or a single feature such as prevalence of hate content (Dodge et al., 2021; Birhane et al., 2021). In contrast, in this work, we will critically examine several provenance features of data *across* text, speech, and video. We conduct the largest and most comprehensive multimodal audit of AI data, to date, reviewing nearly 4000 datasets between 1990-2024, covering 443 unique tasks, 608 languages, derived from 798 original sources, and constructed by 659 organizations, spanning 67 countries, over 1T tokens of text, and 1.9M hours of speech and video content (see Table 1).

There is an unprecedented acceleration in the development of multimodal AI systems, making all the more urgent an understanding of the datasets that underpin these breakthroughs. Our extensive collection of features from unstructured academic papers, websites, and repositories enables us to provide empirical grounding to an ambitious set of research questions surrounding data sourcing trends, intended licenses, and geographical and linguistic representation. Our key findings include:

1. **Multimodal data is increasingly sourced from the web, social media platforms, or synthetically generated;** rather than more curated sources such as movies, audiobooks or manually collected. These sources comprise the vast majority of text tokens, as well as speech and video hours in public data. However, while social media platforms provide data scale, heterogeneity and freshness by nature, they are also particularly prone to anti-crawling, copyright, privacy, and factuality concerns.

2. **Whereas only 25% of text, speech, and video datasets have non-commercial licenses, over 80% of content from each modality carries undocumented restrictions in the dataset's sources.** Dataset licenses are inconsistent with their source's restrictions for over 55% of content. Our audit provides the tools for multimodal developers to identify dataset restrictions, and apply their own standards.

3. **Geographical and linguistic representation have not improved for a decade, across the data ecosystem.** While the amount of data from under-represented creators and languages increases each year, to over 600 languages and 60 countries in 2024, their *relative representation* remains consistently western-centric, with no significant improvements from > 0.7 Gini coefficients. While Africa and South America organizations account for < 0.2% of all modality content, North America or European organizations span 93% of text tokens and 60%+ hours of speech and video.

Our work provides critical insights into the landscape of available multimodal data. We release the entire audit, collected data, and analysis tools, which we believe will bring immense value for data creators, developers, and researchers interested in promoting the responsible development of AI systems and analysis of the AI data ecosystem.

## 2 METHODOLOGY

While many prior works have surveyed the dataset ecosystem (Albalak et al., 2024; Liu et al., 2024c; Malik et al., 2021; Prabhavalkar et al., 2023; Li et al., 2019b), few empirically examine data corpora
at scale, and those that do focus present a more narrow focus around a specific feature like geographic bias or hate content (Birhane et al., 2023; McMillan-Major et al., 2022a; Shankar et al., 2017) or a single modality (Dodge et al., 2021; Caswell et al., 2021; Elazar et al., 2023; Longpre et al., 2024b). The goal of this work is to provide an empirical, ecosystem-level, and multimodal analysis of widely used training datasets (Cen et al., 2023). Our audit focuses on text, speech, and video, as prominent data modalities behind modern multimodal systems, such as Sora, Whisper, Gemini, GPT-4, and others (Brooks et al., 2024; Zheng et al., 2024b; Radford et al., 2023; Peng et al., 2023; Team et al., 2023; OpenAI, 2024). Since training data for modalities can often be independent, multimodal models tend to interleave training batches with different combinations of one or two modalities (Aghajanyan et al., 2023). As such, we focus our analysis on datasets that represent one or a pair of these modalities.

**Annotation Features & Methodology** In particular, we analyze data trends for the state of data permissions (licenses and terms), sourcing (the web, human annotation, and synthetic generation), and representation (of tasks, organizations, languages, and countries). We adopt Longpre et al. (2024b)'s methodology, including the license annotation taxonomy and process, to manually audit these features precisely and rigorously. We go beyond prior work, which considers dataset licenses, by extending the taxonomy to consider the terms of use of the sources of the dataset, either from models used to generate synthetic data (e.g. OpenAI's non-compete clause<fnref n="1" /> or Meta's acceptable use policy for Llama 3.1²), or the source's policy on content restrictions, which can be conveyed in the form of a license, terms of use, or content policy on a website (Klyman, 2024). For each dataset, the source terms are annotated as Unrestricted, Unspecified, Source Closed or Model Closed, as defined in Table 2. For Figure 2 we combine Source Closed and Model Closed into *Restricted*.

As with prior work (Longpre et al., 2024b;c), we engage domain experts for these annotation tasks— AI researchers whose work pertains to the modality and topic. Because many datasets are iteratively re-packaged before they appear in their final form and often shared on popular dataset marketplaces like HuggingFace, Papers with Code or Github, prior work has found that relevant licensing terms or sourcing information for AI training data is frequently omitted (Longpre et al., 2024b). To ensure we collect this information, we require a full trace of metadata back to their original sources (sometimes a chain of github repositories, websites, or academic papers). This search can be onerous, especially for terms and licenses, but ensures rigor in the results. Table 1 enumerates the full statistics of our audit. All annotations and analysis code will be made publicly available on release.

**Scope & Dataset Selection** For each modality, we define the scope of the audit (detailed separately below), then aggregate resources to distill a list of relevant datasets. The scope is focused on (a) publicly available datasets, (b) widely used tasks in the context of general-purpose model development, and (c) relevance to generative tasks. However, we do consider classification-based datasets in text, speech, and video that can and are frequently re-purposed for generative uses (e.g. instruction tuning). Within the defined audit scope, we use a mix of the HuggingFace Datasets platform, survey papers, survey repositories, workshop proceedings, and expert review to accumulate relevant datasets. More detail about the dataset selection and collection process is given for each modality below. Each modality requires its own independent process, by virtue of their community dataset ecosystems being unique (discussed in Section 4). Note that text has a wider heterogeneity of published publicly available datasets than speech or video. Typically those datasets have been aggregated into large, standardized text-to-text collections, and as such we trace both these *Text (Collections)* and their constituent *Text (Datasets)*. All datasets are described, linked, and attributed in Appendix D.

## 2.1 TEXT

**Scope** We focus on providing an extensive audit for *post-training* datasets, used in training language models. We include single and multi-turn formats, encompassing both datasets typically used for instruction finetuning (SFT) and preference alignment Rafailov et al. (2023). This scope reflects the prominent role of general-purpose language models, which benefit from multi-task training on heterogeneous collections that span a variety of linguistic, reasoning, and knowledge intensive tasks like question answering, coding, tool use, translation, and classification (Wei et al., 2021; Ouyang et al., 2022).

<footnote n="1">¹OpenAI Terms of Use
²Llama 3.1 Acceptable Use Policy</footnote>
**Dataset Selection** We expand the study conducted by the Data Provenance Collection (Longpre et al., 2024b), from 44 dataset collections (of 1858 supervised text datasets) to a superset of 108 collections of 3717 datasets, prioritizing recent, popular publicly available HuggingFace Datasets introduced between 2022 and April 2024. Our collection sourced popular datasets from recent survey papers (Albalak et al., 2024; Liu et al., 2024c) and tools (Longpre et al., 2024a). We additionally reviewed HuggingFace Datasets' most downloaded datasets every month, from April to July 2024, under the Natural Language Processing category, as well as the SFT/DPO datasets associated with popular open model releases. We also drew from major multilingual data repositories, including the SEACrowd Catalogue (Lovenia et al., 2024), the Masader Arabic Data Catalogue (Alyafeai et al., 2022), AI4Bharat (Kunchukuttan et al., 2020), and the Aya Collection (Singh et al., 2024a). Lastly, our list of datasets was reviewed and supplemented by language model experts to fill in notable omissions. In total, we trace the provenance and features of 3713 text datasets from 108 collections, covering 395 popular tasks, spanning from 1994 to 2024.

## 2.2 SPEECH

**Scope** We audit speech datasets for which automatic speech recognition (ASR) was noted as a primary task. We focus on ASR datasets because: (1) ASR is fundamental to many speech technologies, including dictation tools, voice assistants, and chatbots (Aksënova et al., 2021; Zhang et al., 2022c); (2) large-scale speech datasets are typically designed for ASR (Li et al., 2023b); (3) ASR data follows standardized formats, making comparisons easier (e.g., corpus of audio clips paired with text); and (4) ASR data can often be reused for other tasks like text to speech (TTS) (Ito & Johnson, 2017) or language identification (Ardila et al., 2020b).

**Dataset Selection** To curate a representative sample of popular ASR datasets, we relied on a combination of survey repositories<fnref n="3" />, and HuggingFace Datasets using the “Automatic Speech Recognition” and “Text-to-Speech” task tags. We expanded coverage to well-documented datasets on the OpenSLR⁴ platform, even if they were newer or less widely used. We expect this might reflect datasets that could be adopted more widely in the future. Finally, we included datasets related to low-resource languages and other languages not well-covered by our initial searches. Speech recognition models are increasingly highly multilingual Babu et al. (2021); Radford et al. (2023); Pratap et al. (2024), and datasets serving different communities of builders and end-users around the world are a priority for making speech recognition technologies more inclusive. In total, we trace the provenance and features of 95 speech datasets, covering 18 popular ASR tasks, spanning from 1990 to 2024.

## 2.3 VIDEO

**Scope** Early video understanding models primarily focused on video classification, detection and action recognition, where short clips were categorized into predefined classes (Zheng et al., 2022; Zhu et al., 2020). More advanced tasks such as temporal action segmentation, video question answering, and video captioning were later introduced to build upon these foundational tasks (Moctezuma et al., 2022; Zhu et al., 2023). Recently, following the success in the field of image generation, video generation from text has become a new task that has shown promising results (Brooks et al., 2024; Zheng et al., 2024b; Blattmann et al., 2023; Esser et al., 2023). Given the scarcity of datasets for text-to-video and the often undocumented sources of data used in recent video generation models (Mauran, 2024), we take a broader approach to our collection of video datasets. We focus on annotating popular video tasks and limit our scope to datasets corresponding to video tasks that are either published, highly cited, or have 100+ downloads on HuggingFace. This approach is justified by three key factors: (1) the usefulness of video data to the research community stems from its collection and presentation in peer-reviewed work, (2) datasets can often be repurposed between different tasks, allowing for applicability to new tasks such as video generation from text, and (3) focusing on highly cited datasets ensures that datasets' quality and relevance has been validated by the research community.

**Dataset Selection** We include datasets tagged with “Video Classification”, “Text-to-Video”, and “Video-Text-to-Text” from HuggingFace Datasets. We augmented this with datasets tagged by “Video Understanding” or “Video Generation” in PapersWithCode, as well as datasets listed in a popular Github survey repository. We also consulted the proceedings of recent video workshops: the Large Scale Video Understanding and Egocentric Vision workshops. We separately consulted a committee

<footnote n="3">³The Speech Datasets Collection
⁴openslr.org: Open Speech and Language Resources. OpenSLR is a widely used platform in the speech community, dedicated to hosting resources for speech tasks.</footnote>
of non-author video experts to supplement the list with relevant datasets published at CVPR, ICCV, ECCV, and IJCV. In total, we trace the provenance and features of 104 video datasets, covering 33 popular video tasks, spanning from 2009 to 2024.

# 3 RESULTS

We discuss three key results related to (1) the rising use of web, social media and synthetic sources, (2) inconsistent and opaque restrictions on data use, and (3) a lack of improvement in geographical or linguistic representation. Each of these findings holds across modalities, at the ecosystem level.

## 3.1 RISING USE OF WEB, SOCIAL MEDIA & SYNTHETIC DATA

<figure><img src="image_2.png" /><figcaption>Figure 1: The cumulative size of data (log-scale tokens for text, hours for speech/video) from each source category, across modalities. The source categories in the legend are sorted descending by quantity. **Speech and video sources are increasingly dominated by internet videos and YouTube.** Though text is especially web or encyclopedia (wiki) sources, synthetic text is rising in popularity.</figcaption></figure>

**The need for scale, and heterogeneity have driven rising use of data from web-crawled, social media, and synthetic data sources.** Developers have sought out ever larger and conveniently accessible sources of training data (Hoffmann et al., 2022; Henighan et al., 2020). While small, human-curated datasets are often sufficient and sometimes preferred due to higher quality, these sources often do not scale to present demands (Kaplan et al., 2020; Henighan et al., 2020). In Figure 1, we empirically measure the rising use of web crawling and social media (or “forum”) websites that provide some of the most scalable and fresh content. While web-sourced data was always prominent, the balance of sources becomes much more skewed after 2018—note the use of the y-axis log scale. We find for Speech and Video that by far the most prominent source of data has become internet videos, and specifically YouTube. Nearly 1M hours each of Speech and Video data from this source far outstrips the next most common sources, which comprise less than 100K hours. For Speech, the primary data sources used to be Calling Platforms (pre-2017), content manually collected with Human Participation, and Audiobooks, but since 2018 internet videos have supplanted these other sources. For Video, since 2013, YouTube, synthetic, and general web data sources all constitute a significantly larger portion of data used in prominent video datasets, outstripping the use of Movies, Flickr, Getty, or human curated sources. Among text post-training datasets, we see a similar trend with general or news web-based sources, including encyclopedic sources (mainly Wikipedia), providing the majority of tokens over time. Encyclopedic sources alone now contribute over 1T tokens in total.

**Synthetic data sources are rising the most rapidly.** Within the video modality, the introduction of VidProm (Wang & Yang, 2024a) in 2024, consisting of nearly 7M synthetically generated videos, offered a large shift in the video source distribution. Within the textual modality, from fig. 1, synthetic data represented <0.1% of the quantity of Web Encyclopedia data in 2020, but is now 10% its proportion in 2024, making up the 5th largest source of tokens. The top models used in generating datasets are mainly from OpenAI. The top 5 consist of ChatGPT, version unspecified (15.0% of synthetic datasets), GPT-4 (14.4%), BART (10.1%), GPT-3 (8.3%) and GPT-3.5-Turbo (4.9%). The average synthetic dataset also has notably longer turns (in tokens) than the average natural dataset: 1,756 tokens vs 1,065. The task distribution of textual synthetic datasets is shifted towards longer form, open-generation and creative tasks. For example, 88.1% of natural datasets contain classification tasks, compared to only 66.3% of synthetic datasets. Natural data is also more likely to cover translation than synthetic data (72.4% of datasets vs only 22.9% of synthetic datasets).
<figure><img src="image_3.png" /><figcaption>Figure 2: The distribution of restrictions from dataset *licenses* and their sources' *terms*. We break this down by the count of datasets (top), as well as total tokens or hours (bottom). Each license is categorized as Non-commercial/Academic (NC/Acad), Unspecified, or Commercially licensed. Each dataset may also have terms from the source: Restricted to non-commercial use, Unspecified restrictions, or Unrestricted. **Two main findings across modalities emerge: (1) Commercially licensed datasets represent a larger set of tokens and hours, relative to number of datasets; however, (2) the vast majority of those commercially licensed tokens/hours bare restrictions from their sources.** Tables 3 and 4 in the appendix provide detailed numbers.</figcaption></figure>

## 3.2 INCONSISTENT USE RESTRICTIONS

In the United States, creators of a work automatically have a copyright interest that gives them exclusive rights to make copies and derivatives of the work (17 U.S.C. § 106). Licenses are legal documents through which the owners of a work express how others may use their work. By contrast, Terms of Service are a contract between a platform and its users governing how a platform and its content may be used (Robinson & Zhu, 2020). For simplicity, we use "Licenses" to refer to dataset restrictions, and "Terms" to refer to restrictions on the sources of datasets. There remain open questions about whether certain data licenses are enforceable, but these licenses signal the intention of data creators and therefore warrant consideration as the data creators may be best positioned to understand the sensitivities of the data (privacy, copyright, representation, etc.), and the most impacted by its downstream use (Morton-Park, 2023; Lee et al., 2023b; Mahari & Longpre, 2023; Mahari et al., 2023). How closely practitioners adhere to dataset licenses or source terms remains an open question, and may depend on jurisdiction or the desired model's use cases (Lee et al., 2023b). This work does not propose one standard for all developers. For these reasons we restrict our treatment and discussion here to tracing the lineage and distribution of licenses and terms for a given modality.

**Data source terms are much more restrictive than the dataset's documented license restrictions.** In Figure 2, we find only 25%, 33%, and 32% of text/speech/video datasets are licensed non-commercially. This value is even lower if we consider the proportion of tokens or hours, with 21%, 26%, and 33% of text/speech/video quantities carrying license restrictions. However, a staggering 99.8%, 78%, and 99% of those quantities carry some form of non-commercial restriction on one of their sources. For text, these restrictions are frequently from being generated by OpenAI or other models with a non-compete clause, while for speech and videos this is often since the datasets are derived from web or social media sources.

**Inconsistencies between dataset licenses and their source's restrictions pose challenges to practitioners.** A large amount of datasets have permissive or unspecified licenses, but some set of their sources carry non-commercial restrictions. This inconsistency is measurable—representing 79% of tokens in text datasets, 55% of speech hours, and 65% of video hours. Additionally, 19%, 14%, and 36% of text, speech, and video datasets have no license or intended use documentation
(from our audit of the datasets' documentation on Hugging Face Datasets, GitHub, and Papers with Code). A lack of centralized documentation around these restrictions means it can be misleading to developers who are attempting to source data according to their own legal standards for copyright and privacy. Furthermore, lack of documentation can hamper developers following best practices around data preparation and transparency (Gebru et al., 2021; Bommasani et al., 2023).

**Large quantities of commercially licensed text datasets are locked in collections without clear information to separate them from restrictive datasets.** In Figure 2 (top and bottom), we see the number of datasets and number of tokens *without* restrictions is significantly higher for Text (Datasets) than Text (Collections). Specifically, 60% more Datasets (or 75% more tokens) are commercially licensed, than for Collections. This demonstrates that many collections contain significant amounts of commercially licensed data. While our audit traces licenses for all datasets within a collection, most collections do not aggregate or expose this documentation. As a result, practitioners may be left without easy access to filter for the subsets appropriate for their sourcing standards.

## 3.3 GEOGRAPHICAL & LINGUISTIC REPRESENTATION IS NOT IMPROVING

<figure><img src="image_4.png" /><figcaption>Figure 3: The geographical distribution of countries (world maps) and continents (table) represented by dataset creators. **Despite some differences in European, Russian, and Middle Eastern representation, creators are heavily concentrated in the US, China, and Western Europe, with little to no representation in South America or Africa, across modalities.** The current Gini coefficient for (Text, Speech, Video) = (0.92, 0.86, 0.74), where higher values indicate more concentration.</figcaption></figure>

**The importance and progress of representation in AI training data.** Diversity and representation in training datasets, and among their creators, are widely acknowledged as essential to building AI models that are less biased, more useful, and more equitable (Joshi et al., 2020; Singh et al., 2024a; Üstün et al., 2024; Adelani et al., 2021; 2024; Aakanksha et al., 2024; McMillan-Major et al., 2022b; Porgali et al., 2023; Monfort et al., 2019a; Sigurdsson et al., 2016a). Prior work has measured cultural, ideological, geographic and linguistic imbalances in data (Faisal et al., 2022; Shankar et al., 2017; McMillan-Major et al., 2022a; De Vries et al., 2019; Mahadev & Chakravarti, 2021). These studies have exposed significant flaws, often in the form of bias and discrimination, stemming directly from poor representation in data (Buolamwini & Gebru, 2018b; Birhane et al., 2021). As this problem has now been widely acknowledged for decades, recent efforts have foregrounded sourcing data multilingually and multi-culturally, from native speakers and creators (e.g. ROOTS (Laurençon et al., 2022), the Aya Dataset (Singh et al., 2024a), the SEACrowd Catalogue (Lovenia et al., 2024), the Masader Catalogue (Alyafeai et al., 2022), Common Voice (Ardila et al., 2019), Causal Conversations V2 (Porgali et al., 2023) or Moments in Time (Monfort et al., 2019a)).
**Measuring geographical and linguistic representation.** Naturally, we aim to use our audit to measure the progress of these efforts on geographical and linguistic representation in the AI ecosystem. We measure the progress of two forms of representation: (1) language diversity of text and speech data, and (2) geographical diversity of the creators, in all three modalities. For languages, we use the ISO 639-1 and 639-3 language codes and top-level language families from Glottolog 5.0. In Figure 4(a, c) we display the cumulative sum of unique languages and countries present across all audited datasets, at each time period since 2013. While these measurements illustrate the absolute rise in diversity, we also hope to measure the relative dispersion, or equality of languages and countries in the distribution. In Figure 4(b, d), we use the Gini Index (Wilson, 1914; Atkinson et al., 1970), a traditional measure of statistical dispersion, frequently used to quantify inequality. This allows us to understand if the distributions of languages and creators are more representative of the international community over the last decade, or equally concentrated despite apparent efforts at the margins.

**Inequality in geographical representation remains very high, with few organizations creating datasets from the Global South.** For every dataset, our audit recorded the organizational affiliations of each creator of the dataset.<fnref n="5" /> These organizations were then manually mapped to the country in which they are headquartered. Occasionally, organizations like BigScience, BigCode, or Masakhane have international or continental representation, and were counted as such. In Figure 3, we measure the current state of diversity among these creator organizations—where a Gini coefficient of 1 indicates highest concentration, and lower values more broad representation. Without taking up the normative question of what a truly “fair” score would be, these values provide useful comparisons across modalities and over time. We find that Text dataset developers are particularly homogeneous, with a Gini-coefficient of 0.92; followed by Speech, at 0.86 and Video at 0.74, which remain high, but are meaningfully less concentrated. Figure 3 also illustrates that even this limited diversity is still concentrated in North America, Europe, East Asia, and less so in the Global South.

In Figure 3, we also compare the distribution of datasets, and of tokens or hours by continent. Dataset creators affiliated with African or South American organizations account for fewer than 0.2% of all tokens or hours, in each modality. In contrast, Asian affiliated organizations represent large proportions of the data, particularly for speech (39% of hours, attributed predominantly to YODAS (Li et al., 2023b)). Much of this driven by Chinese, Indian, Russian, and Saudi Arabian creators. Most prominently, the combination of North American and European datasets comprises 93% of text tokens, 61% of speech hours, and 60% of video hours.

**Geographical representation has not significantly improved for over a decade.** In Figure 4(c), we measure the total unique number of countries represented across all dataset creator organizations. While individual creators will have varying ethnic and national affiliation, we treat this as an estimate for the influence of each locale in dataset development. We find that while the number of represented countries has risen steadily each year, for each modality, this represents only an illusion of progress. Empirically, the Gini coefficient for each modality has not significantly changed since the start of the period we examine in 2013. Geographic diversity has increased only among Video datasets, and these increases are not significant at the *p* = 0.05 level. Text and Speech geographical representations appear to remain stable over the last decade of AI development.

**Multilingual representation has not improved by most measures.** Similar to geographical representation, we measure the cumulative number of ISO 639-1 languages and language families over time, as well as the per-modality Gini-coefficient. Figure 4(a) shows significant increases in the number of languages available for speech and text, especially in 2019, and 2023, with the introduction of large sets like Flores (Goyal et al., 2022), xP3x (Muennighoff et al., 2023), Common Voice (Ardila et al., 2019), and the Aya Collection (Singh et al., 2024a). However, once again, when measuring the cumulative dispersion of these datasets in Figure 4(b), only Text language families demonstrate any improvement from pre-2013 to the present. Improvements in the Gini coefficient appear to be largely driven by individual large-scale projects like xP3x and Common Voice, both introduced in 2019. Subsequently, newer datasets remain predominantly monolingual, causing measures of concentration in text languages, speech languages, and language families to remain consistently high.

**Academia, research non-profits, and industry labs continue to drive public dataset development.** We manually categorize the organizations creating popular datasets into: Academic Organization

<footnote n="5">⁵A dataset creator, following (Longpre et al., 2024b), is defined as an organization associated with the release of the dataset as created for machine learning—not any of the upstream sources. More details in Appendix D.</footnote>
<figure><img src="image_5.png" /><figcaption>(a) Total Language Representation</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Gini-coefficient for Language Representation</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) Total Geographical Representation</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(d) Gini-coefficient for Geographical Representation</figcaption></figure>

Figure 4: The cumulative totals (left) of languages and countries represented in the data over time, and the 95% confidence intervals of the gini-coefficients over time (right) to measure the representativeness of these variables. Gini-coefficients are a measure of statistical dispersion, frequently used to quantify inequality. A Gini coefficient of 1 indicates highest concentration, and lower values more broad representation. **While the number of represented languages and geographies continue to rise (left), the equality of their distribution has in most cases, not significantly changed.**

<figure><img src="image_9.png" /></figure>

Figure 5: The distribution of creator organizations by modality. **Most public speech and video datasets are developed by academic organizations, whereas text datasets are developed by a wide mix of academia, non-profit or industry labs, as well as startups.**

(e.g., universities), Research Groups (e.g., non-profits like BigScience, EleutherAI and AI2), Industry Labs (e.g., Cohere for AI, Google DeepMind), Corporations (e.g. Google, Meta), Startups (e.g., OpenAI, Anthropic), Governments, Unspecified (owner affiliation not shared), and Other. When a dataset is released in collaboration between organizations, we record each organization. In Figure 5, we find that universities and other academic organizations account for 16%, 47%, and 71% of all recorded dataset releases, across Text, Speech, and Video respectively. Research groups, industry labs and even corporations are also significant contributors, especially for Text datasets, where ecosystem contributors are far more distributed. Academic organizations' greater role in Video and Speech may suggest special impediments (e.g., privacy-related) to commercial releases of these datasets.

# 4 DISCUSSION

**The rise of web, social media, and synthetic data may pose greater privacy, copyright, and bias risks.** Section 3.1 discusses the rise of web content and particularly social media as primary sources for speech and video. Figure 1 shows these sources now exceed by at least an order of magnitude more traditional, curated sources like movies, audiobooks, radio, TV, or data produced by human participants. Their mostly user-generated content makes these websites a natural source for the quantity, freshness, and heterogeneity needed to train general-purpose models (Longpre et al.,
2023; Aghajanyan et al., 2023). However, prior work suggests that user-generated web data is also more challenging to use than curated data, particularly for privacy, copyright, bias, and factuality. Web-based and particularly user-generated content is disproportionately likely to include personally identifiable information (PII) Luccioni & Viviano (2021); Subramani et al. (2023); Elazar et al. (2023), and copyrighted content (Meese & Hagedorn, 2019; Lee et al., 2023b). These can be reproduced in the outputs of AI models (Carlini et al., 2022; Chen et al., 2023c), creating privacy and copyright concerns (Zhang et al., 2023a). Open datasets being used to train GPAI often attempt to filter—but frequently miss—PII and copyrighted data (Soldaini et al., 2024; Subramani et al., 2023) (although not all do (Penedo et al., 2023)). Social media, in particular, is also known to have bias, toxicity and factuality issues (Olteanu et al., 2019), which can manifest in trained models, even after alignment (Kotha et al., 2023). Lastly, while synthetic data can help reduce the prevalence of PII, copyright, or bias in data, it comes with its own challenges (Kurakin et al., 2023; Liu et al., 2024a).

**Social media sites are a very prominent data source, but their terms often restrict crawling or commercial use.** We find 71% of Video and 69% of Speech data to be from YouTube, whose scale, freshness, and multimodality (containing videos, speech, images, and text) have made it a prominent data source (Abu-El-Haija et al., 2016b; Aytar et al., 2018; Chang et al., 2020; Uthus et al., 2023; Coats, 2023; Li et al., 2023b). However, YouTube is a social media platform owned by Google and its Terms of Service<fnref n="6" /> prohibit third-party crawling. While content creators retain ownership of material uploaded to YouTube, the YouTube terms of service also grant Google a license to reproduce, modify, display, and use the content for YouTube's "business" (which may include building machine learning models), and forbid crawling by third parties, even if the copyright holder has selected a permissive license. Model developers like Nvidia and OpenAI have been sued in the U.S. by content creators who allege that they unlawfully trained on YouTube videos (Cole, 2024; Skolnik, 2024). Large social media sites like Reddit and StackOverflow⁷ have also recently adopted restrictive terms, with access being restricted just as these data sources become critical to scaling AI systems. The enforceability of these licenses and terms, however, is an open legal question beyond the scope of our work.

**Ambiguous and poorly documented use restrictions may significantly inhibit model developers adhering to cautious legal and ethical data sourcing standards.** In Section 3.2, we find that a significant amount of data carry non-commercial restrictions in their sources, rather than on the final dataset, which can contain no license or a permissive one. For text and video, these restrictions can reach 99% of all tokens and hours. These inconsistencies are the result of datasets being iteratively re-packaged and re-licensed, without carrying on documentation (Longpre et al., 2024b). While not every developer will employ the same filtering standards, our work shows that separating and identifying appropriate datasets remains difficult across these modalities. Without continued audits and documentation, practitioners may have to either take on avoidable risk or forgo large collections of partially viable data, hampering data scaling laws (Kaplan et al., 2020). We hope our audit will help practitioners apply their own standards and make informed decisions on training data use.

**The limitations of measures of geographical and linguistic representation.** Measures of geographic and linguistic representation are imperfect: We have only partial information about the developers' identities (including for privacy reasons), limited transparency into how frequently these datasets are used, and cannot say how far proprietary datasets may fill in representation gaps behind closed doors. Nonetheless, we believe the breadth and rigour of the audit make this the best available empirical measure of representation in *public* datasets. Indeed, we contend that it is necessary to measure representation in AI data to understand progress, or its absence, towards AI systems that fairly serve the broad user community. Figures 3 and 4 show that despite the absolute rise of geographic and linguistic representation, a western-centric skew persists across thousands of surveyed datasets. We release all audit materials for transparency, replicability, and further research use.

**Conducting representative analyses of an ecosystem comes with assumptions.** The AI ecosystem is decentralized. Text datasets, for example, are often hosted on HuggingFace, unlike Speech and Video. Similarly, while Text data is frequently repackaged for general-purpose post-training, this is less true of other modalities. Scoping and dataset selection thus need to be modality-specific, rather than a single general protocol unable to capture these nuances. Similarly, we studied modalities of interest to foundation model development (Brooks et al., 2024; Radford et al., 2023), but many others remain for future work (e.g., images, 3D shapes, tabular, time series, graphs, and geospatial data).

<footnote n="6">⁶YouTube Terms of Service.
⁷Reddit User Agreement and StackOverflow Terms of Service.</footnote>
## ACKNOWLEDGMENTS

This research was conducted by the Data Provenance Initiative, a collective of independent and academic researchers volunteering their time to data transparency projects. The Data Provenance Initiative is supported by the Mozilla Data Futures Lab Infrastructure Fund.

## A EXTENDED RELATED WORK

Progress in machine learning across modalities from speech (Radford et al., 2023) to vision (Dosovitskiy et al., 2021) to text (Brown et al., 2020a; Wei et al., 2021) has benefited from advancements in large pre-training and fine-tuning corpora. The development of multimodal corpora has also been key to several recent advances, as with CLIP in the image/text domain Radford et al. (2021), CLAP for audio/text settings Elizalde et al. (2022), and a number of other models involving both text and images, audio or video (Radford et al., 2023; Ramirez et al., 2024; Singer et al., 2022; Ramesh et al., 2022).

The datasets powering these advances are not, however, always well-documented, despite the existence of standards and frameworks for recording and annotating dataset metadata that range from ‘data statements’ (Bender & Friedman, 2018) to ‘datasheets for datasets’ (Gebru et al., 2021) and others (Mitchell et al., 2019). The key problem is not a deficiency of any particular framework, but rather inconsistent adoption and fragmentation (Longpre et al., 2024d). Much prior work has argued for the need to document and audit these datasets (Rogers, 2021; Paullada et al., 2021), motivated by concerns from reproducibility (Kapoor & Narayanan, 2022) to interpretability (Longpre et al., 2023) to bias and fairness problems that may stem from problematic content in training data (Birhane et al., 2021).

There have been several attempts to carry out such audits, with prior work examining pretraining data (Longpre et al., 2024c), general web corpora (Gao et al., 2020; Dodge et al., 2021), instruction fine-tuning datasets (Longpre et al., 2024b), and the documentation fields of the HuggingFace Datasets platform in particular (Yang et al., 2024). For speech and vision, there has been less work, with many discussions of datasets in the aggregate occurring in survey papers (Schiappa et al., 2023; Chaquet et al., 2013), research aimed directly at improving model performance Gadre et al. (2023) or close examinations of questions like bias in small groups of datasets (Buolamwini & Gebru, 2018b; Romanou et al., 2024).

Prior work has also examined the identities, affiliations and national origin of paper authors (Movva et al., 2024) in AI, but an analogous look at the producers of datasets is lacking. We aim to carry out such analyses: replicating those for pretraining and text finetuning datasets in video and audio domains, and surveying provenance and legal status. Finally, there has also been significant recent attention to legal questions in the collection and use of AI training data (Sag, 2020; Henderson et al., 2023). The complex process involved in preparing these datasets (Lee et al., 2023b), and the ambiguous licensing of inputs, can make understanding the legal status of the final output quite difficult.

## B DATASET LICENSES & TERMS

**Detailed taxonomy** We code the legal restrictions placed on use of datasets along two axes. First, we identify whether a dataset’s license permits commercial use (“Commercial” in Table 3), only non-commercial / academic use (“NC / Acad”), or does not clearly specify what is permitted (“Unspecified”). The latter category includes datasets for which we were unable to locate a license. Datasets which are in the public domain and not subject to a license are counted as commercially usable. Second, we annotate the contractual or terms-of-use restrictions placed on dataset use by the source of each dataset. There are four levels, defined in Table 3. Note that the Model Closed status can only apply to datasets that are AI-generated, at least in part. Some datasets can carry both Model Closed and Source Closed status, but we count the Model Closed first for simplicity.

**Detailed breakdown** Tables 3 and 4 present crosstabs of these two dimensions, according to respectively the total amount of content and the number of datasets. The most notable finding, as discussed in the main text, is the frequency of clashing restriction status between licenses and terms.
<table><caption>Table 2: **The taxonomy used to determine use restrictions on each dataset source.** Each source in a dataset is examined and fit into one of these categories. The dataset Terms are then labelled according to the strictest terms across the sources, with Model Closed and Source Closed considered stricter than Unspecified which is in turn stricter than Unrestricted.</caption><thead><tr><th>LABEL</th><th>DEFINITION</th></tr></thead><tbody><tr><td>MODEL CLOSED</td><td>A model used to generate part or all of the dataset prohibits using its outputs commercially, to develop a competing AI model, or in general.</td></tr><tr><td>SOURCE CLOSED</td><td>The source has a license or terms that prohibits use of the data, either commercially, from being crawled, to develop AI, or in general.</td></tr><tr><td>UNSPECIFIED</td><td>No information can be found relevant to restrictions, or lack thereof, for this source.</td></tr><tr><td>UNRESTRICTED</td><td>The source has a commercially permissive license, such as CC BY, or explicitly states the data is open for broad use.</td></tr></tbody></table>

By amount of content, fully 73.0% of text content, 55.0% of speech content, and 21.6% of video content is subject to a license permitting commercial use but also to terms restrictions forbidding it, or the reverse. The absolute level of restrictions is also high, with < 0.1% of text content, 5.4% of speech content, and 0.6% of video content usable for commercial purposes under both licenses and terms.

<table><caption>Table 3: **A breakdown of the percentage of license and terms restrictions across datasets**, by total tokens or hours of content. The much higher frequency of restrictions at the collection level is because we consider a collection's license or terms status to be the most restrictive of those for its datasets. Note that percentages may not add to exactly 100% because of rounding.</caption><thead><tr><th>LICENSE / TERMS</th><th>RESTRICTED</th><th>UNSPECIFIED</th><th>UNRESTRICTED</th><th>TOTAL</th></tr></thead><tbody><tr><td colspan="5"><strong>Text Collections</strong></td></tr><tr><td>NC/ACAD</td><td>96.0</td><td>0.0</td><td>0.0</td><td>96.0</td></tr><tr><td>UNSPECIFIED</td><td>2.3</td><td>0.1</td><td>0.0</td><td>2.4</td></tr><tr><td>COMMERCIAL</td><td>1.5</td><td>0.0</td><td>0.0</td><td>1.6</td></tr><tr><td>TOTAL</td><td>99.8</td><td>0.1</td><td>0.1</td><td></td></tr><tr><td colspan="5"><strong>Text Datasets</strong></td></tr><tr><td>NC/ACAD</td><td>21.1</td><td>0.0</td><td>0.0</td><td>21.2</td></tr><tr><td>UNSPECIFIED</td><td>5.7</td><td>0.1</td><td>0.0</td><td>5.7</td></tr><tr><td>COMMERCIAL</td><td>73.0</td><td>0.0</td><td>0.0</td><td>73.1</td></tr><tr><td>TOTAL</td><td>99.8</td><td>0.1</td><td>0.1</td><td></td></tr><tr><td colspan="5"><strong>Speech Datasets</strong></td></tr><tr><td>NC/ACAD</td><td>23.9</td><td>1.4</td><td>0.8</td><td>26.2</td></tr><tr><td>UNSPECIFIED</td><td>0.5</td><td>0.0</td><td>0.4</td><td>0.9</td></tr><tr><td>COMMERCIAL</td><td>54.2</td><td>13.3</td><td>5.4</td><td>73.0</td></tr><tr><td>TOTAL</td><td>78.6</td><td>14.7</td><td>6.7</td><td></td></tr><tr><td colspan="5"><strong>Video Datasets</strong></td></tr><tr><td>NC/ACAD</td><td>33.7</td><td>0.0</td><td>0.1</td><td>33.8</td></tr><tr><td>UNSPECIFIED</td><td>43.9</td><td>0.1</td><td>0.1</td><td>44.1</td></tr><tr><td>COMMERCIAL</td><td>21.5</td><td>0.0</td><td>0.6</td><td>22.1</td></tr><tr><td>TOTAL</td><td>99.1</td><td>0.1</td><td>0.8</td><td></td></tr></tbody></table>
<table><caption>Table 4: A breakdown of the percentage of license and terms restrictions by dataset count. The much higher frequency of restrictions at the collection level is because we consider a collection's license or terms status to be the most restrictive of those for its datasets. Note that percentages may not add to exactly 100% because of rounding.</caption><thead><tr><th>LICENSE / TERMS</th><th>RESTRICTED</th><th>UNSPECIFIED</th><th>UNRESTRICTED</th><th>TOTAL</th></tr></thead><tbody><tr><td colspan="5"><strong>Text Collections</strong></td></tr><tr><td>NC/ACAD</td><td>84.5</td><td>0.0</td><td>0.3</td><td>84.8</td></tr><tr><td>UNSPECIFIED</td><td>1.5</td><td>7.5</td><td>0.0</td><td>8.9</td></tr><tr><td>COMMERCIAL</td><td>1.5</td><td>0.2</td><td>4.5</td><td>6.3</td></tr><tr><td>TOTAL</td><td>87.5</td><td>7.7</td><td>4.8</td><td></td></tr><tr><td colspan="5"><strong>Text Datasets</strong></td></tr><tr><td>NC/ACAD</td><td>25.0</td><td>0.0</td><td>0.3</td><td>25.3</td></tr><tr><td>UNSPECIFIED</td><td>17.3</td><td>1.2</td><td>0.0</td><td>18.5</td></tr><tr><td>COMMERCIAL</td><td>45.2</td><td>6.5</td><td>4.5</td><td>56.2</td></tr><tr><td>TOTAL</td><td>87.5</td><td>7.7</td><td>4.8</td><td></td></tr><tr><td colspan="5"><strong>Speech Datasets</strong></td></tr><tr><td>NC/ACAD</td><td>9.5</td><td>9.5</td><td>13.7</td><td>32.6</td></tr><tr><td>UNSPECIFIED</td><td>6.3</td><td>0.0</td><td>7.4</td><td>13.7</td></tr><tr><td>COMMERCIAL</td><td>7.4</td><td>18.9</td><td>27.4</td><td>53.7</td></tr><tr><td>TOTAL</td><td>23.2</td><td>28.4</td><td>48.4</td><td></td></tr><tr><td colspan="5"><strong>Video Datasets</strong></td></tr><tr><td>NC/ACAD</td><td>22.1</td><td>0.0</td><td>9.6</td><td>31.7</td></tr><tr><td>UNSPECIFIED</td><td>23.1</td><td>1.0</td><td>11.5</td><td>35.6</td></tr><tr><td>COMMERCIAL</td><td>25.0</td><td>0.0</td><td>7.7</td><td>32.7</td></tr><tr><td>TOTAL</td><td>70.2</td><td>1.0</td><td>28.8</td><td></td></tr></tbody></table>

## C ADDITIONAL RESULTS

Figures 6 and 7 report the size distributions of the datasets. We measure size differently for different types of datasets: Text datasets are in tokens, and audio/video in hours of content. The lack of standard tokenization or preprocessing schemes for those modalities makes it simplest to report raw dataset size.

Notably, we find quite different size distributions by modality. The distribution of dataset sizes has the thickest right tail for text, followed by speech and then by video. Most video datasets are short in hour terms, with speech datasets tending to be somewhat longer and text datasets having a greater prevalence of both very small and very large datasets relative to the mean size.

Dataset tasks, meanwhile, reflect traditional approaches and research programs for each modality. Classification is the most common task for both text and video, with the video community's long-standing interest in captioning also visible in its role as the second most common task for video datasets. Q&A occupies a similar role for text, though text datasets have a more balanced distribution over other, increasingly prominent tasks like generation and reasoning. Given our selection criteria, all datasets for speech are for ASR tasks, but other tasks like speaker identification and translation are also represented.

## D DATASETS

This section provides a detailed overview of the datasets we have collected and analyzed. Table 5 summarizes the text datasets, Table 6 the audio datasets, and Table 7 the video datasets. Each of these
<figure><img src="image_10.png" /><figcaption>Figure 6: The distribution of dataset sizes for each modality. Most text data collections are between 100M-1B tokens. Speech datasets average 100-1k hours, and video datasets are usually the smallest, commonly less than 100 hours.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 7: The task distribution of datasets, across modalities. Post-training text and video datasets are predominantly based on classification. For text, generation and reasoning are rising categories. All speech datasets are recognition-based, particularly for speaker, language, or in the process of translation.</figcaption></figure>

tables lists broad collections of data, sorted in chronological order, and provides information about their properties, sizes, sources and permissions. Each collection can include multiple datasets, and they generally reflect the ways dataset creators have grouped their datasets (such as in the same paper). Because of the large number of datasets, we provide detailed information about their licenses and original published papers, where applicable, in the supplementary Attribution Card in Appendix F.

**Annotation Details: Text** For post-training text datasets it is common to package many together as collections, such as Flan (Wei et al., 2021) or P3 (Sanh et al., 2021). This practice is not common to the same extent for speech or video datasets. For much of the text analysis, where possible, we chose to analyze statistics at the collection-level, since practitioners are more likely to adopt a collection for general-purpose post-training, than an individual dataset within the collection. Also, in dataset-level statistics, metadata for a single collection with many datasets can get repeated and overwhelm the statistics unfairly (e.g. the dataset aggregator/creator being repeated hundreds of times). Consequently, our collection-level analysis of the text modality is reflected in Figure 1, Figure 3, Figure 5, Figure 4, Figure 7, and Figure 6. However, for Figure 2 we draw the distinction between collection and dataset metrics, as practitioners may wish to unpack collections to extract only commercially licensed data. In that case a Collection inherits the most restrictive license and terms of its constituent datasets.

For annotating creator organizations, we follow prior work's instructions (Longpre et al., 2024b). For each dataset they record the affiliations listed on the academic paper or GitHub or HuggingFace object in which the dataset was released. This does not include the organizations who created or owned the sources from which the data was derived. For instance, the SQuAD dataset (Rajpurkar et al., 2016a) would be associated with Stanford (the authors' affiliation), but not Wikipedia, which the data was partially derived from. For a dataset that has authors affiliated with multiple organizations, the dataset will be counted towards each organization.

**Annotation Details: Speech** In many cases, multiple versions of a dataset exist due to datasets being expanded or updated. In these scenarios, we used the release date from the initial version (since release dates for subsequent versions were not always clear), but used metadata from the most recently released version for which information was available to offer an overview of the current landscape of data. However, if the dataset versions could not be meaningfully aggregated (e.g. different licenses),
or did not appear to be cumulatively designed (non-overlapping or otherwise semantically disjoint
data), we maintained separate records. We kept only datasets for which ASR was noted as a primary
task. For example, if a dataset was primarily intended for text-to-speech or speaker recognition, we
did not keep it even if it could conceivably be repurposed for ASR. When computing hours, we
excluded any hours without supervisory transcripts/scripts (unlabeled data), but kept hours with
“weak supervision” (e.g. model-generated transcripts from speech audio). We recognize the difficulty
in comprehensively covering all relevant datasets.

**Annotation Details: Video** In video, a single dataset can be re-purposed and annotated to address different tasks Monfort et al. (2019a; 2021a). We consider these as two different datasets even if they have the same video source since now they can be used for different computer vision tasks.
<table><caption>**Table 5: Alignment tuning (text) collections and properties.** Collection properties include numbers of datasets, tasks, languages, and text domains. The SOURCE column indicates whether a collection contains human-generated web text (), language model outputs («) or both («). The USE column indicates whether a collection includes data freely usable even for commercial purposes (●), data usable only for noncommercial purposes or academic research (●) and data whose license status is not specified precisely enough to allow us to determine commercial use permissions (●). Note that each collection may have different datasets with one, two, or all three of these statuses. Finally, the OAI column indicates collections which include OpenAI model generations. Datasets are sorted chronologically to highlight trends over time.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="4">PROPERTY COUNTS</th><th rowspan="2">TYPES<br>SOURCE</th><th colspan="2">PERMISSIONS</th></tr><tr><th>DATASETS</th><th>TASKS</th><th>LANGS</th><th>DOMAINS</th><th>USE</th><th>OAI</th></tr></thead><tbody><tr><td>RiddleSense</td><td>2021</td><td>1</td><td>3</td><td>1</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>MathInstr.</td><td>2023</td><td>1</td><td>3</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>No Robots</td><td>2023</td><td>1</td><td>8</td><td>1</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>Nectar</td><td>2023</td><td>1</td><td>1</td><td>1</td><td>2</td><td> </td><td>●</td><td>✓</td></tr><tr><td>MetaMathQA</td><td>2023</td><td>8</td><td>2</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>MegaWika</td><td>2023</td><td>50</td><td>1</td><td>50</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>MedInstr.</td><td>2023</td><td>1</td><td>1</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>MathDial</td><td>2023</td><td>1</td><td>2</td><td>1</td><td>4</td><td> </td><td>●</td><td>✓</td></tr><tr><td>PII-Masking-200k</td><td>2023</td><td>1</td><td>2</td><td>4</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>Pure-Dove</td><td>2023</td><td>1</td><td>4</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>LMSYS-Chat-1M</td><td>2023</td><td>1</td><td>9</td><td>5</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>PygmalionAI-PIPPA</td><td>2023</td><td>1</td><td>3</td><td>1</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>HelpSteer</td><td>2023</td><td>1</td><td>5</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>SeaBench</td><td>2023</td><td>9</td><td>4</td><td>9</td><td>5</td><td> </td><td>●</td><td></td></tr><tr><td>Open Asst. v2</td><td>2023</td><td>19</td><td>4</td><td>19</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>Feedback Coll.</td><td>2023</td><td>1</td><td>2</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Glaive Code Asst.</td><td>2023</td><td>1</td><td>2</td><td>2</td><td>1</td><td> </td><td>●</td><td></td></tr><tr><td>EverythingLM</td><td>2023</td><td>1</td><td>8</td><td>2</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Bactrian-X</td><td>2023</td><td>6</td><td>4</td><td>6</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>COBRA Frames</td><td>2023</td><td>1</td><td>1</td><td>1</td><td>2</td><td> </td><td>●</td><td>✓</td></tr><tr><td>UltraFeedback Argilla</td><td>2023</td><td>9</td><td>16</td><td>1</td><td>20</td><td> </td><td>●</td><td>✓</td></tr><tr><td>ExpertQA</td><td>2023</td><td>1</td><td>3</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>ChatDoctor</td><td>2023</td><td>3</td><td>1</td><td>1</td><td>2</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Capybara</td><td>2023</td><td>11</td><td>17</td><td>2</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>UltraChat-200k</td><td>2023</td><td>1</td><td>7</td><td>1</td><td>2</td><td> </td><td>●</td><td>✓</td></tr><tr><td>CollectiveCognition</td><td>2023</td><td>1</td><td>6</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Thai Gen AI</td><td>2023</td><td>9</td><td>11</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Deita 10K</td><td>2023</td><td>2</td><td>11</td><td>1</td><td>3</td><td> </td><td>●</td><td>✓</td></tr><tr><td>SelFee</td><td>2023</td><td>1</td><td>5</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>ChatbotArena</td><td>2023</td><td>1</td><td>4</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>OpenGPT Healthcare</td><td>2023</td><td>3</td><td>4</td><td>1</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Orca-Math</td><td>2024</td><td>1</td><td>1</td><td>1</td><td>3</td><td> </td><td>●</td><td>✓</td></tr><tr><td>OpenMathInstr.-1</td><td>2024</td><td>2</td><td>3</td><td>1</td><td>3</td><td> </td><td>●</td><td>✓</td></tr><tr><td>WildChat</td><td>2024</td><td>2</td><td>7</td><td>10</td><td>1</td><td> </td><td>●</td><td>✓</td></tr><tr><td>Magpie-Pro</td><td>2024</td><td>1</td><td>9</td><td>1</td><td>1</td><td> </td><td>●</td><td></td></tr></tbody></table>

Continued on next page
<table><caption>Table 5: Alignment tuning (text) collections and properties.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="4">PROPERTY COUNTS</th><th rowspan="2">TYPES<br>SOURCE</th><th colspan="2">PERMISSIONS</th></tr><tr><th>DATASETS</th><th>TASKS</th><th>LANGS</th><th>DOMAINS</th><th>USE</th><th>OAI</th></tr></thead><tbody><tr><td>10k Prompt Ranked</td><td>2024</td><td>1</td><td>13</td><td>1</td><td>4</td><td></td><td>●</td><td>✓</td></tr><tr><td>Synth.-GSM8K-Refl.</td><td>2024</td><td>1</td><td>3</td><td>1</td><td>1</td><td>Ξ</td><td>●</td><td></td></tr><tr><td>LongAlign-10k</td><td>2024</td><td>1</td><td>3</td><td>1</td><td>1</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Llama2-MedTuned-Instr.</td><td>2024</td><td>1</td><td>4</td><td>1</td><td>1</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>KIWI</td><td>2024</td><td>1</td><td>1</td><td>1</td><td>2</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Indic-Instr.</td><td>2024</td><td>8</td><td>7</td><td>2</td><td>3</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Gretel Text-to-SQL</td><td>2024</td><td>1</td><td>1</td><td>3</td><td>1</td><td>Ξ</td><td>●</td><td></td></tr><tr><td>Conifer</td><td>2024</td><td>1</td><td>8</td><td>1</td><td>2</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Cidar</td><td>2024</td><td>1</td><td>8</td><td>1</td><td>1</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Aya</td><td>2024</td><td>71</td><td>7</td><td>71</td><td>1</td><td>Ξ</td><td>●</td><td></td></tr><tr><td>Reasoning</td><td>2024</td><td>1</td><td>4</td><td>1</td><td>1</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>AgentInstruct</td><td>Mult.</td><td>6</td><td>3</td><td>1</td><td>7</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>InstAr</td><td>Mult.</td><td>24</td><td>13</td><td>1</td><td>9</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Dynosaur</td><td>Mult.</td><td>1k</td><td>21</td><td>1</td><td>22</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Medical Meadow</td><td>Mult.</td><td>8</td><td>2</td><td>1</td><td>3</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>Open-Platypus</td><td>Mult.</td><td>10</td><td>10</td><td>36</td><td>8</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>PMC-LLaMA Instr.</td><td>Mult.</td><td>7</td><td>1</td><td>1</td><td>2</td><td>Ξ</td><td>●</td><td>✓</td></tr><tr><td>COIG</td><td>Mult.</td><td>18</td><td>13</td><td>2</td><td>22</td><td>Ξ</td><td>●</td><td></td></tr><tr><td>DialogStudio</td><td>Mult.</td><td>83</td><td>3</td><td>5</td><td>3</td><td>Ξ</td><td>●</td><td></td></tr></tbody></table>

<table><caption>Table 6: **Audio collections and properties.** Collection properties include numbers of audio hours (HR), speakers (SPKR), languages (LANG), creator institutions (CREAT), tasks (TASKS), data sources (SRC), and topics (TOPICS). The number of datasets is not listed because all collections include only one dataset, except for M2ASR which has four. The US column indicates datasets from or partly from the United States, the AC column datasets created by academic institutions, and the IND column datasets created by industry. Note that a dataset can have all of these, none of them, or any combination of them. The USE column indicates whether a collection includes data freely usable even for commercial purposes (●), data usable only for noncommercial purposes or academic research (●) and data whose license status is not specified precisely enough to allow us to determine commercial use permissions (●). Note that each collection may have different datasets with one, two, or all three of these statuses. Datasets are sorted chronologically to highlight trends over time.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="7">PROPERTY COUNTS</th><th rowspan="2">CATEGORY<br>US AC IND</th><th rowspan="2">PERM<br>USE</th></tr><tr><th>HR</th><th>SPKR</th><th>LANG</th><th>CREAT</th><th>TASKS</th><th>SRC</th><th>TOP</th></tr></thead><tbody><tr><td>TIMIT</td><td>1990</td><td>5</td><td>630</td><td>1</td><td>3</td><td>3</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Switchboard</td><td>1992</td><td>250</td><td>543</td><td>1</td><td>1</td><td>1</td><td>1</td><td>70</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>African Acc. French</td><td>2003</td><td>22</td><td>232</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td></td><td>●</td></tr><tr><td>CSJ</td><td>2003</td><td>661</td><td>1k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td></td><td></td><td></td></tr><tr><td>Fisher</td><td>2004</td><td>2k</td><td>12k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>36</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>CSLU 22 Langs.</td><td>2005</td><td>84</td><td>-</td><td>21</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>AMI</td><td>2005</td><td>100</td><td>-</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>CSLU 1.2</td><td>2007</td><td>25</td><td>5k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>ALLSSTAR</td><td>2010</td><td>86</td><td>140</td><td>27</td><td>1</td><td>1</td><td>1</td><td>3</td><td>✓</td><td>✓</td><td>●</td></tr></tbody></table>

Continued on next page
<table><caption>Table 6: Audio collections and properties.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="7">PROPERTY COUNTS</th><th colspan="3">CATEGORY</th><th rowspan="2">PERM USE</th></tr><tr><th>HR</th><th>SPKR</th><th>LANG</th><th>CREAT</th><th>TASKS</th><th>SRC</th><th>TOP</th><th>US</th><th>AC</th><th>IND</th></tr></thead><tbody><tr><td>TED-LIUM3</td><td>2012</td><td>452</td><td>2k</td><td>1</td><td>2</td><td>2</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>NST Norwegian</td><td>2013</td><td>540</td><td>870</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td></td><td></td><td></td><td>●</td></tr><tr><td>NST Danish</td><td>2013</td><td>500</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td></td><td></td><td></td><td>●</td></tr><tr><td>NST Swedish</td><td>2013</td><td>300</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td></td><td></td><td></td><td>●</td></tr><tr><td>Vystadial</td><td>2014</td><td>56</td><td>-</td><td>2</td><td>1</td><td>1</td><td>2</td><td>3</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>THCHS-30</td><td>2015</td><td>35</td><td>40</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>LibriSpeech</td><td>2015</td><td>1k</td><td>2k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>106</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>THUYG-20</td><td>2015</td><td>20</td><td>371</td><td>1</td><td>2</td><td>2</td><td>1</td><td>3</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>VCTK</td><td>2016</td><td>44</td><td>110</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Spoken Wikipedia</td><td>2016</td><td>1k</td><td>960</td><td>3</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>AISHELL-1</td><td>2017</td><td>520</td><td>400</td><td>1</td><td>2</td><td>2</td><td>2</td><td>11</td><td></td><td></td><td>✓</td><td>●</td></tr><tr><td>LJSpeech</td><td>2017</td><td>24</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>ClarinPL</td><td>2017</td><td>56</td><td>317</td><td>1</td><td>1</td><td>1</td><td>2</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>AISHELL-2</td><td>2018</td><td>1k</td><td>2k</td><td>1</td><td>2</td><td>2</td><td>1</td><td>8</td><td></td><td></td><td>✓</td><td>●</td></tr><tr><td>Regional Af. Am. Lang.</td><td>2018</td><td>159</td><td>222</td><td>1</td><td>1</td><td>1</td><td>1</td><td>8</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Crowd Sourced Speech</td><td>2018</td><td>1k</td><td>3k</td><td>5</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Zeroth-Korean</td><td>2018</td><td>96</td><td>181</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td></td><td></td><td></td><td>●</td></tr><tr><td>RTVE</td><td>2018</td><td>691</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>OpenSTT</td><td>2019</td><td>20k</td><td>-</td><td>1</td><td>2</td><td>2</td><td>2</td><td>6</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>MuST-C</td><td>2019</td><td>4k</td><td>2k</td><td>16</td><td>2</td><td>2</td><td>1</td><td>4</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>M-AILABS</td><td>2019</td><td>1k</td><td>-</td><td>8</td><td>1</td><td>1</td><td>1</td><td>33</td><td></td><td></td><td></td><td>●</td></tr><tr><td>MAGICDATA</td><td>2019</td><td>755</td><td>1k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Common Voice 17</td><td>2019</td><td>31k</td><td>330k</td><td>124</td><td>3</td><td>3</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>CoNASE</td><td>2019</td><td>154k</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>6</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Nigerian English</td><td>2019</td><td>6</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Norwegian Parl. Speech</td><td>2019</td><td>140</td><td>309</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td></td><td></td><td></td><td>●</td></tr><tr><td>120h Spanish Speech</td><td>2019</td><td>120</td><td>17</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td></td><td></td><td></td><td>●</td></tr><tr><td>DiDiSpeech</td><td>2020</td><td>800</td><td>6k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Czech Parliament</td><td>2020</td><td>444</td><td>212</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>CoVoST-2</td><td>2020</td><td>3k</td><td>78k</td><td>22</td><td>1</td><td>1</td><td>2</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>KSC</td><td>2020</td><td>332</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Basq., Cat. and Gal.</td><td>2020</td><td>34</td><td>132</td><td>3</td><td>1</td><td>1</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>KsponSpeech</td><td>2020</td><td>969</td><td>2k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>6</td><td></td><td></td><td></td><td>●</td></tr><tr><td>Samromur</td><td>2020</td><td>145</td><td>8k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Multiling. LibriSpeech</td><td>2020</td><td>50k</td><td>6k</td><td>8</td><td>1</td><td>1</td><td>1</td><td>33</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>MaSS</td><td>2020</td><td>160</td><td>-</td><td>8</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>FT SPEECH</td><td>2020</td><td>2k</td><td>434</td><td>1</td><td>2</td><td>2</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Eng. Acc. in Brit. Isles</td><td>2020</td><td>31</td><td>120</td><td>1</td><td>1</td><td>1</td><td>1</td><td>4</td><td></td><td></td><td></td><td>●</td></tr><tr><td>Highland Puebla Nahuatl</td><td>2021</td><td>156</td><td>-</td><td>1</td><td>3</td><td>3</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>QASR</td><td>2021</td><td>2k</td><td>11k</td><td>1</td><td>2</td><td>2</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Multiling. TEDx</td><td>2021</td><td>765</td><td>-</td><td>9</td><td>3</td><td>3</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Minds14</td><td>2021</td><td>25</td><td>-</td><td>14</td><td>1</td><td>1</td><td>2</td><td>7</td><td>✓</td><td>✓</td><td></td><td>●</td></tr><tr><td>Golos</td><td>2021</td><td>1k</td><td>-</td><td>1</td><td>3</td><td>3</td><td>1</td><td>6</td><td>✓</td><td>✓</td><td></td><td>●</td></tr></tbody><tfoot><tr><td colspan="14">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 6: Audio collections and properties.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="8">PROPERTY COUNTS</th><th colspan="3">CATEGORY</th><th rowspan="2">PERM USE</th></tr><tr><th>HR</th><th>SPKR</th><th>LANG</th><th>CREAT</th><th>TASKS</th><th>SRC</th><th>TOP</th><th>US</th><th>AC</th><th>IND</th></tr></thead><tbody><tr><td>MASC</td><td>2021</td><td>1k</td><td>14k</td><td>1</td><td>3</td><td>3</td><td>1</td><td>15</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>LaboroTVSpeech</td><td>2021</td><td>2k</td><td>-</td><td>2</td><td>2</td><td>2</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>KeSpeech</td><td>2021</td><td>2k</td><td>27k</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>JTUBESPEECH</td><td>2021</td><td>1k</td><td>-</td><td>2</td><td>4</td><td>4</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>GigaSpeech</td><td>2021</td><td>10k</td><td>-</td><td>1</td><td>9</td><td>9</td><td>3</td><td>24</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>VoxPopuli</td><td>2021</td><td>2k</td><td>4k</td><td>16</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>SPGISpeech</td><td>2021</td><td>5k</td><td>50k</td><td>1</td><td>4</td><td>4</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>West Afr. Radio</td><td>2021</td><td>142</td><td>-</td><td>10</td><td>2</td><td>2</td><td>1</td><td>3</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>AISHELL-4</td><td>2021</td><td>120</td><td>61</td><td>1</td><td>4</td><td>4</td><td>2</td><td>6</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>West Afr. Virt. Asst.</td><td>2021</td><td>2</td><td>49</td><td>3</td><td>2</td><td>2</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>MediaSpeech</td><td>2021</td><td>40</td><td>-</td><td>4</td><td>5</td><td>5</td><td>12</td><td>1</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>People's Speech</td><td>2021</td><td>30k</td><td>-</td><td>1</td><td>7</td><td>7</td><td>2</td><td>14</td><td>✓</td><td>✓</td><td>✓</td><td>●</td></tr><tr><td>1111 Hours Hindi</td><td>2022</td><td>108</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Shrutilipi</td><td>2022</td><td>6k</td><td>-</td><td>12</td><td>2</td><td>2</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>WenetSpeech</td><td>2022</td><td>10k</td><td>-</td><td>1</td><td>4</td><td>4</td><td>2</td><td>10</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Samromur Children</td><td>2022</td><td>131</td><td>3k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>SDS-200</td><td>2022</td><td>200</td><td>4k</td><td>1</td><td>3</td><td>3</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>aidatatang</td><td>2022</td><td>200</td><td>600</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Fleurs</td><td>2022</td><td>1k</td><td>-</td><td>102</td><td>3</td><td>3</td><td>1</td><td>11</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>OLKAVS</td><td>2022</td><td>1k</td><td>1k</td><td>1</td><td>2</td><td>2</td><td>1</td><td>14</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Norwegian Parl.</td><td>2022</td><td>140</td><td>267</td><td>1</td><td>2</td><td>2</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>MagicData-RAMC</td><td>2022</td><td>180</td><td>663</td><td>1</td><td>4</td><td>4</td><td>1</td><td>15</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Kathbath</td><td>2022</td><td>2k</td><td>1k</td><td>12</td><td>2</td><td>2</td><td>1</td><td>3</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Hebrew Kan</td><td>2022</td><td>9</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>3</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Hebrew Coursera</td><td>2022</td><td>36</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Bloom Speech</td><td>2022</td><td>428</td><td>-</td><td>56</td><td>5</td><td>5</td><td>1</td><td>8</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>English-Vietnamese</td><td>2022</td><td>508</td><td>-</td><td>2</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Earnings-22</td><td>2022</td><td>119</td><td>125</td><td>1</td><td>1</td><td>1</td><td>3</td><td>2</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>YODAS</td><td>2023</td><td>370k</td><td>-</td><td>149</td><td>3</td><td>3</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>AFRISPEECH-200</td><td>2023</td><td>200</td><td>2k</td><td>20</td><td>14</td><td>14</td><td>1</td><td>6</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Aalto Finnish Parl.</td><td>2023</td><td>3k</td><td>449</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>ReazonSpeech</td><td>2023</td><td>35k</td><td>-</td><td>1</td><td>2</td><td>2</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>EdAcc</td><td>2023</td><td>40</td><td>120</td><td>1</td><td>1</td><td>1</td><td>1</td><td>8</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>RixVox</td><td>2023</td><td>5k</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Japanese Anime Speech</td><td>2023</td><td>110</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>7</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Snow Mountain</td><td>2023</td><td>273</td><td>11</td><td>14</td><td>2</td><td>2</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Samromur Milljon</td><td>2023</td><td>967</td><td>17k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>Bud500</td><td>2024</td><td>500</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>VibraVox</td><td>2024</td><td>18</td><td>200</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr><tr><td>M2ASR</td><td>Mult.</td><td>448</td><td>655</td><td>4</td><td>3</td><td>3</td><td>1</td><td>9</td><td>✓</td><td>✓</td><td>●</td><td>●</td></tr></tbody></table>
<table><caption>Table 7: Video collections and properties. Collection properties include numbers of hours of video, datasets, creator institutions, countries of creator institutions, and data sources. The USE column indicates whether a collection includes data freely usable even for commercial purposes (●), data usable only for noncommercial purposes or academic research (●) and data whose license status is not specified precisely enough to allow us to determine commercial use permissions (●). Note that each collection may have different datasets with one, two, or all three of these statuses. Finally, the AVAIL column indicates whether a dataset is available online (✔) or has been taken down, usually for legal reasons (✗). Datasets are sorted chronologically to highlight trends over time.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="5">PROPERTY COUNTS</th><th colspan="2">PERMISSIONS</th></tr><tr><th>HOURS</th><th>DATASETS</th><th>COUNTRIES</th><th>CREATORS</th><th>SOURCES</th><th>USE</th><th>AVAIL</th></tr></thead><tbody><tr><td>HOLLYWOOD2</td><td>2009</td><td>20</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Collective</td><td>2009</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>HMDB</td><td>2011</td><td>7k</td><td>1</td><td>2</td><td>3</td><td>5</td><td>●</td><td>✓</td></tr><tr><td>UCF101</td><td>2012</td><td>26</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>YouCook</td><td>2013</td><td>1k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>50 Salads</td><td>2013</td><td>40</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>StoryGraphs</td><td>2014</td><td>7</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Hollywood Ext.</td><td>2014</td><td>9</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Breakfast</td><td>2014</td><td>77</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Sports-1M</td><td>2014</td><td>106k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>THUMOS</td><td>2014</td><td>254</td><td>1</td><td>2</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>VideoStory</td><td>2014</td><td>743</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>SumMe</td><td>2014</td><td>1</td><td>1</td><td>2</td><td>3</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>TVSum</td><td>2015</td><td>4</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Volleyball</td><td>2015</td><td>-</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>ActivityNet</td><td>2015</td><td>849</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MovieQA</td><td>2015</td><td>381</td><td>1</td><td>3</td><td>3</td><td>1</td><td>●</td><td>✗</td></tr><tr><td>Mars</td><td>2016</td><td>-</td><td>1</td><td>1</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>NTU RGB+D</td><td>2016</td><td>74</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MSR-VTT</td><td>2016</td><td>41</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Charades</td><td>2016</td><td>82</td><td>1</td><td>2</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>VTW</td><td>2016</td><td>213</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Youtube-8M</td><td>2016</td><td>350k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Narrated Instr. Vid.</td><td>2016</td><td>7</td><td>1</td><td>2</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>TGIF</td><td>2016</td><td>86</td><td>1</td><td>1</td><td>3</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MultiTHUMOS</td><td>2017</td><td>30</td><td>1</td><td>2</td><td>3</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>ImageNet-Vid</td><td>2017</td><td>9</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>PKU-MMD</td><td>2017</td><td>50</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>20BN-SOMETHING</td><td>2017</td><td>121</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>YouCook2</td><td>2017</td><td>176</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>VoxCeleb</td><td>2017</td><td>2k</td><td>1</td><td>2</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Davis</td><td>2017</td><td>-</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>QFVS</td><td>2017</td><td>20</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>DiDeMo</td><td>2018</td><td>275</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>SOA</td><td>2018</td><td>2k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Charades-Ego</td><td>2018</td><td>69</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>EPIC-KITCHENS</td><td>2018</td><td>100</td><td>1</td><td>3</td><td>3</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MovieGraphs</td><td>2018</td><td>94</td><td>1</td><td>1</td><td>3</td><td>1</td><td>●</td><td>✗</td></tr><tr><td>How2</td><td>2018</td><td>2k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr></tbody><tfoot><tr><td colspan="9" style="text-align: right;">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 7: Video collections and properties.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="5">PROPERTY COUNTS</th><th colspan="2">PERMISSIONS</th></tr><tr><th>HOURS</th><th>DATASETS</th><th>COUNTRIES</th><th>CREATORS</th><th>SOURCES</th><th>USE</th><th>AVAIL</th></tr></thead><tbody><tr><td>VLOG</td><td>2018</td><td>336</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>VaTeX</td><td>2019</td><td>115</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>20BN-jester</td><td>2019</td><td>13</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>HowTo100M</td><td>2019</td><td>134k</td><td>1</td><td>2</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>COIN</td><td>2019</td><td>476</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MMAct</td><td>2019</td><td>100</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>HACS</td><td>2019</td><td>833</td><td>1</td><td>1</td><td>3</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>CrossTask</td><td>2019</td><td>376</td><td>1</td><td>4</td><td>5</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Moments in Time</td><td>2019</td><td>833</td><td>1</td><td>1</td><td>1</td><td>11</td><td>●</td><td>✓</td></tr><tr><td>TRECVid</td><td>2019</td><td>1k</td><td>1</td><td>1</td><td>1</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>MSA</td><td>2019</td><td>516</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Toyota Smarthome</td><td>2019</td><td>269</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>TITAN</td><td>2020</td><td>3</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>VIOLIN</td><td>2020</td><td>582</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>RareAct</td><td>2020</td><td>21</td><td>1</td><td>3</td><td>5</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>TinyVIRAT</td><td>2020</td><td>11</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>100DOH</td><td>2020</td><td>5k</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Oops!</td><td>2020</td><td>50</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>OmniSource-Web</td><td>2020</td><td>13k</td><td>1</td><td>1</td><td>1</td><td>3</td><td>●</td><td>✓</td></tr><tr><td>Condensed Movies</td><td>2020</td><td>1k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MovieScenes</td><td>2020</td><td>250</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>EEV</td><td>2020</td><td>370</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Movie-Net</td><td>2020</td><td>3k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>FineGym</td><td>2020</td><td>708</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>HAA500</td><td>2020</td><td>5</td><td>1</td><td>2</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>LEMMA</td><td>2020</td><td>11</td><td>1</td><td>1</td><td>1</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>HVU</td><td>2020</td><td>96k</td><td>1</td><td>3</td><td>5</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Apes</td><td>2021</td><td>36</td><td>1</td><td>3</td><td>3</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>WebVid</td><td>2021</td><td>13k</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>X</td></tr><tr><td>VideoLT</td><td>2021</td><td>14k</td><td>1</td><td>2</td><td>4</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>HOMAGE</td><td>2021</td><td>30</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>UAV-Human</td><td>2021</td><td>18</td><td>1</td><td>2</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>HD-VILA-100M</td><td>2021</td><td>372</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>M-MiT</td><td>2021</td><td>833</td><td>1</td><td>1</td><td>1</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>Mimetics</td><td>2021</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Spoken Moments</td><td>2021</td><td>417</td><td>1</td><td>1</td><td>3</td><td>11</td><td>●</td><td>✓</td></tr><tr><td>QuerYD</td><td>2021</td><td>207</td><td>1</td><td>1</td><td>1</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>MAD</td><td>2022</td><td>1k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>FERV39k</td><td>2022</td><td>16</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>CDAD</td><td>2022</td><td>215</td><td>1</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MVBench</td><td>2023</td><td>-</td><td>1</td><td>1</td><td>6</td><td>12</td><td>●</td><td>✓</td></tr><tr><td>VidProm</td><td>2024</td><td>240k</td><td>1</td><td>2</td><td>2</td><td>5</td><td>●</td><td>✓</td></tr><tr><td>ShareGPT4Video</td><td>2024</td><td>3k</td><td>1</td><td>1</td><td>4</td><td>5</td><td>●</td><td>✓</td></tr><tr><td>OpenVid-1M</td><td>2024</td><td>52k</td><td>1</td><td>1</td><td>3</td><td>5</td><td>●</td><td>✓</td></tr><tr><td>FineVideo</td><td>2024</td><td>3k</td><td>1</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Disney Vid. Gen.</td><td>2024</td><td>7</td><td>1</td><td>1</td><td>-</td><td>2</td><td>●</td><td>✓</td></tr></tbody><tfoot><tr><td colspan="9" style="text-align: right;">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 7: Video collections and properties.</caption><thead><tr><th rowspan="2">COLLECTION</th><th rowspan="2">YEAR</th><th colspan="5">PROPERTY COUNTS</th><th colspan="2">PERMISSIONS</th></tr><tr><th>HOURS</th><th>DATASETS</th><th>COUNTRIES</th><th>CREATORS</th><th>SOURCES</th><th>USE</th><th>AVAIL</th></tr></thead><tbody><tr><td>Kinetics</td><td>Mult.</td><td>4k</td><td>3</td><td>1</td><td>1</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>Ego4D</td><td>Mult.</td><td>5k</td><td>2</td><td>1</td><td>2</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>MPII</td><td>Mult.</td><td>110</td><td>3</td><td>1</td><td>2</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>Project-Aria</td><td>Mult.</td><td>1k</td><td>2</td><td>1</td><td>1</td><td>1</td><td>●</td><td>✓</td></tr><tr><td>Ava</td><td>Mult.</td><td>146</td><td>2</td><td>1</td><td>1</td><td>2</td><td>●</td><td>✓</td></tr><tr><td>LSMDC</td><td>Mult.</td><td>316</td><td>2</td><td>4</td><td>10</td><td>1</td><td>●</td><td>✓</td></tr></tbody></table>

## E CONTRIBUTIONS

Here we break down contributions to this work. Contributors are listed alphabetically, except for team leads who are placed first.

* **Text Datasets** Shayne Longpre (lead), Jad Kabbara (lead), Ahmad Anis, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Kun Qian, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Nayan Saxena, Niklas Muennighoff, Naana Obeng-Marnu, Robert Mahari, Seonghyeon Ye, Seungone Kim, Shayne Longpre, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, William Brannon, Xuhui Zhou, Yizhi Li, An Dinh, Caroline Chitongo, Christopher Klamm, Da Yin, Damien Sileo, Ariel Lee

* **Reviewing Text Dataset Metadata** Jad Kabbara (lead), Shayne Longpre (lead), Robert Mahari, Damien Sileo, Niklas Muennighoff, William Brannon,

* **Data Explorer Features** Shayne Longpre (lead), Christopher Klamm, Vu Minh Chien,

* **Speech Datasets** Nikhil Singh (lead), Manuel Cherep (lead), An Dinh, Minnie Liang, Shrestha Mohanty

* **Video Datasets** Kush Tiwary (lead), Joanna Materzynska (lead), Vivek Sharma, Shayne Longpre, Robert Mahari, Jad Kabbara, William Brannon, Tobin South, Shrestha Mohanty, Nikhil Singh, Manuel Cherep

* **Data Analysis** Shayne Longpre (lead), Nikhil Singh (lead), Manuel Cherep (lead), Kush Tiwary (lead), Joanna Materzynska (lead), Naana Obeng-Marnu (lead), William Brannon (lead),

* **Writing** Shayne Longpre (lead), Jad Kabbara (lead), Nikhil Singh, Manuel Cherep, Kush Tiwary, Joanna Materzynska, Robert Mahari

* **Legal Analysis** Robert Mahari (lead), Luis Villa

* **Visualizations & Visual Data Analysis** Nikhil Singh (lead), Manuel Cherep (lead), Kush Tiwary (lead), Joanna Materzynska (lead), Naana Obeng-Marnu (lead), William Brannon (lead), Shayne Longpre (lead), Ariel Lee, Hamidah Oderinwale, Campbell Lund

* **Senior Advisors** Stella Biderman, Sara Hooker, Jad Kabbara, Sandy Pentland, Luis Villa, Caiming Xiong

## F ATTRIBUTION CARD

Here we provide detailed information about the licenses of each data collection and its constituent datasets, and cite all of the papers (455 in all) which introduced datasets we consider. Text datasets are laid out in Table 8, audio datasets in Table 9, and video datasets in Table 10. Because of the large number of references, we include a second bibliography after the tables (named 'Attribution Card References'), with numbered citations in this section referring to that second bibliography.
<table><caption>Table 8: **References and licenses for alignment-tuning (text) dataset collections** presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>10k Prompt Ranked<br>AgentInstruct</td><td>Unspecified<br>Unspecified, CC BY 4.0, MIT Li-<br>cense</td><td>–<br>Shridhar et al. (2021); Yao et al.<br>(2023); Liu et al. (2023c); Zeng et al.<br>(2023); Deng et al. (2023)</td></tr><tr><td>Aya</td><td>Apache License 2.0</td><td>Singh et al. (2024b)</td></tr><tr><td>Bactrian-X</td><td>CC BY-SA 3.0, CC BY-NC 4.0</td><td>Li et al. (2023a)</td></tr><tr><td>COBRA Frames</td><td>BigScience OpenRAIL-M</td><td>Zhou et al. (2023b)</td></tr><tr><td>COIG</td><td>Various</td><td>Zhang et al. (2023b); Bai et al.<br>(2024a)</td></tr><tr><td>Capybara</td><td>Various</td><td>–</td></tr><tr><td>ChatDoctor</td><td>Unspecified</td><td>Li et al. (2023d)</td></tr><tr><td>ChatbotArena</td><td>CC BY 4.0, CC BY-NC 4.0</td><td>Zheng et al. (2023)</td></tr><tr><td>Cidar</td><td>CC BY-NC 4.0</td><td>Alyafeai et al. (2024)</td></tr><tr><td>CollectiveCognition</td><td>MIT License</td><td>–</td></tr><tr><td>Conifer</td><td>Apache License 2.0</td><td>Sun et al. (2024)</td></tr><tr><td>Deita 10K</td><td>Apache License 2.0, CC BY-NC 4.0</td><td>Liu et al. (2024b)</td></tr></tbody></table>

Continued on next page
<table><caption>Table 8: **References and licenses for alignment-tuning (text) dataset collections** presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>DialogStudio</td><td>Various</td><td>Chen et al. (2021a); Wei et al. (2018); Lin et al. (2021b); Chawla et al. (2021); He et al. (2018); Mrk-sic et al. (2017); Qian et al. (2022); Liu et al. (2021); El Asri et al. (2017); Quan et al. (2019); Chen et al. (2019; 2022b); Eric & Man-ning (2017); Zang et al. (2020); Sha-lyminov et al. (2019); Martin et al. (2020); Peskov et al. (2019); Eric et al. (2019); Moon et al. (2019); Rastogi et al. (2020); Mosig et al. (2020); Chiu et al. (2022); Shah et al. (2018); Byrne et al. (2019); Mrkšić & Vulić (2018); Shang et al. (2018); Rameshkumar & Bailey (2020); Fabbri et al. (2021); Chen et al. (2021c); Mukherjee et al. (2022); Shang et al. (2018); Zhu et al. (2021); Zhong et al. (2021); Gliwa et al. (2019); Chen et al. (2022a); Feigenblat et al. (2021); Li et al. (2019c); Dinan et al. (2019a); Rashkin et al. (2019); Bai et al. (2022); Chen et al. (2023a); Kim et al. (2022); Myers et al. (2020); Reddy et al. (2019); Yu et al. (2019a); Talmor & Berant (2018); Nan et al. (2021; 2022); Gu et al. (2021); Chen et al. (2020b); Gupta et al. (2018a); Li et al. (2021a); Talmor et al. (2021); Yu et al. (2019c); Iyyer et al. (2017); Yu et al. (2019b); Parikh et al. (2020); Yih et al. (2016); Zhong et al. (2017); Pasupat & Liang (2015); Komeili et al. (2022); Dinan et al. (2019b); Hemphill et al. (1990); Casanueva et al. (2020); Zhang et al. (2022b); Larson et al. (2019); Rastogi et al. (2020); Liu et al. (2019; 2013); Coope et al. (2020); Coucke et al. (2018); Gupta et al. (2018b)</td></tr></tbody><tfoot><tr><td colspan="3">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 8: References and licenses for alignment-tuning (text) dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>Dynosaur</td><td>Various</td><td>Adlakha et al. (2022); Agarwal et al. (2021); Akyürek et al. (2022); Amini et al. (2019); Ardanuy et al. (2020); Austin et al. (2021); Azer-bayev et al. (2023); Bai et al. (2022); Bajaj et al. (2018); Balakr-ishnan et al. (2019); Bartolo et al. (2020); Bisk et al. (2019); Boratko et al. (2020); Botha et al. (2018); Boudin & Gallina (2021); Bravo et al. (2015); Brown et al. (2020b); Byrne et al. (2019); Cao & Wang (2021); Cao et al. (2022); Casanueva et al. (2020); Cetoli et al. (2019); Chalkidis et al. (2019b;a; 2021); Chan et al. (2022); Chapuis et al. (2021); Chen et al. (2020a); Cheng et al. (2022); Chouldechova (2017); Christmann et al. (2019); Clark et al. (2019; 2018); Cobbe et al. (2021); Coucke et al. (2018); Dankers et al. (2022); Dasigi et al. (2019); De-varaj et al. (2021); DeYoung et al. (2021); Diggelmann et al. (2021); Emelin et al. (2020); Fabbri et al. (2019); Faruqui & Das (2018); Feng et al. (2021); Gallina et al. (2019); Ganesan et al. (2010); Gazzola et al. (2019); George & Mamidi (2019); Geva et al. (2019); Gliwa et al. (2019); Gorrell et al. (2018); Gu et al. (2022); Gupta et al. (2021); Ha & Eck (2017); Haagsma et al. (2020); Hazoom et al. (2021); Hen-derson et al. (2022); Hendrycks et al. (2021); Huang et al. (2019; 2021); Huang (2022); Irwin et al. (2020); Ivgi et al. (2022); Iyer et al. (2017); Jiang et al. (2020; 2021); Jin et al. (2019); Joshi et al. (2017); Juraska et al. (2019); Jurczyk et al. (2016); Kanade et al. (2020); Kaushik et al. (2020); Khot et al. (2020; 2018); Kim et al. (2018); Kornilova & Ei-delman (2019); Kury et al. (2020); Lai et al. (2017); Lake & Baroni (2018); Lebret et al. (2016); Lewis et al. (2017); Li et al. (2022; 2019a); Lin et al. (2020a)</td></tr></tbody><tfoot><tr><td colspan="3">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 8: References and licenses for alignment-tuning (text) dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>Dynosaur (cont'd)</td><td>Various</td><td>Lin et al. (2020b; 2019; 2022); Ling et al. (2017); Liu et al. (2019); Louis et al. (2020); Lowe et al. (2016); Malo et al. (2013); Martin et al. (2018); Merity et al. (2016); Mihaylov et al. (2018); Mishra et al. (2023); Moniz & Torgo (2018); Mostafazadeh et al. (2020); Nan et al. (2021); Narayan et al. (2018); Nguyen et al. (2021); Nie et al. (2020); Novikova et al. (2017); Paik et al. (2021); Pakhomov et al. (2010); Pang & Lee (2005); Pavlichenko et al. (2021); Pedersen et al. (2007); Perez-Beltrachini et al. (2019); Petroni et al. (2019); Pham et al. (2023); Rajani et al. (2019); Rajpurkar et al. (2016b); Rameshkar & Bailey (2020); Rashkin et al. (2019); Rastogi et al. (2020); Royer et al. (2018); Rush et al. (2015); Rust et al. (2023); Saeidi et al. (2018); Saha et al. (2018); Sakaguchi et al. (2019); Sanh et al. (2022); Sap et al. (2019); Schulz et al. (2020); See et al. (2017); Sharma et al. (2019); Shriberg et al. (1998); Sileo & Moens (2023); Soleimani et al. (2021); Stolcke et al. (2000); Tafjord et al. (2019; 2018); Talmor et al. (2019); Tandon et al. (2019); Tang et al. (2020); Thawani et al. (2021); Thorne et al. (2018); Tyleček & Šára (2013); Ullrich et al. (2023); Ushio et al. (2023); Wang et al. (2022a; 2020c; 2019; 2023a); Warstadt et al. (2023); Welbl et al. (2018; 2017); Weller et al. (2020); Weston et al. (2015); Williams et al. (2020); Wu et al. (2018); Xiong et al. (2019a); Yang et al. (2018); Yu et al. (2019b); Zellers et al. (2019); Zhang et al. (2016; 2023c; 2019); Zhou et al. (2019; 2023a); Zhu et al. (2022)</td></tr><tr><td>EverythingLM</td><td>MIT License</td><td>-</td></tr><tr><td>ExpertQA</td><td>MIT License</td><td>Malaviya et al. (2024)</td></tr><tr><td>Feedback Coll.</td><td>MIT License</td><td>Kim et al. (2024)</td></tr></tbody></table>

Continued on next page
<table><caption>Table 8: References and licenses for alignment-tuning (text) dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>Glaive Code Asst.</td><td>Apache License 2.0</td><td>–</td></tr><tr><td>Gretel Text-to-SQL</td><td>Apache License 2.0</td><td>–</td></tr><tr><td>HelpSteer</td><td>CC BY 4.0</td><td>Wang et al. (2023b)</td></tr><tr><td>Indic-Instr.</td><td>Various</td><td>Gala et al. (2024)</td></tr><tr><td>InstAr</td><td>Various</td><td>Hu et al. (2020); Einea et al. (2019); Mozannar et al. (2019); Pratapa et al. (2022); Chouikhi et al. (2024); Ab-bas et al. (2011); Abdelghany et al. (2020); Orabi et al. (2020); ElSa-har & El-Beltagy (2015); Elnagar & Einea (2016); Pieri et al. (2024); Al-ghamdi et al. (2022); Abdallah et al. (2024); Biltawi et al. (2020); Aloui et al. (2024); El-khair (2016)</td></tr><tr><td>KIWI</td><td>CC BY-SA 4.0</td><td>Xu et al. (2024a)</td></tr><tr><td>LMSYS-Chat-1M</td><td>LMSYS-Chat-1M Dataset License, Anthropic, Llama 2</td><td>Zheng et al. (2024a)</td></tr><tr><td>Llama2-MedTuned-Instr.</td><td>CC BY-NC 4.0</td><td>Rohanian et al. (2023)</td></tr><tr><td>LongAlign-10k</td><td>Anthropic, Apache License 2.0</td><td>Bai et al. (2024b)</td></tr><tr><td>Magpie-Pro</td><td>Meta Llama3 Community License</td><td>Xu et al. (2024b)</td></tr><tr><td>MathDial</td><td>CC BY-SA 4.0, MIT License</td><td>Macina et al. (2023)</td></tr><tr><td>MathInstr.</td><td>MIT License</td><td>Yue et al. (2023)</td></tr><tr><td>MedInstr.</td><td>Unspecified</td><td>Zhang et al. (2024)</td></tr><tr><td>Medical Meadow</td><td>Various</td><td>Han et al. (2023); Wang et al. (2020b); Jin et al. (2020); Savery et al. (2020)</td></tr><tr><td>MegaWika</td><td>CC BY-SA 4.0</td><td>Barham et al. (2023)</td></tr><tr><td>MetaMathQA</td><td>MIT License</td><td>Yu et al. (2023)</td></tr><tr><td>Nectar</td><td>Various</td><td>–</td></tr><tr><td>No Robots</td><td>CC BY-NC 4.0</td><td>–</td></tr><tr><td>Open Asst. v2</td><td>Apache License 2.0</td><td>Köpf et al. (2023)</td></tr><tr><td>Open-Platypus</td><td>Various</td><td>Sawada et al. (2023); Dettmers et al. (2023); Lightman et al. (2023); Yu et al. (2020); Wang et al. (2024); Lu et al. (2022); Chen et al. (2023b)</td></tr><tr><td>OpenGPT Healthcare</td><td>Unspecified, OGL 3.0</td><td>–</td></tr><tr><td>OpenMathInstr.-1</td><td>Custom, MIT License, Apache License 2.0</td><td>Toshniwal et al. (2024)</td></tr><tr><td>Orca-Math</td><td>Various</td><td>Mitra et al. (2024)</td></tr><tr><td>PII-Masking-200k</td><td>Non Commercial</td><td>–</td></tr><tr><td>PMC-LLaMA Instr.</td><td>Unspecified, Apache License 2.0</td><td>Wu et al. (2023); Jin et al. (2019)</td></tr><tr><td>Pure-Dove</td><td>Apache License 2.0</td><td>–</td></tr><tr><td>PygmalionAI-PIPPA</td><td>Apache License 2.0</td><td>Gosling et al. (2023)</td></tr></tbody><tfoot><tr><td colspan="3">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 8: **References and licenses for alignment-tuning (text) dataset collections** presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><td>Collection</td><td>Licenses</td><td>Cite</td></tr></thead><tbody><tr><td>Reasoning</td><td>Apache License 2.0</td><td>–</td></tr><tr><td>RiddleSense</td><td>MIT License</td><td>Lin et al. (2021a)</td></tr><tr><td>SeaBench</td><td>Apache License 2.0</td><td>Nguyen et al. (2023)</td></tr><tr><td>SelFee</td><td>MIT License</td><td>Ye et al. (2023)</td></tr><tr><td>Synth.-GSM8K-Refl.</td><td>Meta Llama3 Community License</td><td>–</td></tr><tr><td>Thai Gen AI</td><td>Various</td><td>–</td></tr><tr><td>UltraChat-200k</td><td>CC BY-NC 4.0</td><td>Ding et al. (2023)</td></tr><tr><td>UltraFeedback Argilla</td><td>Various</td><td>–</td></tr><tr><td>WildChat</td><td>AI2 ImpACT License - Low Risk</td><td>Zhao et al. (2023)</td></tr></tbody></table>
<table><caption>Table 9: References and licenses for audio dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>1111 Hours Hindi</td><td>Custom</td><td>Bhanushali et al. (2022)</td></tr><tr><td>120h Spanish Speech</td><td>CC0 1.0</td><td>–</td></tr><tr><td>AFRISPEECH-200</td><td>CC BY-NC-SA 4.0</td><td>Olatunji et al. (2023)</td></tr><tr><td>AISHELL-1</td><td>Apache 2.0</td><td>Bu et al. (2017)</td></tr><tr><td>AISHELL-2</td><td>Unspecified</td><td>Du et al. (2018)</td></tr><tr><td>AISHELL-4</td><td>CC BY-SA 4.0</td><td>Fu et al. (2021)</td></tr><tr><td>ALLSSTAR</td><td>CC BY 4.0</td><td>Bradlow (2010)</td></tr><tr><td>AMI</td><td>CC BY 4.0</td><td>Carletta et al. (2006)</td></tr><tr><td>Aalto Finnish Parl.</td><td>Custom</td><td>Virkkunen et al. (2022)</td></tr><tr><td>African Acc. French</td><td>Apache 2.0</td><td>–</td></tr><tr><td>Basq., Cat. and Gal.</td><td>CC BY-SA 4.0</td><td>Kjartansson et al. (2020)</td></tr><tr><td>Bloom Speech</td><td>Various</td><td>Leong et al. (2022)</td></tr><tr><td>Bud500</td><td>Apache 2.0, CC BY-NC-SA 4.0</td><td>–</td></tr><tr><td>CSJ</td><td>Custom</td><td>Maekawa (2003)</td></tr><tr><td>CSLU 1.2</td><td>CSLU Agreement</td><td>Lander, T (2007)</td></tr><tr><td>CSLU 22 Langs.</td><td>CSLU Agreement</td><td>Lander, T (2005)</td></tr><tr><td>ClarinPL</td><td>CC BY 4.0</td><td>Korzinek et al. (2017)</td></tr><tr><td>CoNASE</td><td>Custom</td><td>Coats (2019)</td></tr><tr><td>CoVoST-2</td><td>CC0 1.0</td><td>Wang et al. (2020a)</td></tr><tr><td>Common Voice 17</td><td>CC0 1.0</td><td>Ardila et al. (2020a)</td></tr><tr><td>Crowd Sourced Speech</td><td>CC BY-SA 4.0</td><td>Kjartansson et al. (2018)</td></tr><tr><td>Czech Parliament</td><td>CC BY 4.0</td><td>Kratochvil et al. (2020)</td></tr><tr><td>DiDiSpeech</td><td>Unspecified</td><td>Guo et al. (2021)</td></tr><tr><td>Earnings-22</td><td>Unspecified</td><td>Del Rio et al. (2022)</td></tr><tr><td>EdAcc</td><td>CC BY-SA 4.0</td><td>Sanabria et al. (2023)</td></tr><tr><td>Eng. Acc. in Brit. Isles</td><td>CC BY-SA 4.0</td><td>Demirsahin et al. (2020)</td></tr><tr><td>English-Vietnamese</td><td>CC BY-NC-ND 4.0</td><td>Nguyen et al. (2022)</td></tr><tr><td>FT SPEECH</td><td>Custom</td><td>Kirkedal et al. (2020)</td></tr><tr><td>Fisher</td><td>LDC User Agreement</td><td>Cieri et al. (2004)</td></tr><tr><td>Fleurs</td><td>CC BY 4.0</td><td>Conneau et al. (2022)</td></tr><tr><td>GigaSpeech</td><td>Apache 2.0</td><td>Chen et al. (2021b)</td></tr><tr><td>Golos</td><td>Custom</td><td>Karpov et al. (2021)</td></tr><tr><td>Hebrew Coursera</td><td>Unspecified</td><td>–</td></tr><tr><td>Hebrew Kan</td><td>Unspecified</td><td>–</td></tr><tr><td>Highland Puebla Nahuatl</td><td>CC BY-NC-SA 3.0</td><td>Shi et al. (2021)</td></tr><tr><td>JTUBESPEECH</td><td>Unspecified</td><td>Takamichi et al. (2021)</td></tr><tr><td>Japanese Anime Speech</td><td>CC0 1.0</td><td>–</td></tr><tr><td>KSC</td><td>CC BY 4.0</td><td>Khassanov et al. (2021)</td></tr><tr><td>Kathbath</td><td>CC0 1.0</td><td>Javed et al. (2022)</td></tr><tr><td>KeSpeech</td><td>Custom</td><td>Tang et al. (2021)</td></tr><tr><td>KsponSpeech</td><td>Unspecified</td><td>Bang et al. (2020)</td></tr></tbody></table>

Continued on next page
<table><caption>Table 9: References and licenses for audio dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>LJSpeech</td><td>Public Domain</td><td>Ito &amp; Johnson (2017)</td></tr><tr><td>LaboroTVSpeech</td><td>Custom</td><td>Ando &amp; Fujihara (2021)</td></tr><tr><td>LibriSpeech</td><td>CC BY 4.0</td><td>Panayotov et al. (2015)</td></tr><tr><td>M-AILABS</td><td>Custom</td><td>Solak (2024)</td></tr><tr><td>M2ASR</td><td>Unspecified</td><td>Shi et al. (2017); Mamtimin et al. (2023); Zhi et al. (2021); Li et al. (2017)</td></tr><tr><td>MAGICDATA</td><td>CC BY-NC-ND 4.0</td><td>—</td></tr><tr><td>MASC</td><td>CC BY 4.0</td><td>Al-Fetyani et al. (2023)</td></tr><tr><td>MaSS</td><td>Unspecified</td><td>Boito et al. (2020)</td></tr><tr><td>MagicData-RAMC</td><td>CC BY-NC-ND 4.0</td><td>Yang et al. (2022)</td></tr><tr><td>MediaSpeech</td><td>CC BY 4.0</td><td>Kolobov et al. (2021)</td></tr><tr><td>Minds14</td><td>CC BY 4.0</td><td>Gerz et al. (2021)</td></tr><tr><td>MuST-C</td><td>CC BY-NC-ND 4.0</td><td>Di Gangi et al. (2019)</td></tr><tr><td>Multiling. LibriSpeech</td><td>CC BY 4.0</td><td>Pratap et al. (2020)</td></tr><tr><td>Multiling. TEDx</td><td>CC BY-NC-ND 4.0</td><td>Salesky et al. (2021)</td></tr><tr><td>NST Danish</td><td>CC0 1.0</td><td>—</td></tr><tr><td>NST Norwegian</td><td>CC0 1.0</td><td>—</td></tr><tr><td>NST Swedish</td><td>CC0 1.0</td><td>—</td></tr><tr><td>Nigerian English</td><td>CC BY-SA 4.0</td><td>—</td></tr><tr><td>Norwegian Parl.</td><td>CC0 1.0</td><td>Solberg &amp; Ortiz (2022)</td></tr><tr><td>Norwegian Parl. Speech</td><td>CC0 1.0</td><td>Solberg &amp; Ortiz (2022)</td></tr><tr><td>OLKAVS</td><td>Custom</td><td>Park et al. (2023)</td></tr><tr><td>OpenSTT</td><td>CC BY-NC 4.0</td><td>Andrusenko et al. (2020)</td></tr><tr><td>People's Speech</td><td>Various</td><td>Galvez et al. (2021)</td></tr><tr><td>QASR</td><td>Unspecified</td><td>Mubarak et al. (2021)</td></tr><tr><td>RTVE</td><td>Custom</td><td>—</td></tr><tr><td>ReazonSpeech</td><td>CDLA-Sharing-1.0</td><td>Yin et al. (2023)</td></tr><tr><td>Regional Af. Am. Lang.</td><td>CC BY-NC-SA 4.0</td><td>—</td></tr><tr><td>RixVox</td><td>CC BY 4.0</td><td>—</td></tr><tr><td>SDS-200</td><td>Custom</td><td>Plüss et al. (2022)</td></tr><tr><td>SPGISpeech</td><td>Custom</td><td>O'Neill et al. (2021)</td></tr><tr><td>Samromur</td><td>CC BY 4.0</td><td>Mollberg et al. (2020)</td></tr><tr><td>Samromur Children</td><td>CC BY 4.0</td><td>Hernandez Mena et al. (2022)</td></tr><tr><td>Samromur Milljon</td><td>CC BY 4.0</td><td>—</td></tr><tr><td>Shrutilipi</td><td>CC0 1.0</td><td>Bhogale et al. (2022)</td></tr><tr><td>Snow Mountain</td><td>CC BY-SA 4.0</td><td>Raju et al. (2023)</td></tr><tr><td>Spoken Wikipedia</td><td>CC BY-SA 4.0</td><td>Baumann et al. (2019)</td></tr><tr><td>Switchboard</td><td>LDC User Agreement</td><td>Godfrey et al. (1992)</td></tr><tr><td>TED-LIUM3</td><td>CC BY-NC-ND 3.0</td><td>Hernandez et al. (2018)</td></tr><tr><td>THCHS-30</td><td>Apache 2.0</td><td>Wang &amp; Zhang (2015)</td></tr><tr><td>THUYG-20</td><td>Apache 2.0</td><td>Rozi et al. (2015)</td></tr></tbody></table>

Continued on next page
<table><caption>Table 9: References and licenses for audio dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><td>Collection</td><td>Licenses</td><td>Cite</td></tr></thead><tbody><tr><td>TIMIT</td><td>LDC User Agreement</td><td>Garofolo, John S. et al. (1993)</td></tr><tr><td>VCTK</td><td>CC BY 4.0</td><td>–</td></tr><tr><td>VibraVox</td><td>CC BY 4.0</td><td>–</td></tr><tr><td>VoxPopuli</td><td>CC0 1.0</td><td>Wang et al. (2021)</td></tr><tr><td>Vystadial</td><td>CC BY-SA 3.0</td><td>Korvas et al. (2014)</td></tr><tr><td>WenetSpeech</td><td>CC BY 4.0</td><td>Zhang et al. (2022a)</td></tr><tr><td>West Afr. Radio</td><td>CC BY-SA 4.0</td><td>Doumbouya et al. (2021)</td></tr><tr><td>West Afr. Virt. Asst.</td><td>CC BY-SA 4.0</td><td>Doumbouya et al. (2021)</td></tr><tr><td>YODAS</td><td>CC BY 3.0</td><td>Li et al. (2023c)</td></tr><tr><td>Zeroth-Korean</td><td>CC BY 4.0</td><td>–</td></tr><tr><td>aidatatang</td><td>CC BY-NC-ND 4.0</td><td>–</td></tr></tbody></table>
<table><caption>Table 10: References and licenses for video dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>100DOH</td><td>Custom</td><td>Shan et al. (2020)</td></tr><tr><td>20BN-SOMETHING</td><td>Custom</td><td>Goyal et al. (2017)</td></tr><tr><td>20BN-jester</td><td>Custom</td><td>Materzynska et al. (2019)</td></tr><tr><td>50 Salads</td><td>CC BY-NC-SA 4.0</td><td>Stein & McKenna (2013)</td></tr><tr><td>ActivityNet</td><td>MIT License</td><td>Heilbron et al. (2015)</td></tr><tr><td>Apes</td><td>Unspecified</td><td>Alcazar et al. (2021)</td></tr><tr><td>Ava</td><td>CC BY 4.0</td><td>Roth et al. (2019); Gu et al. (2018)</td></tr><tr><td>Breakfast</td><td>CC BY 4.0</td><td>Kuehne et al. (2014)</td></tr><tr><td>CDAD</td><td>Unspecified</td><td>Xiang et al. (2022)</td></tr><tr><td>COIN</td><td>Custom</td><td>Tang et al. (2019)</td></tr><tr><td>Charades</td><td>Custom</td><td>Sigurdsson et al. (2016b)</td></tr><tr><td>Charades-Ego</td><td>Custom</td><td>Sigurdsson et al. (2018)</td></tr><tr><td>Collective</td><td>Unspecified</td><td>Wongun Choi et al. (2009)</td></tr><tr><td>Condensed Movies</td><td>CC BY 4.0</td><td>Bain et al. (2020)</td></tr><tr><td>CrossTask</td><td>Unspecified</td><td>Zhukov et al. (2019)</td></tr><tr><td>Davis</td><td>Custom</td><td>Perazzi et al. (2016)</td></tr><tr><td>DiDeMo</td><td>BSD 2-Clause License</td><td>Hendricks et al. (2018)</td></tr><tr><td>Disney Vid. Gen.</td><td>Apache 2.0</td><td>—</td></tr><tr><td>EEV</td><td>CC BY 4.0</td><td>Sun et al. (2021)</td></tr><tr><td>EPIC-KITCHENS</td><td>CC BY-NC 4.0</td><td>Damen et al. (2018)</td></tr><tr><td>Ego4D</td><td>Custom, MIT License</td><td>Grauman et al. (2022)</td></tr><tr><td>FERV39k</td><td>CC BY-NC 4.0</td><td>Wang et al. (2022b)</td></tr><tr><td>FineGym</td><td>CC BY-NC 4.0</td><td>Shao et al. (2020)</td></tr><tr><td>FineVideo</td><td>CC BY 4.0</td><td>—</td></tr><tr><td>HAA500</td><td>Unspecified</td><td>Chung et al. (2021)</td></tr><tr><td>HACS</td><td>Custom</td><td>Zhao et al. (2019)</td></tr><tr><td>HD-VILA-100M</td><td>Custom</td><td>Xue et al. (2022)</td></tr><tr><td>HMDB</td><td>CC BY 4.0</td><td>Kuehne et al. (2011)</td></tr><tr><td>HOLLYWOOD2</td><td>Unspecified</td><td>Marszalek et al. (2009)</td></tr><tr><td>HOMAGE</td><td>Unspecified</td><td>Rai et al. (2021)</td></tr><tr><td>HVU</td><td>Custom</td><td>Diba et al. (2020)</td></tr><tr><td>Hollywood Ext.</td><td>MIT License</td><td>Bojanowski et al. (2014)</td></tr><tr><td>How2</td><td>Various</td><td>Sanabria et al. (2018)</td></tr><tr><td>HowTo100M</td><td>Unspecified</td><td>Miech et al. (2019)</td></tr><tr><td>ImageNet-Vid</td><td>CC BY-NC 4.0</td><td>Russakovsky et al. (2015)</td></tr><tr><td>Kinetics</td><td>Unspecified</td><td>Kay et al. (2017); Carreira et al. (2018); Smaira et al. (2020)</td></tr><tr><td>LEMMA</td><td>Unspecified</td><td>Jia et al. (2020)</td></tr><tr><td>LSMDC</td><td>Custom, MIT License</td><td>Rohrbach et al. (2016a); Sharma et al. (2020)</td></tr><tr><td>M-MiT</td><td>Unspecified</td><td>Monfort et al. (2021b)</td></tr></tbody></table>

Continued on next page
<table><caption>Table 10: References and licenses for video dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><th>Collection</th><th>Licenses</th><th>Cite</th></tr></thead><tbody><tr><td>MAD</td><td>Custom</td><td>Soldan et al. (2022)</td></tr><tr><td>MMAct</td><td>Custom</td><td>Kong et al. (2019)</td></tr><tr><td>MPII</td><td>Unspecified, Custom</td><td>Rohrbach et al. (2016b; 2015)</td></tr><tr><td>MSA</td><td>Unspecified</td><td>Xiong et al. (2019b)</td></tr><tr><td>MSR-VTT</td><td>Unspecified</td><td>Xu et al. (2016)</td></tr><tr><td>MVBench</td><td>MIT License</td><td>Li et al. (2024)</td></tr><tr><td>Mars</td><td>Unspecified</td><td>Zheng et al. (2016)</td></tr><tr><td>Mimetics</td><td>Unspecified</td><td>Weinzaepfel & Rogez (2021)</td></tr><tr><td>Moments in Time</td><td>Custom</td><td>Monfort et al. (2019b)</td></tr><tr><td>Movie-Net</td><td>Unspecified</td><td>Huang et al. (2020)</td></tr><tr><td>MovieGraphs</td><td>Custom</td><td>Vicol et al. (2018)</td></tr><tr><td>MovieQA</td><td>Unspecified</td><td>Tapaswi et al. (2016)</td></tr><tr><td>MovieScenes</td><td>Unspecified</td><td>Rao et al. (2020)</td></tr><tr><td>MultiTHUMOS</td><td>CC BY 4.0</td><td>Yeung et al. (2017)</td></tr><tr><td>NTU RGB+D</td><td>Custom</td><td>Shahroudy et al. (2016)</td></tr><tr><td>Narrated Instr. Vid.</td><td>MIT License</td><td>Alayrac et al. (2016)</td></tr><tr><td>OmniSource-Web</td><td>Apache License 2.0</td><td>Duan et al. (2020)</td></tr><tr><td>Oops!</td><td>CC BY-NC-SA 4.0</td><td>Epstein et al. (2020)</td></tr><tr><td>OpenVid-1M</td><td>CC-BY-4.0</td><td>Nan et al. (2024)</td></tr><tr><td>PKU-MMD</td><td>Unspecified</td><td>Liu et al. (2017)</td></tr><tr><td>Project-Aria</td><td>Apache License 2.0</td><td>Pan et al. (2023); Lv et al. (2024)</td></tr><tr><td>QFVS</td><td>Unspecified</td><td>Sharghi et al. (2017)</td></tr><tr><td>QuerYD</td><td>Unspecified</td><td>Oncescu et al. (2021)</td></tr><tr><td>RareAct</td><td>Unspecified</td><td>Miech et al. (2020)</td></tr><tr><td>SOA</td><td>Unspecified</td><td>Diba et al. (2020)</td></tr><tr><td>ShareGPT4Video</td><td>Attribution-NonCommercial 4.0 International</td><td>Chen et al. (2024)</td></tr><tr><td>Spoken Moments</td><td>Custom</td><td>Monfort et al. (2021a)</td></tr><tr><td>Sports-1M</td><td>CC BY 3.0</td><td>Karpathy et al. (2014)</td></tr><tr><td>StoryGraphs</td><td>Unspecified</td><td>Tapaswi et al. (2014)</td></tr><tr><td>SumMe</td><td>Unspecified</td><td>Gygli et al. (2014)</td></tr><tr><td>TGIF</td><td>Custom</td><td>Li et al. (2016)</td></tr><tr><td>THUMOS</td><td>Custom</td><td>Idrees et al. (2017)</td></tr><tr><td>TITAN</td><td>Non Commercial</td><td>Malla et al. (2020)</td></tr><tr><td>TRECVid</td><td>CC BY-NC-SA 4.0</td><td>Awad et al. (2020)</td></tr><tr><td>TVSum</td><td>CC BY 3.0</td><td>Yale Song et al. (2015)</td></tr><tr><td>TinyVIRAT</td><td>Unspecified</td><td>Demir et al. (2020)</td></tr><tr><td>Toyota Smarthome</td><td>Custom</td><td>Das et al. (2019)</td></tr><tr><td>UAV-Human</td><td>Custom</td><td>Li et al. (2021b)</td></tr><tr><td>UCF101</td><td>Unspecified</td><td>Soomro et al. (2012)</td></tr><tr><td>VIOLIN</td><td>Unspecified</td><td>Liu et al. (2020)</td></tr><tr><td>VLOG</td><td>Custom</td><td>Fouhey et al. (2017)</td></tr></tbody></table>

Continued on next page
<table><caption>Table 10: **References and licenses for video** dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having “Various” licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.</caption><thead><tr><td>Collection</td><td>Licenses</td><td>Cite</td></tr></thead><tbody><tr><td>VTW</td><td>Unspecified</td><td>Zeng et al. (2016)</td></tr><tr><td>VaTeX</td><td>CC BY 4.0</td><td>Wang et al. (2020d)</td></tr><tr><td>VidProm</td><td>CC-BY-NC 4.0</td><td>Wang & Yang (2024b)</td></tr><tr><td>VideoLT</td><td>Non Commercial</td><td>Zhang et al. (2021)</td></tr><tr><td>VideoStory</td><td>Unspecified</td><td>Habibian et al. (2014)</td></tr><tr><td>Volleyball</td><td>Unspecified</td><td>Ibrahim et al. (2016)</td></tr><tr><td>VoxCeleb</td><td>Custom</td><td>Nagrani et al. (2018)</td></tr><tr><td>WebVid</td><td>Custom</td><td>Bain et al. (2022)</td></tr><tr><td>YouCook</td><td>Unspecified</td><td>Das et al. (2013)</td></tr><tr><td>YouCook2</td><td>MIT License</td><td>Zhou et al. (2017)</td></tr><tr><td>Youtube-8M</td><td>Unspecified</td><td>Abu-El-Haija et al. (2016a)</td></tr></tbody></table>

REFERENCES

Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. The multilingual alignment prism: Aligning global and local preferences to reduce harm, 2024. URL https://arxiv.org/abs/2406.18682.

Mourad Abbas, Kamel Smaïli, and D. Berkani. Evaluation of Topic Identification Methods on Arabic Corpora. Journal of Digital Information Management, 9(5), October 2011. URL https://www.dline.info/fpaper/jdim/v9i5/1.pdf.

Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. ArabicaQA: A Comprehensive Dataset for Arabic Question Answering, March 2024. URL http://arxiv.org/abs/2403.17848. arXiv:2403.17848 [cs].

Ahmed Abdelghany, Hammam Abdelaal, Abdulrahman Kamr, and Passent Elkafrawy. Doc2Vec: An approach to identify Hadith Similarities. Australian Journal of Basic and Applied Sciences, pp. 46–53, December 2020. doi: 10.22587/ajbas.2020.14.12.5.

Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. YouTube-8M: A Large-Scale Video Classification Benchmark, September 2016a. URL http://arxiv.org/abs/1609.08675. arXiv:1609.08675 [cs].

Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016b.

David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 9:1116–1131, 2021.

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kebede Guge, and Pontus Stenetorp. Irokobench: A new benchmark for african languages in the age of large language models, 2024. URL https://arxiv.org/abs/2406.03368.
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. Topi-OCQA: Open-domain Conversational Question Answering with Topic Switching, February 2022. URL http://arxiv.org/abs/2110.00768. arXiv:2110.00768 [cs].

Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training, March 2021. URL http://arxiv.org/abs/2010.12688. arXiv:2010.12688 [cs].

Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In *International Conference on Machine Learning*, pp. 265–279. PMLR, 2023.

Alëna Aksënova, Daan van Esch, James Flynn, and Pavel Golik. How might we create better benchmarks for speech recognition? In Kenneth Church, Mark Liberman, and Valia Kordoni (eds.), *Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future*, pp. 22–34, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.bppf-1.4. URL https://aclanthology.org/2021.bppf-1.4.

Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards Tracing Factual Knowledge in Language Models Back to the Training Data, October 2022. URL http://arxiv.org/abs/2205.11482. arXiv:2205.11482 [cs].

Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. MASC: Massive Arabic Speech Corpus. In *2022 IEEE Spoken Language Technology Workshop (SLT)*, pp. 1006–1013, Doha, Qatar, January 2023. IEEE. ISBN 978-8-3503-9690-4. doi: 10/gtsqzj. URL https://ieeexplore.ieee.org/document/10022652/.

Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised Learning from Narrated Instruction Videos, June 2016. URL http://arxiv.org/abs/1506.09215. arXiv:1506.09215 [cs].

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. *arXiv preprint arXiv:2402.16827*, 2024.

Juan Leon Alcazar, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbelaez, Bernard Ghanem, and Fabian Caba Heilbron. APES: Audiovisual Person Search in Untrimmed Video, June 2021. URL http://arxiv.org/abs/2106.01667. arXiv:2106.01667 [cs].

Reem Alghamdi, Zhenwen Liang, and Xiangliang Zhang. ArMATH: a Dataset for Solving Arabic Math Word Problems. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pp. 351–362, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.37.

Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 101 Billion Arabic Words Dataset, April 2024. URL http://arxiv.org/abs/2405.01590. arXiv:2405.01590 [cs].

Zaid Alyafeai, Maraim Masoud, Mustafa Ghaleb, and Maged S Al-shaibani. Masader: Metadata sourcing for arabic text and speech data resources. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pp. 6340–6351, 2022.

Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran A. Q. Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged S. Al-Shaibani. CIDAR: Culturally Relevant Instruction Dataset For Arabic, February 2024. URL http://arxiv.org/abs/2402.03177. arXiv:2402.03177 [cs].

Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms, May 2019. URL http://arxiv.org/abs/1905.13319. arXiv:1905.13319 [cs].
Shintaro Ando and Hiromasa Fujihara. Construction of a Large-scale Japanese ASR Corpus on TV Recordings, March 2021. URL http://arxiv.org/abs/2103.14736. arXiv:2103.14736 [cs, eess].

Andrei Andrusenko, Aleksandr Laptev, and Ivan Medennikov. Exploration of End-to-End ASR for OpenSTT – Russian Open Speech-to-Text Dataset. In Lecture Notes in Computer Science, volume 12335, pp. 35–44. Springer, Cham, 2020. doi: 10.1007/978-3-030-60276-5_4. URL http://arxiv.org/abs/2006.08274. arXiv:2006.08274 [cs, eess].

Mariona Coll Ardanuy, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Katherine McDonough, Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. Living Machines: A study of atypical animacy, November 2020. URL http://arxiv.org/abs/2005.11140. arXiv:2005.11140 [cs].

Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019.

Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common Voice: A Massively-Multilingual Speech Corpus, March 2020a. URL http://arxiv.org/abs/1912.06670. arXiv:1912.06670 [cs].

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pp. 4218–4222, Marseille, France, May 2020b. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.520.

Anthony B Atkinson et al. On the measurement of inequality. *Journal of economic theory*, 2(3): 244–263, 1970.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do-han, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL http://arxiv.org/abs/2108.07732. arXiv:2108.07732 [cs].

George Awad, Asad A. Butt, Keith Curtis, Yooyoung Lee, Jonathan Fiscus, Afzal Godil, Andrew Delgado, Jesse Zhang, Eliot Godard, Lukas Diduch, Alan F. Smeaton, Yvette Graham, Wessel Kraaij, and Georges Quenot. TRECVID 2019: An Evaluation Campaign to Benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & Retrieval, September 2020. URL http://arxiv.org/abs/2009.09984. arXiv:2009.09984 [cs].

Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/35309226eb45ec366ca86a4329a2b7c3-Paper.pdf.

Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics, February 2023. URL http://arxiv.org/abs/2302.12433. arXiv:2302.12433 [cs].

Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Von Platen, Yatharth Saraf, Juan Pino, et al. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296, 2021.
Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Min Yang, Shiwen Ni, and Ge Zhang. COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning, March 2024a. URL http://arxiv.org/abs/2403.18058. arXiv:2403.18058 [cs] version: 1.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022. URL http://arxiv.org/abs/2204.05862. arXiv:2204.05862 [cs].

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A Recipe for Long Context Alignment of Large Language Models, January 2024b. URL http://arxiv.org/abs/2401.18058. arXiv:2401.18058 [cs].

Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed Movies: Story Based Retrieval with Contextual Embeddings, October 2020. URL http://arxiv.org/abs/2005.04208. arXiv:2005.04208 [cs].

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval, May 2022. URL http://arxiv.org/abs/2104.00650. arXiv:2104.00650 [cs].

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated Machine Reading Comprehension Dataset, October 2018. URL http://arxiv.org/abs/1611.09268. arXiv:1611.09268 [cs].

Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue, June 2019. URL http://arxiv.org/abs/1906.07220. arXiv:1906.07220 [cs].

Jack Bandy and Nicholas Vincent. Addressing “documentation debt” in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241, 2021.

Jeong-Uk Bang, Seung Yun, Seung-Hi Kim, Mu-Yeol Choi, Min-Kyu Lee, Yeo-Jeong Kim, Dong-Hyun Kim, Jun Park, Young-Jik Lee, and Sang-Hun Kim. KsponSpeech: Korean Spontaneous Speech Corpus for Automatic Speech Recognition. Applied Sciences, 10(19):6936, January 2020. ISSN 2076-3417. doi: 10/gtwwck. URL https://www.mdpi.com/2076-3417/10/19/6936. Number: 19 Publisher: Multidisciplinary Digital Publishing Institute.

Samuel Barham, Orion Weller, Michelle Yuan, Kenton Murray, Mahsa Yarmohammadi, Zheng-ping Jiang, Siddharth Vashishtha, Alexander Martin, Anqi Liu, Aaron Steven White, Jordan Boyd-Graber, and Benjamin Van Durme. MegaWika: Millions of reports and their sources across 50 diverse languages, July 2023. URL http://arxiv.org/abs/2307.07049. arXiv:2307.07049 [cs].

Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension. Transactions of the Association for Computational Linguistics, 8:662–678, December 2020. ISSN 2307-387X. doi: 10/gjzgwj. URL http://arxiv.org/abs/2002.00293. arXiv:2002.00293 [cs].

Timo Baumann, Arne Köhn, and Felix Hennig. The Spoken Wikipedia Corpus collection: Harvesting, alignment and an application to hyperlistening. Language Resources and Evaluation, 53(2): 303–329, June 2019. ISSN 1574-0218. doi: 10/gq5xdf. URL https://doi.org/10.1007/s10579-017-9410-y.
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. *Transactions of the Association for Computational Linguistics*, 6:587–604, 2018. doi: 10.1162/tacl_a_00041. URL https://aclanthology.org/Q18-1041.

Anish Bhanushali, Grant Bridgman, Deekshitha G, Prasanta Ghosh, Pratik Kumar, Saurabh Kumar, Adithya Raj Kolladath, Nithya Ravi, Aaditeshwar Seth, Ashish Seth, Abhayjeet Singh, Vrunda Sukhadia, Umesh S, Sathvik Udupa, and Lodagala V. S. V. Durga Prasad. Gram Vaani ASR Challenge on spontaneous telephone speech recordings in regional variations of Hindi. In *Interspeech 2022*, pp. 3548–3552. ISCA, September 2022. doi: 10/gtsqzn. URL https://www.isca-archive.org/interspeech_2022/bhanushali22_interspeech.html.

Kaushal Santosh Bhogale, Abhigyan Raman, Tahir Javed, Sumanth Doddapaneni, Anoop Kunchukutan, Pratyush Kumar, and Mitesh M. Khapra. Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages, August 2022. URL http://arxiv.org/abs/2208.12666. arXiv:2208.12666 [cs, eess].

Mariam Biltawi, Arafat Awajan, and Sara Tedmori. Arabic Reading Comprehension Benchmarks Created Semiautomatically. In *2020 21st International Arab Conference on Information Technology (ACIT)*, pp. 1–6, Giza, Egypt, November 2020. IEEE. ISBN 978-1-72818-855-3. doi: 10/g6k6b8. URL https://ieeexplore.ieee.org/document/9300111/.

Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. *arXiv preprint arXiv:2110.01963*, 2021.

Abeba Birhane, Vinay Prabhu, Sang Han, Vishnu Naresh Boddeti, and Alexandra Sasha Luccioni. Into the laions den: Investigating hate in multimodal datasets. *arXiv preprint arXiv:2311.03449*, 2023.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language, November 2019. URL http://arxiv.org/abs/1911.11641. arXiv:1911.11641 [cs].

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. URL https://arxiv.org/abs/2311.15127.

Marcely Zanon Boito, William N. Havard, Mahault Garnerin, Eric Le Ferrand, and Laurent Besacier. MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible, February 2020. URL http://arxiv.org/abs/1907.12895. arXiv:1907.12895 [cs].

Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly Supervised Action Labeling in Videos Under Ordering Constraints, July 2014. URL http://arxiv.org/abs/1407.1208. arXiv:1407.1208 [cs].

Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index, 2023.

Michael Boratko, Xiang Lorraine Li, Rajarshi Das, Tim O'Gorman, Dan Le, and Andrew McCallum. ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning, October 2020. URL http://arxiv.org/abs/2005.00771. arXiv:2005.00771 [cs].

Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. Learning To Split and Rephrase From Wikipedia Edit History, August 2018. URL http://arxiv.org/abs/1808.09468. arXiv:1808.09468 [cs].

Florian Boudin and Ygor Gallina. Redefining Absent Keyphrases and their Effect on Retrieval Effectiveness. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 4185–4193, Online, June 2021. Association for
Computational Linguistics. doi: 10/g6k64d. URL https://aclanthology.org/2021.nacl1-main.330.

A.R. Bradlow. ALLSSTAR: Archive of L1 and L2 Scripted and Spontaneous Transcripts And Recordings, 2010. URL https://speechbox.linguistics.northwestern.edu/#/?goto=allsstar.

Alex Bravo, Janet Pinero, Nuria Queralt-Rosinach, Michael Rautschka, and Laura I. Furlong. Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC Bioinformatics, 16(1):55, February 2015. ISSN 1471-2105. doi: 10/f7kn8s. URL https://doi.org/10.1186/s12859-015-0472-9.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/video-generation-models-as-world-simulators.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 1877–1901. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020b. URL http://arxiv.org/abs/2005.14165. arXiv:2005.14165 [cs].

Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline, September 2017. URL http://arxiv.org/abs/1709.05522. arXiv:1709.05522 [cs].

Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.), *Proceedings of the 1st Conference on Fairness, Accountability and Transparency*, volume 81 of *Proceedings of Machine Learning Research*, pp. 77–91. PMLR, February 2018a. URL https://proceedings.mlr.press/v81/buolamwini18a.html.

Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.), *Proceedings of the 1st Conference on Fairness, Accountability and Transparency*, volume 81 of *Proceedings of Machine Learning Research*, pp. 77–91. PMLR, February 2018b. URL https://proceedings.mlr.press/v81/buolamwini18a.html.

Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset, September 2019. URL http://arxiv.org/abs/1909.05358. arXiv:1909.05358 [cs].

Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base, June 2022. URL http://arxiv.org/abs/2007.03875. arXiv:2007.03875 [cs].
Shuyang Cao and Lu Wang. Controllable Open-ended Question Generation with A New Question Type Ontology, June 2021. URL http://arxiv.org/abs/2107.00152. arXiv:2107.00152 [cs].

Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. The AMI Meeting Corpus: A Pre-announcement. In Steve Renals and Samy Bengio (eds.), *Machine Learning for Multimodal Interaction*, volume 3869, pp. 28–39. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006. ISBN 978-3-540-32549-9. doi: 10.1007/11677482_3. URL http://link.springer.com/10.1007/11677482_3. Series Title: Lecture Notes in Computer Science.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. February 2022.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In *The Eleventh International Conference on Learning Representations*. OpenReview, 2023a.

Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In *32nd USENIX Security Symposium (USENIX Security 23)*, pp. 5253–5270, Anaheim, CA, August 2023b. USENIX Association. ISBN 978-1-939133-37-3. URL https://www.usenix.org/conference/usenixsecurity23/presentation/carlini.

Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A Short Note about Kinetics-600, August 2018. URL http://arxiv.org/abs/1808.01340. arXiv:1808.01340 [cs].

Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient Intent Detection with Dual Sentence Encoders. In Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kumar, Iñigo Casanueva, and Rushin Shah (eds.), *Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI*, pp. 38–45, Online, July 2020. Association for Computational Linguistics. doi: 10/gjhzzs. URL https://aclanthology.org/2020.nlp4convai-1.5. arXiv:2003.04807 [cs].

Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. arXiv preprint arXiv:2103.12028, 2021.

Sarah Huiyi Cen, Aspen Hopkins, Andrew Ilyas, Aleksander Madry, Isabella Struckman, and Luis Videgaray Caso. AI Supply Chains, April 2023. URL http://dx.doi.org/10.2139/ssrn.4789403.

Alberto Cetoli, Mohammad Akbari, Stefano Bragaglia, Andrew D. O’Harney, and Marc Sloan. Named Entity Disambiguation using Deep Learning on Graphs. volume 11438, pp. 78–86. 2019. doi: 10.1007/978-3-030-15719-7_10. URL http://arxiv.org/abs/1810.09164. arXiv:1810.09164 [cs].

Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural Legal Judgment Prediction in English, June 2019a. URL http://arxiv.org/abs/1906.02059. arXiv:1906.02059 [cs].

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. Large-Scale Multi-Label Text Classification on EU Legislation, June 2019b. URL http://arxiv.org/abs/1906.02192. arXiv:1906.02192 [cs].

Ilias Chalkidis, Manos Fergadiotis, Nikolaos Manginas, Eva Katakalou, and Prodromos Malakasiotis. Regulatory Compliance through Doc2Doc Information Retrieval: A case study in EU/UK legislation where text similarity has limitations, January 2021. URL http://arxiv.org/abs/2101.10726. arXiv:2101.10726 [cs].
Jun Shern Chan, Michael Pieler, Jonathan Jao, Jérémy Scheurer, and Ethan Perez. Few-shot Adaptation Works with UnpredictTable Data, August 2022. URL http://arxiv.org/abs/2208.01009. arXiv:2208.01009 [cs].

Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 4283-4294. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/2cd4e8a2ce081c3d7c32c3cde4312ef7-Paper.pdf.

Xinyu Chang. Gender bias in hiring: An analysis of the impact of amazon's recruiting algorithm. *Advances in Economics, Management and Political Sciences*, 23:134-140, 09 2023. doi: 10.54254/2754-1169/23/20230367.

Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, and Chloe Clavel. Hierarchical Pre-training for Sequence Labelling in Spoken Dialog, February 2021. URL http://arxiv.org/abs/2009.11152. arXiv:2009.11152 [cs].

Jose M. Chaquet, Enrique J. Carmona, and Antonio Fernández-Caballero. A survey of video datasets for human action and activity recognition. *Computer Vision and Image Understanding*, 117(6):633-659, June 2013. ISSN 1077-3142. doi: 10.1016/j.cviu.2013.01.013. URL http://dx.doi.org/10.1016/j.cviu.2013.01.013.

Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 3167-3185, Online, June 2021. Association for Computational Linguistics. doi: 10/gtsqxv. URL https://aclanthology.org/2021.naacl-main.254.

Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 3002-3017, Online, June 2021a. Association for Computational Linguistics. doi: 10/gtsqxt. URL https://aclanthology.org/2021.naacl-main.239.

Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuai-jiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, Zhao You, and Zhiy-ong Yan. GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio, June 2021b. URL http://arxiv.org/abs/2106.06909. arXiv:2106.06909 [cs, eess].

Hannah Chen, Yangfeng Ji, and David Evans. Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory, November 2020a. URL http://arxiv.org/abs/2011.01856. arXiv:2011.01856 [cs].

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, June 2024. URL http://arxiv.org/abs/2406.04325. arXiv:2406.04325 [cs].

Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. PLACES: Prompting Language Models for Social Conversation Synthesis, February 2023a. URL http://arxiv.org/abs/2302.03269. arXiv:2302.03269 [cs].
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. SummScreen: A Dataset for Abstractive Screenplay Summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 8602–8615, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10/gtsqxx. URL https://aclanthology.org/2022.acl-long.589.

Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention, June 2019. URL http://arxiv.org/abs/1905.12866. arXiv:1905.12866 [cs].

Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data. In Trevor Cohn, Yulan He, and Yang Liu (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2020*, pp. 1026–1036, Online, November 2020b. Association for Computational Linguistics. doi: 10/gpmd4x. URL https://aclanthology.org/2020.findings-emnlp.91.

Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A Theorem-driven Question Answering dataset, December 2023b. URL http://arxiv.org/abs/2305.12524. arXiv:2305.12524 [cs].

Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can language models be instructed to protect personal information? 2023c.

Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A Real-Life Scenario Dialogue Summarization Dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pp. 5062–5074, Online, August 2021c. Association for Computational Linguistics. doi: 10/gtsqxs. URL https://aclanthology.org/2021.findings-acl.449.

Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, and William Yang Wang. KETOD: Knowledge-Enriched Task-Oriented Dialogue, May 2022b. URL http://arxiv.org/abs/2205.05589. arXiv:2205.05589 [cs].

Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation, March 2022. URL http://arxiv.org/abs/2108.06712. arXiv:2108.06712 [cs].

Ssu Chiu, Maolin Li, Yen-Ting Lin, and Yun-Nung Chen. SalesBot: Transitioning from Chit-Chat to Task-Oriented Dialogues. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 6143–6158, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10/gtsqxw. URL https://aclanthology.org/2022.acl-long.425.

Hasna Chouikhi, Manel Aloui, Cyrine Ben Hammou, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning, July 2024. URL http://arxiv.org/abs/2407.02147. arXiv:2407.02147 [cs].

Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, February 2017. URL http://arxiv.org/abs/1703.00056. arXiv:1703.00056 [cs, stat].

Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, and Gerhard Weikum. Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Ju-dicious Context Expansion. In *Proceedings of the 28th ACM International Conference on Information and Knowledge Management*, pp. 729–738, November 2019. doi: 10/gkz233. URL http://arxiv.org/abs/1910.03262. arXiv:1910.03262 [cs].

Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang, Yu-Wing Tai, and Chi-Keung Tang. HAA500: Human-Centric Atomic Action Dataset with Curated Videos, August 2021. URL http://arxiv.org/abs/2009.05224. arXiv:2009.05224 [cs, eess].
Christopher Cieri, David Miller, and Kevin Walker. The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text. In Maria Teresa Lino, Maria Francisca Xavier, Fátima Ferreira, Rute Costa, and Raquel Silva (eds.), *Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)*, Lisbon, Portugal, May 2004. European Language Resources Association (ELRA). URL http://www.lrec-conf.org/proceedings/lrec2004/pdf/767.pdf.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, May 2019. URL http://arxiv.org/abs/1905.10044. arXiv:1905.10044 [cs].

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, March 2018. URL http://arxiv.org/abs/1803.05457. arXiv:1803.05457 [cs].

Steven Coats. A Corpus of Regional American Language from YouTube. In *Proceedings of the Digital Humanities in the Nordic Countries 5th Conference*, 2019. URL https://www.semanticscholar.org/paper/A-Corpus-of-Regional-American-Language-from-YouTube-Coats/bc428db824d261794a7e081a53c4315b8e02f855.

Steven Coats. Dialect corpora from youtube. *Language and linguistics in a complex world*, 2023.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL http://arxiv.org/abs/2110.14168. arXiv:2110.14168 [cs].

Samantha Cole. Nvidia sued for scraping youtube after 404 media investigation. *404 Media*, August 2024. URL https://www.404media.co/nvidia-sued-for-scraping-youtube-after-404-media-investigation/.

Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech, May 2022. URL http://arxiv.org/abs/2205.12446. arXiv:2205.12446 [cs, eess] version: 1.

Sam Coope, Tyler Farghly, Daniela Gerz, Ivan Vulić, and Matthew Henderson. Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations, July 2020. URL http://arxiv.org/abs/2005.08866. arXiv:2005.08866 [cs].

Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces, December 2018. URL http://arxiv.org/abs/1805.10190. arXiv:1805.10190 [cs].

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms. *arXiv preprint*, 2024.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling Egocentric Vision: The EPIC-KITCHENS Dataset, July 2018. URL http://arxiv.org/abs/1804.02748. arXiv:1804.02748 [cs].

Verna Dankers, Christopher Lucas, and Ivan Titov. Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 3608–3626, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10/g6k6xn. URL https://aclanthology.org/2022.acl-long.252.
Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In *2013 IEEE Conference on Computer Vision and Pattern Recognition*, pp. 2634–2641, Portland, OR, USA, June 2013. ISBN 978-0-7695-4989-7. doi: 10/gtsqzr. URL http://ieeexplore.ieee.org/document/6619184/.

Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota Smarthome: Real-World Activities of Daily Living. In *2019 IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 833–842, Seoul, Korea (South), October 2019. ISBN 978-1-72814-803-8. doi: 10/ghfjc7. URL https://ieeexplore.ieee.org/document/9008135/.

Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning. September 2019. URL http://arxiv.org/abs/1908.05803. arXiv:1908.05803 [cs].

Emilia David. Ai image training dataset found to include child sexual abuse imagery. *The Verge*, December 2023. URL https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford. 7:57 AM PST.

Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops*, pp. 52–59, 2019.

Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra. Earnings-22: A Practical Benchmark for Accents in the Wild, March 2022. URL http://arxiv.org/abs/2203.15591. arXiv:2203.15591 [cs].

Ugur Demir, Yogesh S. Rawat, and Mubarak Shah. TinyVIRAT: Low-resolution Video Action Recognition, July 2020. URL http://arxiv.org/abs/2007.07355. arXiv:2007.07355 [cs, eess].

Isin Demirsahin, Oddur Kjartansson, Alexander Gutkin, and Clara Rivera. Open-source Multi-speaker Corpora of the English Accents in the British Isles. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pp. 6532–6541, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.804.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web, December 2023. URL http://arxiv.org/abs/2306.06070. arXiv:2306.06070 [cs].

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Fine-tuning of Quantized LLMs, May 2023. URL http://arxiv.org/abs/2305.14314. arXiv:2305.14314 [cs].

Ashwin Devaraj, Iain J. Marshall, Byron C. Wallace, and Junyi Jessy Li. Paragraph-level Simplification of Medical Texts, April 2021. URL http://arxiv.org/abs/2104.05767. arXiv:2104.05767 [cs].

Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. MS2: Multi-Document Summarization of Medical Studies, November 2021. URL http://arxiv.org/abs/2104.06486. arXiv:2104.06486 [cs].

Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST-C: a Multilingual Speech Translation Corpus. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 2012–2017, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10/gtsqzk. URL https://aclanthology.org/N19-1202.
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jurgen Gall, Rainer Stiefelhagen, and Luc Van Gool. Large Scale Holistic Video Understanding, December 2020. URL http://arxiv.org/abs/1904.11451. arXiv:1904.11451 [cs].

Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims, January 2021. URL http://arxiv.org/abs/2012.00614. arXiv:2012.00614 [cs].

Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. The Second Conversational Intelligence Challenge (ConvAI2), January 2019a. URL http://arxiv.org/abs/1902.00098. arXiv:1902.00098 [cs].

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of Wikipedia: Knowledge-Powered Conversational agents, February 2019b. URL http://arxiv.org/abs/1811.01241. arXiv:1811.01241 [cs].

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations, May 2023. URL http://arxiv.org/abs/2305.14233. arXiv:2305.14233 [cs].

Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286–1305, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.

Moussa Doumbouya, Lisa Einstein, and Chris Piech. Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Users, April 2021. URL http://arxiv.org/abs/2104.13083. arXiv:2104.13083 [cs].

Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale, September 2018. URL http://arxiv.org/abs/1808.10583. arXiv:1808.10583 [cs].

Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin. Omni-sourced Webly-supervised Learning for Video Recognition, August 2020. URL http://arxiv.org/abs/2003.13042. arXiv:2003.13042 [cs].

Omar Einea, Ashraf Elnagar, and Ridhwan Al Debsi. SANAD: Single-label Arabic News Articles Dataset for automatic text categorization. Data in Brief, 25:104076, August 2019. ISSN 23523409. doi: 10/g6k6cn. URL https://linkinghub.elsevier.com/retrieve/pii/S2352340919304305.

Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis (eds.), Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 207–219, Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi: 10/gtsqx4. URL https://aclanthology.org/W17-5526.

Ibrahim Abu El-khair. 1.5 billion words Arabic Corpus, November 2016. URL http://arxiv.org/abs/1611.04033. arXiv:1611.04033 [cs].

Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? In The Twelfth International Conference on Learning Representations, 2023.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision, 2022.

Ashraf Elnagar and Omar Einea. BRAD 1.0: Book reviews in Arabic dataset. In *2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA)*, pp. 1–8, Agadir, Morocco, November 2016. IEEE. ISBN 978-1-5090-4320-0. doi: 10/g6k6jm. URL http://ieeexplore.ieee.org/document/7945800/.

Hady ElSahar and Samhaa R. El-Beltagy. Building Large Arabic Multi-domain Resources for Sentiment Analysis. In Alexander Gelbukh (ed.), *Computational Linguistics and Intelligent Text Processing*, pp. 23–34, Cham, 2015. Springer International Publishing. ISBN 978-3-319-18117-2. doi: 10/g6k58r.

Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences, December 2020. URL http://arxiv.org/abs/2012.15738. arXiv:2012.15738 [cs].

Dave Epstein, Boyuan Chen, and Carl Vondrick. Oops! Predicting Unintentional Action in Video. In *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 916–926, Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817-168-5. doi: 10/ghbckh. URL https://ieeexplore.ieee.org/document/9156404/.

Mihail Eric and Christopher D. Manning. Key-Value Retrieval Networks for Task-Oriented Dialogue, July 2017. URL http://arxiv.org/abs/1705.05414. arXiv:1705.05414 [cs].

Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur. MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines, December 2019. URL http://arxiv.org/abs/1907.01669. arXiv:1907.01669 [cs].

Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models, 2023. URL https://arxiv.org/abs/2302.03011.

Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pp. 6866–6880, Online, August 2021. Association for Computational Linguistics. doi: 10/gmf9qs. URL https://aclanthology.org/2021.acl-long.535.

Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model, June 2019. URL http://arxiv.org/abs/1906.01749. arXiv:1906.01749 [cs].

Fahim Faisal, Yinkai Wang, and Antonios Anastasopoulos. Dataset geography: Mapping language data to language users. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 3381–3411, 2022.

Manaal Faruqui and Dipanjan Das. Identifying Well-formed Natural Language Questions, August 2018. URL http://arxiv.org/abs/1808.09419. arXiv:1808.09419 [cs].

Guy Feigenblat, Chulaka Gunasekara, Benjamin Sznajder, Sachindra Joshi, David Konopnicki, and Ranit Aharonov. TWEETSUMM – A Dialog Summarization Dataset for Customer Service, November 2021. URL http://arxiv.org/abs/2111.11894. arXiv:2111.11894 [cs].

Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 6162–6176, 2021. doi: 10/g6k938. URL http://arxiv.org/abs/2109.12595. arXiv:2109.12595 [cs].
David F. Fouhey, Wei-cheng Kuo, Alexei A. Efros, and Jitendra Malik. From Lifestyle Vlogs to Everyday Interactions, December 2017. URL http://arxiv.org/abs/1712.02310. arXiv:1712.02310 [cs].

Yihui Fu, Luyao Cheng, Shubo Lv, Yukai Jv, Yuxiang Kong, Zhuo Chen, Yanxin Hu, Lei Xie, Jian Wu, Hui Bu, Xin Xu, Jun Du, and Jingdong Chen. AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario, August 2021. URL http://arxiv.org/abs/2104.03603. arXiv:2104.03603 [cs, eess].

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei W Koh, Olga Saukh, Alexander J Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beau-
mont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 27092–27112. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/56332d41d55ad7ad8024aac625881be7-Paper-Datasets_and_Benchmarks.pdf.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. *Advances in Neural Information Processing Systems*, 36, 2024.

Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Aswanth Kumar M, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M. Khapra, Raj Dabre, Rudra Murthy, and Anoop Kunchukuttan. Airavata: Introducing Hindi Instruction-tuned LLM, February 2024. URL http://arxiv.org/abs/2401.15006. arXiv:2401.15006 [cs].

Ygor Gallina, Florian Boudin, and Beatrice Daille. KPTimes: A Large-Scale Dataset for Keyphrase Generation on News Documents. In Kees van Deemter, Chenghua Lin, and Hiroya Takamura (eds.), *Proceedings of the 12th International Conference on Natural Language Generation*, pp. 130–135, Tokyo, Japan, October 2019. Association for Computational Linguistics. doi: 10/g6k64k. URL https://aclanthology.org/W19-8617.

Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage, November 2021. URL http://arxiv.org/abs/2111.09344. arXiv:2111.09344 [cs, stat].

Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions. In Chu-Ren Huang and Dan Jurafsky (eds.), *Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)*, pp. 340–348, Beijing, China, August 2010. Coling 2010 Organizing Committee. URL https://aclanthology.org/C10-1039.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. *arXiv preprint arXiv:2101.00027*, 2020.

Garofolo, John S., Lamel, Lori F., Fisher, William M., Pallett, David S., Dahlgren, Nancy L., Zue, Victor, and Fiscus, Jonathan G. TIMIT Acoustic-Phonetic Continuous Speech Corpus, 1993. URL https://catalog.ldc.upenn.edu/LDC93S1. Artwork Size: 715776 KB Pages: 715776 KB.

Murilo Gleyson Gazzola, Sidney Evaldo Leal, and Sandra Maria Aluísio. Predição da complexidade textual de recursos educacionais abertos em português. *Symposium in Information and Human Language Technology - STIL*, 2019. URL https://repositorio.usp.br/item/002971271.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. *Communications of the ACM*, 64(12): 86–92, 2021.

Elizabeth Jasmi George and Radhika Mamidi. Conversational implicatures in English dialogue: Annotated dataset, November 2019. URL http://arxiv.org/abs/1911.10704. arXiv:1911.10704 [cs].

Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michał Lis, Eshan Singhal, Nikola Mrkšić, Tsung-Hsien Wen, and Ivan Vulić. Multilingual and Cross-Lingual Intent Detection from Spoken Data, April 2021. URL http://arxiv.org/abs/2104.08524. arXiv:2104.08524 [cs].

Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion, March 2019. URL http://arxiv.org/abs/1902.10526. arXiv:1902.10526 [cs].

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), *Proceedings of the 2nd Workshop on New Frontiers in Summarization*, pp. 70–79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10/gmjqgr. URL https://aclanthology.org/D19-5409.

J.J. Godfrey, E.C. Holliman, and J. McDaniel. SWITCHBOARD: telephone speech corpus for research and development. In [*Proceedings* ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 517–520 vol.1, San Francisco, CA, USA, 1992. IEEE. ISBN 978-0-7803-0532-8. doi: 10/fp48kw. URL http://ieeexplore.ieee.org/document/225858/.

Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. RumourEval 2019: Determining Rumour Veracity and Support for Rumours, September 2018. URL http://arxiv.org/abs/1809.06683. arXiv:1809.06683 [cs].

Tear Gosling, Alpin Dale, and Yinhe Zheng. PIPPA: A Partially Synthetic Conversational Dataset, August 2023. URL http://arxiv.org/abs/2308.05884. arXiv:2308.05884 [cs].

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. *Transactions of the Association for Computational Linguistics*, 10:522–538, 2022.

Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyńska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense, June 2017. URL http://arxiv.org/abs/1706.04261. arXiv:1706.04261 [cs].

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4D: Around the World in 3,000 Hours of Egocentric Video, March 2022. URL http://arxiv.org/abs/2110.07058. arXiv:2110.07058 [cs].
Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions, April 2018. URL http://arxiv.org/abs/1705.08421. arXiv:1705.08421 [cs].

Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases. In *Proceedings of the Web Conference 2021*, pp. 3477-3488, April 2021. doi: 10/gnnfbt. URL http://arxiv.org/abs/2011.07743. arXiv:2011.07743 [cs].

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. *ACM Transactions on Computing for Healthcare*, 3(1):1-23, January 2022. ISSN 2691-1957, 2637-8051. doi: 10/gnmkjx. URL http://arxiv.org/abs/2007.15779. arXiv:2007.15779 [cs].

Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, and Xiangang Li. DiDiSpeech: A Large Scale Mandarin Speech Corpus, February 2021. URL http://arxiv.org/abs/2010.09275. arXiv:2010.09275 [eess].

Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui. Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering, June 2021. URL http://arxiv.org/abs/2106.04016. arXiv:2106.04016 [cs].

Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. MMQA: A Multi-domain Multi-lingual Question-Answering Framework for English and Hindi. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan, May 2018a. European Language Resources Association (ELRA). URL https://aclanthology.org/L18-1440.

Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. Semantic Parsing for Task Oriented Dialog using Hierarchical Representations, October 2018b. URL http://arxiv.org/abs/1810.07942. arXiv:1810.07942 [cs].

Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating Summaries from User Videos. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), *Computer Vision - ECCV 2014*, volume 8695, pp. 505-520. Springer International Publishing, Cham, 2014. ISBN 978-3-319-10583-3. doi: 10.1007/978-3-319-10584-0_33. URL http://link.springer.com/10.1007/978-3-319-10584-0_33. Series Title: Lecture Notes in Computer Science.

David Ha and Douglas Eck. A Neural Representation of Sketch Drawings, May 2017. URL http://arxiv.org/abs/1704.03477. arXiv:1704.03477 [cs, stat].

Hessel Haagsma, Johan Bos, and Malvina Nissim. MAGPIE: A Large Corpus of Potentially Idiomatic Expressions. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pp. 279-287, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.35.

Amirhossein Habibian, Thomas Mensink, and Cees G.M. Snoek. VideoStory: A New Multimedia Embedding for Few-Example Recognition and Translation of Events. In *Proceedings of the 22nd ACM international conference on Multimedia*, pp. 17-26, Orlando Florida USA, November 2014. ACM. ISBN 978-1-4503-3063-3. doi: 10/ggs25n. URL https://dl.acm.org/doi/10.1145/2647868.2654913.
Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. MedAlpaca – An Open-Source Collection of Medical Conversational AI Models and Training Data, October 2023. URL http://arxiv.org/abs/2304.08247. arXiv:2304.08247 [cs].

Moshe Hazoom, Vibhor Malik, and Ben Bogin. Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data, June 2021. URL http://arxiv.org/abs/2106.05006. arXiv:2106.05006 [cs].

He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling Strategy and Generation in Negotiation Dialogues, August 2018. URL http://arxiv.org/abs/1808.09637. arXiv:1808.09637 [cs].

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 961–970, Boston, MA, USA, June 2015. IEEE. ISBN 978-1-4673-6964-0. doi: 10/gfsvdw. URL http://ieeexplore.ieee.org/document/7298698/.

Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS Spoken Language Systems Pilot Corpus. In *Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990*, 1990. doi: 10/cz3442. URL https://aclanthology.org/H90-1021.

Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset, November 2022. URL http://arxiv.org/abs/2207.00220. arXiv:2207.00220 [cs].

Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. arXiv preprint arXiv:2303.15715, 2023.

Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing Moments in Video with Temporal Language. In *Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 1380–1390, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10/gtsqzq. URL https://aclanthology.org/D18-1168.

Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review, November 2021. URL http://arxiv.org/abs/2103.06268. arXiv:2103.06268 [cs].

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020. URL https://arxiv.org/abs/2010.14701.

François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation. In *Lecture Notes in Computer Science*, volume 11096, pp. 198–208. Springer, Cham, 2018. doi: 10.1007/978-3-319-99579-3_21. URL http://arxiv.org/abs/1805.04699. arXiv:1805.04699 [cs].

Carlos Daniel Hernandez Mena, David Erik Mollberg, Michal Borsky, and Jon Gudnason. Samrótur Children: An Icelandic Speech Corpus. In *Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pp. 995–1002, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.105.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. *arXiv preprint arXiv:2203.15556*, 2022.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization, September 2020. URL http://arxiv.org/abs/2003.11080. arXiv:2003.11080 [cs].

Albert Huang. 'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities, February 2022. URL http://arxiv.org/abs/2202.13581. arXiv:2202.13581 [cs].

Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning, September 2019. URL http://arxiv.org/abs/1909.00277. arXiv:1909.00277 [cs].

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient Attentions for Long Document Summarization, April 2021. URL http://arxiv.org/abs/2104.02112. arXiv:2104.02112 [cs].

Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. MovieNet: A Holistic Dataset for Movie Understanding, July 2020. URL http://arxiv.org/abs/2007.10937. arXiv:2007.10937 [cs].

Moustafa Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A Hierarchical Deep Temporal Model for Group Activity Recognition, April 2016. URL http://arxiv.org/abs/1511.06040. arXiv:1511.06040 [cs].

Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS Challenge on Action Recognition for Videos "in the Wild". *Computer Vision and Image Understanding*, 155:1–23, February 2017. ISSN 10773142. doi: 10/f9rwnr. URL http://arxiv.org/abs/1604.06182. arXiv:1604.06182 [cs].

John J. Irwin, Khanh G. Tang, Jennifer Young, Chinzorig Dandarchuluun, Benjamin R. Wong, Munkhzul Khurelbaatar, Yurii S. Moroz, John Mayfield, and Roger A. Sayle. ZINC20—A Free Ultralarge-Scale Chemical Database for Ligand Discovery. *Journal of Chemical Information and Modeling*, 60(12):6065–6073, December 2020. ISSN 1549-9596. doi: 10/gmjg8b. URL https://doi.org/10.1021/acs.jcim.0c00675. Publisher: American Chemical Society.

Keith Ito and Linda Johnson. The LJ Speech Dataset, 2017. URL https://keithito.com/LJ-Speech-Dataset.

Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient Long-Text Understanding with Short-Text Models, December 2022. URL http://arxiv.org/abs/2208.00748. arXiv:2208.00748 [cs].

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a Neural Semantic Parser from User Feedback, April 2017. URL http://arxiv.org/abs/1704.08760. arXiv:1704.08760 [cs].

Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based Neural Structured Learning for Sequential Question Answering. In Regina Barzilay and Min-Yen Kan (eds.), *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 1821–1831, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10/gf6nx8. URL https://aclanthology.org/P17-1167.

Tahir Javed, Kaushal Santosh Bhogale, Abhigyan Raman, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages, December 2022. URL http://arxiv.org/abs/2208.11761. arXiv:2208.11761 [cs, eess].
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities, July 2020. URL http://arxiv.org/abs/2007.15781. arXiv:2007.15781 [cs].

Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. Neural CRF Model for Sentence Alignment in Text Simplification, August 2021. URL http://arxiv.org/abs/2005.02324. arXiv:2005.02324 [cs].

Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification, November 2020. URL http://arxiv.org/abs/2011.03088. arXiv:2011.03088 [cs].

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams, September 2020. URL http://arxiv.org/abs/2009.13081. arXiv:2009.13081 [cs].

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA: A Dataset for Biomedical Research Question Answering, September 2019. URL http://arxiv.org/abs/1909.06146. arXiv:1909.06146 [cs, q-bio].

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, May 2017. URL http://arxiv.org/abs/1705.03551. arXiv:1705.03551 [cs].

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095, 2020.

Juraj Juraska, Kevin K. Bowden, and Marilyn Walker. ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation, October 2019. URL http://arxiv.org/abs/1910.12129. arXiv:1910.12129 [cs].

Tomasz Jurczyk, Michael Zhai, and Jinho D. Choi. SelQA: A New Benchmark for Selection-based Question Answering, October 2016. URL http://arxiv.org/abs/1606.08513. arXiv:1606.08513 [cs].

Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and Evaluating Contextual Embedding of Source Code, August 2020. URL http://arxiv.org/abs/2001.00059. arXiv:2001.00059 [cs].

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.

Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in ml-based science. arXiv preprint arXiv:2207.07048, 2022.

Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-Scale Video Classification with Convolutional Neural Networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1725-1732, Columbus, OH, USA, June 2014. IEEE. ISBN 978-1-4799-5118-5. doi: 10/gf4hdn. URL https://ieeexplore.ieee.org/document/6909619.

Nikolay Karpov, Alexander Denisenko, and Fedor Minkin. Golos: Russian Dataset for Speech Research, June 2021. URL http://arxiv.org/abs/2106.10161. arXiv:2106.10161 [eess].

Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the Difference that Makes a Difference with Counterfactually-Augmented Data, February 2020. URL http://arxiv.org/abs/1909.12434. arXiv:1909.12434 [cs, stat].