Bridging the Data Provenance Gap Across Text, Speech, and Video
Authors: Shayne Longpre
Nikhil Singh
Manuel Cherep
Kushagra Tiwary
Joanna Materzynska
William Brannon
Robert Mahari
Naana Obeng-Marnu
Manan Dey
Mohammed Hamdy
Nayan Saxena
Ahmad Mustafa Anis
Emad A. Alghamdi
Vu Minh Chien
Da Yin
Kun Qian
Yizhi LI
Minnie Liang
An Dinh
Shrestha Mohanty
Deividas Mataciunas
Tobin South
Jianguo Zhang
Ariel N. Lee
Campbell S. Lund
Christopher Klamm
Damien Sileo
Diganta Misra
Enrico Shippole
Kevin Klyman
Lester James Validad Miranda
Niklas Muennighoff
Seonghyeon Ye
Seungone Kim
Vipul Gupta
Vivek Sharma
Xuhui Zhou
Caiming Xiong
Luis Villa
Stella Biderman
Alex Pentland
Sara Hooker
Jad Kabbara
Venue: ICLR 2025 Poster (2025)
Keywords: training data
audit
speech
video
text
Primary area: datasets and benchmarks
TL;DR: An audit of text, speech, and video training sets for their sources, use restrictions, and representation.

Abstract: Progress in AI is driven largely by the scale and quality of training data. Despite this, there is a deficit of empirical analysis examining the attributes of well-established datasets beyond text. In this work we conduct the largest and first-of-its-kind longitudinal audit across modalities --- popular text, speech, and video datasets --- from their detailed sourcing trends and use restrictions to their geographical and linguistic representation. Our manual analysis covers nearly 4000 public datasets between 1990-2024, spanning 608 languages, 798 sources, 659 organizations, and 67 countries. We find that multimodal machine learning applications have overwhelmingly turned to web-crawled, synthetic, and social media platforms, such as YouTube, for their training sets, eclipsing all other sources since 2019. Secondly, tracing the chain of dataset derivations we find that while less than 33% of datasets are restrictively licensed, over 80% of the source content in widely-used text, speech, and video datasets, carry non-commercial restrictions. Finally, counter to the rising number of languages and geographies represented in public AI training datasets, our audit demonstrates measures of relative geographical and multilingual representation have failed to significantly improve their coverage since 2013. We believe the breadth of our audit enables us to empirically examine trends in data sourcing, restrictions, and Western-centricity at an ecosystem-level, and that visibility into these questions are essential to progress in responsible AI. As a contribution to ongoing improvements in dataset transparency and responsible use, we release our entire multimodal audit, allowing practitioners to trace data provenance across text, speech, and video.

Published as a conference paper at ICLR 2025
BRIDGING THE DATA PROVENANCE GAP ACROSS TEXT, SPEECH, AND VIDEO
Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, Nayan Saxena, Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Da Yin, Kun Qian, Yizhi Li, Minnie Liang. An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang. Ariel N. Lee. Campbell S. Lund, Christopher Klamm, Damien Sileo, Diganta Misra, Enrico Shippole, Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Vipul Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong. Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, Jad Kabbara
The Data Provenance Initiative
ABSTRACT
Progress in is driven largely by the scale and quality of training data. Despite this, there is a deficit of empirical analysis examining the attributes of well-established datasets beyond text. In this work we conduct the largest and first-of-its-kinn kind longitudinal audit across modalities-poppee from their detailed sourcing trends and use restrictions to their geographical and linguistic representation. Our manual analysis covers nearly 4000 public datasets between 1990-2024, spanning 608 languages, 798 sources, 659 organizations, and 67 countries. We find that multimodal machine learning applications have overwhelmingly turned to web-crawled, synthetic, and social media platforms, such YouTuue, for their training sets, eclipsing all other sources since 2019. Secondly, tracing the chain of dataset derivations we find that while less than 33% of datasets are restrictively licensed, over 80% of the source content in widely- used text, speech, and video datasets, carry non-commercial restrictions, Finally, counter to the rising number of languages and geographies represented in public Al training datasets, our audit demonstrates measures of relative geographical and multilingual representation have failed significantly improve their coverage since 2013. We believe the breadth of our audit enables us to empirically examine trends in data sourcing, restrictions, and Western- centriciiy at an ecosystem-lews that visibility into these questions are essential to progress in responsible AI. As a contribution to ongoing improvements in dataset transparency and responsible use, release our entire multimodal audit, allowing practitioners to trace data provenance across text, speech, and video.
text, speech, and video datasets-
and
INTRODUCTION
The capabilities and flaws of multimodal foundation models are often directly attributable their training data (Carlini et al., 2023a Rando et al., 2022, Carlini et al.) 2023b; Parmar et al. 2024]
Liu et al.) 2023b ai Dai et al. (024)) While the importance of data measurement has been widely
established by prior work (Gadre et al. 2024), so has a prevailing absence of data documentation
(Gebru et al. 2021 Bender & Friedman 2018 transparency (Bommasani et al. 2023), and detailed understanding (Dodge et | 2021 Bandy & Vincent 2021 Sambasivan et al. (021) especially for modalities other than text. A lack of thorough data analysis has led to significant challenges, including privacy issues (Subramani al. 2023). retracting datasets with harmful content (Birhane et al. 2021
David 2023), adversarially bypassing safety filters (Rando et al. 2022), facial recognition bias with respect gender and skin type (Buolamwini & Gebru 2018a), gender bias in hiring (Chang) 2023). benchmark contamination from overlapping train and test sets (Lee et al. (023a)., and challenges in copyright (Henderson et al. 2023). Understanding data provenance aid mitigation attempts to

---

Published as a conference paper at ICLR 2025
DATASETS
SOURCES
CREATOR ORGS
LANGUAGES FAMILIES
TASKS
LICENSES
SIZE
DOMAINS
#
COUNTRIES
#
TEXT
3717
2.IT
713
534
60
502
395
50
SPEECH
95
775k
51
16
124
29
260
36
18
19
VIDEO
104
1.13M
44
24
101
23
33
11
TOTAL
3916
798
83
659
67
608
37
443
55
Table We quantify the breadth of our audit, including the total number of datasets (#), their size in tokens or hours, the sources, domains, creator organizations, countries, languages, tasks, and licenses. In aggregate, we audited 3916 datasets from 659 organizations in 67 countries, spanning 2.1T tokens, and 1.9M hours. We cataloged nearly 798 unique sources, 443 tasks, and 55 licenses.
reduce model bias and toxicity (Welbl al. 2021 1 Pozzobon et al. 2023 address representation in
data (Xu et al. 202T). contamination (Elazar et al. 2023), and quality (Kreutzer et al., 2022; Marion
et al. 2023), well as practical challenges with identifying copyright-free and permissively licensed
sets (Min et al. 2023).
Despite the urgent need for the provenance and characteristics of widely used datasets, the majority of
attention to date has centered on text datasets (Elazar et al.] 2023 Longpre et al. 2024b), or a single
feature such as prevalence of hate content (Dodge al. [02] Birhane al. 2021 In contrast, in this work, we will critically examine several provenance features of data across text, speech, and video. We conduct the largest and most comprehensive multimodal audit of AI data, to date, reviewing nearly 4000 datasets between 1990-2024, covering 443 unique tasks, 608 languages, derived from 798 original sources, and constructed by 659 organizations, spanning 67 countries, over IT tokens of text, and 1. 1.9M hours of speech and video content (see Table 1).
There is an unprecedented acceleration in the development of multimodal Al systems, making all the more urgent an understanding of the datasets that underpin these breakthroughs. Our extensive collection of features from unstructured academic papers, websites, and repositories enables us to provide empirical grounding to ambitious set of research questions surrounding data sourcing trends, intended licenses, and geographical and linguistic representation. Our key findings include:
1. Multimodal data is increasingly sourced from the web, social media platforms, or synthetically generated; rather than more curated sources such as movies, audiobooks or manually collected. These sources comprise the vast majority of text tokens, as well as speech and video hours in public data. However, while social media platforms provide data scale, heterogeneity and freshness by nature, they are also particularly prone to anti-crawling, copyright, privacy, and factuality concerns.
2. Whereas only 25% of text, speech, and video datasets have non-commercial licenses, over 80% of content from each modality carries undocumented restrictions in the dataset's sources. Dataset licenses are inconsistent with their source's 's restrictions for over 55% of content. Our audit provides the tools for multimodal developers identify dataset restrictions, and apply their own standards.
3. Geographical and linguistic representation have not improved for decade, across the data ecosystem. While the amount of data from under-represented creators and languages increases each year, to over 600 languages and 60 countries in 2024, their relative represen- lation remains consistently western-centric, with no significant improvements from > 0.7 Gini coefficients. While Africa and South America organizations account for < 0.2% of all modality content, North America European organizations span 93% of text tokens and 60%+ hours of speech and video.
Our work provides critical insights into the landscape of available multimodal data. We release the entire audit, collected data, and analysis tools, which we believe will bring immense value for data creators, developers, and researchers interested in promoting the responsible development of AI systems and analysis of the AI data ecosystem.
METHODOLOGY
While many prior works have surveyed the dataset ecosystem (Albalak et al. 2024 Liu et al. 2024c
Malik et al. 2021 Prabhavalkar al. 2023 Li et al. 2019b), few empirically examine data corpora
2

---

Published as a conference paper at ICLR 2025
at scale, and those that do focus present more narrow focus around specific feature like geographic bias or hate content(Birhane (Birhane et al. 2023| LieMillan-Majpie et al. a |2022al Shankar et al., 2017 or a
single modality (Dodge et al. 2022 Caswell et al. 2021 Blazar et al. [023]] Longpre al. 2024b). The goal of this work is to provide an empirical, ecosystem-level. and multimodal analysis of widely used training datasets (Cen al., 2023) Our audit focuses on text, speech, and video, as prominent
data modalities behind modern multimodal systems, such as Sora, Whisper, Gemini, GPT-4o. and
others (Brooks al., 2024; Zheng et al., 2024b Radford et al. 2023 Peng et al., 2023 Team
et al., 2023] OpenAl| 2024). Since training data for modalities can often be independent, multimodal
models tend to interleave training batches with different combinations of one or two modalities
(Aghajanyan et al. 2023). As such, we focus our analysis on datasets that represent one or pair of these modalities.
Annotation Features& & Methodology In particular, we analyze data trends for the state of data permissions (licenses and terms), sourcing (the web, human annotation, and synthetic generation). and representation (of tasks, organizations, languages, and countries). We adopt Longpre et al. (2024b)ss 's methodology, including the license annotation taxonomy and process, to manually audit these features precisely and rigorously. We go beyond prior work, which considers dataset licenses, by extending the taxonomy to consider the terms of use of the sources of the dataset, either from models used to generate synthetic data (e.g. OpenAl's non-compete clause] or Meta's acceptable use policy for Llama 3. 3. or the source's s policy on content restrictions, which can be conveyed in the form of a license, terms of use, content policy a website (Klyman 2024). For each dataset, the source terms are annotated as Unrestricted, Unspecified, Source Closed or Model Closed, as defined in Table 2 For Figure 2
we combine Source Closed and Model Closed into Restricted.
As with prior work (Longpre et al., 2024b| we engage domain experts for these annotation tasks Al researchers whose work pertains to the modality and topic. Because many datasets are iteratively re-packaged before they appear in their final form and often shared on popular dataset marketplace klces like HuggingFace, Papers with Code or Github, prior work has found that relevant licensing terms or sourcing information for AI training data is frequently omitted (Longpre et al. 2024b). To ensure we collect this information, we require full trace of metadata back to their original sources (sometimes a chain of github repositories, websites, or academic papers). This search can be onerous, especially for terms and licenses, but ensures rigor in the results. Table Jabbleranneeratte the full statistics of our audit. All annotations and analysis code will be made publicly available on release.
Scope & Dataset Selection For each modality, we define the scope of the audit (detailed separately below), then aggregate resources to distill a list of relevant datasets. The scope is focused on (a) publicly available datasets, (b) widely used tasks in the context of caneral-purpose model development, and (c) relevance to generative tasks. However, we do consider classification-basee datasets in text, speech, and video that can and are frequently re-purposed for generative uses (e.g. instruction tuning). Within the defined audit scope, we use a mix of the HuggingFace Datasets platform, survey papers, survey repositories, workshop proceedings, and expert review to accumulate relevant datasets. More detail about the dataset selection and collection process is given for each modality below. Each modality requires its own independent process, by virtue of their community dataset ecosystems being unique (discussed in Section 4 Note that text has wider heterogeneity of published publicly available datasets than speech or video. Typically those datasets have been aggregated into large, standardized text-to-text collections, and such we trace both these Text (Collections) and their constituent Text (Datasets). All datasets are described, linked, and attributed in Appendix [
2.1 TEXT
Scope We focus on providing an extensive audit for post-training datasets, used in training language models. We include single and multi-turn formats, encompassing both datasets typically used for instruction finetuning (SFT) and preference alignment RRaailoo et al. (2023). This scope reflects the prominent role of general-pun -prrpoe language models, which benefit from multi-task training on heterogeneous collections that span variety of linguistic, reasoning, and knowledge intensive tasks
like question answering, coding, tool use, translation, and classification (Wei et al.) 202T Ouyang et al. 2022).
OpenAl Terms of Use
Jama 3.1 Acceptaale Use Policy
3

---

Published as a conference paper at ICLR 2025
Dataset Selection We expand the study conducted by the Data Provenance Collection Longpre et 2024b), from 44 dataset collections (of 1858 supervised text datasets) to a superset of 108 collections of 3717 datasets, prioritizing recent, popular publicly available HuggingFace Datasets introduced between 2022 and April 2024. Our collection sourced popular datasets from recent survey papers (Albalak al., 2024: Liu al., 2024c) and tools (Longpre et al.) (0024)). We additionally reviewed HuggingFace Datasets' most downloaded datasets every month, from April to July 2024, under the Natural Language Processing category, well the SFT/DPO datasets associated with popular open model releases. We also drew from major multilingual data repositories, including the SEACrowd Catalogue (Lovenia et al. (024). the Masader Arabic Data Catalogue (Alyafeai et al.) 2022), A14Bharat (Kunchukuttan et al. 20200, and the Aya Collection (Singh et al.) 2024a). Lastly, our list of datasets was reviewed and supplemented by language model experts to fill in notable omissions. In total, we trace the provenance and features of 3715 13 text datasets from 108 collections, covering 395 popular tasks, spanning from 1994 to 2024.
2.2 SPEECH
Scope We audit speech datasets for which automatic speech recognition (ASR) was noted as a pri- mary task. We focus ASR datasets because: (1) ASR is fundamental to many speech technologies, including dictation tools, voice assistants, and chatbots (Aksënova et al., 2021 Zhang et al.) 2022c); (2) large-scale speech datasets are typically designed for ASR (Li al. (023b); ; (3) ASR data follows standardized formats, making comparisons easier (c.g., corpus of audio clips paired with text); and
(4) ASR data can often be reused for other tasks like text to speech (TTS) (Ito & Johnson 2017) or language identification (Ardila et al. 2020b).
Dataset Selection To curate a representative sample of popular ASR datasets, we relied on a combination of survey repositories" and HuggingFace Datasets using the "Automatic Speech Recog- nition" and "Text-to-Specch" task tags. We expanded coverage to well-documented datasets on the OpenSL platform, even if they were newer or less widely used. We expect this might reflect datasets that could be adopted more widely in the future. Finally, we included datasets related to low-resource languages and other languages not well-covered by our initial searches. Speech recognition models are increasingly highly multilingual [Babu et al. (202I); 1); Radford et al.] (2023); Pratap et al. (2024), and datasets serving different communities builders and end-users around the world are a priority for making speech recognition technologies more inclusive. In total, we trace the provenance and features of 95 speech datasets, covering 18 popular ASR tasks, spanning from 1990 to 2024.
2.3 VIDEO
Scope Early video understanding models primarily focused on video classification, detection and action recognition, where short clips were categorized into predefined classes (Zheng et al. 2022 Zhu let al. 2020). More advanced tasks such as temporal action segmentation, video question answering and video captioning were later introduced to build upon these foundational tasks (Moctezuma et al., 2022; Zhu et al., 2023). Recently, following the success in the field of image generation, video generation from text has become a new task that has shown promising results (Brooks et al. 2024 Zheng et al. 2024bb Blattmann et al. 2023 Esser et al. 2023). Given the scarcity of datasets for text- to-video and the often undocumented sources data used in recent video generation models (Mauran) 2024), we take broader approach to our collection of video datasets. We focus annotating popular video tasks and limit our scope to datasets corresponding to video tasks that are either published, highly cited, or have 100+ downloads on HuggingFace. This approach is justified by three key factors: (I) the usefulness of video data to the research community stems from its collection and presentation in peer-reviewed work, (2) datasets can often be repurposed between different tasks, allowing for applicability to new tasks such as video generation from text, and (3) focusing on highly cited datasets ensures that datasets' quality and relevance has been validated by the research community.
Dataset Selection We include datasets tagged with "Video Classification", "Text-to- Video". and from HuggingFace Datasets. We augmented this with datasets tagged by "Video Understanding" or VVideo Generation") in PapersWithCode, as well as datasets listed in a popular Github survey repository. We also consulted the proceedings of recent video workshops: the [arge Scale Video Understanding and Egocentric Vision workshops. We separately consulted committee
"Video-Text-ee
The Speech Datasets Collection
opensIr.rgg; Open Speech and I Language Resources. OpenSLR is a widely used platform in the speech community, dedicated to hosting resources for speech tasks.
4

---

Published as a conference paper at ICLR 2025
of non-author video experts to supplement the list with relevant datasets published at CVPR, ICCV, ECCV, and IJCV. In total, we trace the provenance and features of 104 video datasets, covering 33 popular video tasks, spanning from 2009 to 2024.
3 RESULTS
We discuss three key results related to (1) the rising use of web, social media and synthetic sources, (2) inconsistent and opaque restrictions data use, and (3) a lack of improvement in geographical or linguistic representation. Each of these findings holds across modalities, at the ecosystem level.
3.1
RISING USE OF WEB, SOCIAL MEDIA & SYNTHETIC DATA
Text
Sprech
Video
167.
144
1M
158-
18-
-3313
2014
2015
2018
3000
1823
301
3016
-005
2014
2016
Source Categars
Source Catessey
Sourec Category
Lncyclepctis
Newe Staxe
General Web
Insemet Niteo
Astwbboox
Chowd-Sanaan
YYetthe
Synthens
-Genzal Web
Code
Syelefi
Books
TV
Budio
Pedumi
-Gogle Vikeo
-Heras Purks
-Molis
Geverseent
Social Medie
Buses Parie
Poffersce
Maron Parki
Tel Talls
Flickr
Bin Vidoo
-Gety Ineges
Analma Papes
Euraege
Calling Pla.
Gptay
Figure I: The cumulative size of data (log-scale tokens for text, hours for speech/video) from each source category, across modalities. The source categories in the legend sorted descending by quantity. Speech and video sources are increasingly dominated by internet videos and YouTur Tube. Though text is especially web or encyclopedia (wiki) sources, synthetic text is rising in popularity.
The need for scale, and heterogeneity have driven rising use of data from web-crawled, social media, and synthetic data sources. Developers have sought out ever larger and conveniently accessible sources of training data (Hoffmann et al.] 2022; Henighan et al., 2020). While small, human-curated datasets are often sufficient and someti preferred due to higher quality, these sources often do not scale to present demands (Kaplan et al. 2020 Henighan et al. 2020). Figure we empirically measure the rising use of web craw winwling and social media (or "Tormm" ") websites that provide some of the most scalable and fresh content. While web-sourced data was always prominent, the balance of sources becomes much more skewed after 2018 note the use of the y-axis log scale. We find for Speech and Video that by far the most prominent source of data has become internet videos, and specifically YouTube. Nearly 1M hours each of Speech and Video data from this source far outstrips the next most common sources, which comprise less than 100K hours. For Speech, the primary data sources used to be Calling Platforms (pre-2017), content manually collected with Human Participation, and Audiobooks, but since 2018 internet videos have supplanted these other sources. For Video, since 2013, YouTube, synthetic, and general web data sources all constitute significantly larger portion of data used in prominent video datasets, outstripping the use of Movies, Flickr, Getty, or human curated sources. Among text post-tral training datasets, we see a similar trend with general or news web-based sources, including encyclopedic sources (mainly Wikipedia), providing the majority of tokens over time. Encyclopedic sources alone now contribute over IT tokens total.
Synthetic data sources are rising the most rapidly. Within the video modality, the introduction of VidProm (Wang & Yang 2024a) in 2024, consisting of nearly 7M synthetically generated videos, offered a large shift in the video source distribution. Within the textual modality, from fig. [ synthetic data represented <0.1 %% of the quantity of Web Encyclopedia data in 2020, but is now 10% its proportion in 2024, making up the 5th largest source of tokens. The top models used in generating datasets are mainly from OpenAL. The top 5 consist of ChatGPT, version unspecified (15.0% of synthetic datasets), GPT-4 (14.4%), BART (10.1%) %), GPT-3 (8.3%) and GPT-3.5- average synthetic dataset also has notably longer turns (in tokens) than the average natural dataset: 1,756 tokens vs 1,065. The task distribution of textual synthetic datasets is shifted towards longer form, open-generation and creative tasks. For example, 88.15 1% of natural datasets contain classification tasks, compared to only 66.3% of synthetic datasets. Natural data is also more likely to cover translation than synthetic data (72.4% of datasets vs only 22.9% of synthetic datasets).
(4.9%). The
5

---

Published as a conference paper at ICLR 2025
Dataset & Source Restrictions (Datasel Count)
Text (Collections)
$4.5
7.5
(Datasets)
Speech-
9.5
9.5
13.7
7.4
7.4
18.9
27.4
Video
22.1
9.6
23,1
11.5
25.0
7.7
0%
10%
70%
80%
90%
Dataset Source Restrictions (Total Tokens Hours)
Text (Callections)
96.0
Text (Datasets)
21.1
5.6
73.0
Speech-
23.9
54.2
1:3
33.7
0%
10%
20%
30%
40%
30%
60%
70%
80%
90%
100%
License| Terms
NC/Acad Restricted Unspecified Restricted
NCIAcad Unspecified Unspecified Unspecified
NC/Acad Unrestricted Unspeeified Unrestricted
Commercial Restricted
Commercial
Unspecified
Commercial Unrestricted
Figure 2: The distribution of restrictions from dataset licenses and their sources' terms. We break this down by the count of datasets (top), as well total tokens hours (bottom). Each license categorized as Con-ccommerriil//Academmi (NC/Acad), Unspecified, or Commercially licensed. Each dataset may also have terms from the source: Restricted non-commercial use, Unspecified restrictions, or Unrestricted. Two main findings across modalities emerge: (1) Commercially licensed datasets represent larger set of tokens and hours, relative to number of datasets; however, (2) the vast majority of those commercially licensed tokens/hours bare restrictions from their sources. Tables and in the appendix provide detailed numbers.
3.2 INCOSSISTENT USE RESTRICTIONS
In the United States, creators of a work automatically have a copyright interest that gives them exclusive rights make copies and derivatives of the work (17 U.S.C. 106). Licenses are legal documents through which the owners of a work express how others may use their work. By contrast, Terms of Service are contract between platform and its users governing how platform and its content may be used (Robinson & Zhu 2020). For simplicity, we use "Licenses" to refer to dataset restrictions, and "Terms" to refer to restrictions on the sources of datasets. There remain open questions about whether certain data licenses are enforceable, but these licenses signal the intention of data creators and therefore warrant consideration as the data creators may be best positioned to understand the sensitivities of the data (privacy, copyright, representation, etc.), and the most impacted by its downstream use "Morton-Parti 2023 Lee et al. 2023b Mahari & Longpre 2023 Mahari al. 2023). How closely practitioners adhere to dataset licenses source terms remains open question, and may depend on jurisdiction or the desired model's s use cases (Lee et al.) 2023b). This work does not propose
standard for all developers. For these reasons restrict treatment and discussion here to tracing the lineage and distribution of licenses and terms for a given modality,
Data source terms are much more restrictive than the dataset's documented license restrictions. In Figure 2 we find only 25%, 33%, and 32% of text/speech/vides datasets are licensed non- commercially, This value is even lower if we consider the proportion of tokens or hours, with 21 %, 26%. and 33% of text/speech/vide quantities carrying license restrictions. However, staggering 99.8%, 78%, and 99% those quantities carry some form of non-commercial restriction on one of their sources. For text, these restrictions are frequently from being generated by OpenAl other models with a non-compete clause, while for speech and videos this is often since the datasets are derived from web social media sources.
Inconsistencies between dataset licenses and their source's restrictions pose challenges to practitioners. A large amount of datasets have permissive or unspecified licenses, but some set of their sources carry non-commercial restrictions. This inconsistency 79% of tokens text datasets, 55% of speech hours, and 65% video hours. Additionally, 19%, 1 14%, and 36% of text, speech, and video datasets have license intended documentation

---

Published as a conference paper at ICLR 2025
(from audit of the datasets' documentation Hugging Face Datasets, GitHub, and Papers with Code). A lack of centralized documentation around these restrictions means it can be misleading to developers who attempting to source data according their own legal standards for copyright and privacy. Furthermore, lack of documentation can hamper developers following hest practices around data preparation and transparency (Gebru et al. 2021 Bommasani et al. 2023)
Large quantities of commercially licensed text datasets are locked in collections without clear information to separate them from restrictive datasets. In Figure (top and bottom), we see the number of datasets and number of tokens without restrictions significantly higher for Text (Datasets) than Text (Collections). Specifically, 60% more Datasets (or 75% more tokens) are commercially licensed, than for Collections. This demonstrates that many collections contain significant amounts of commercially licensed data. While our audit traces licenses for all datasets within collection, most collections not aggregate or expose this documentation. As result, practitioners may be left without easy access to filter for the subsets appropriate for their sourcing standards.
3.3
GEOGRAPHICAL & LINGUISTIC REPRESENTATION NOT IMPROVING
Text (Gini=0.92)
Speech (Gini=0.86)
Video (Gini-0.74)
AFRICA
ASIA
EUROPE
N AMERICA
OCEANIA
AMERICA
By Count
TEXT
0.3
13.4
24.0
61.5
0.7
0.2
SPEECH
3.6
35.7
30.4
30.4
0.0
0.0
VIDEO
0.0
25.2
24.4
48.0
0.8
1.6
By Tokens Hours
TEXT
0.0
6.1
55.4
38.4
0.1
0.0
SPEECH
0.1
38.8
18.8
42.4
0.0
0.0
VIDEO
0.0
23.1
22.0
38.2
16.7
Figure 3: The geographical distribution of countries (world maps) and continents (table) represented by dataset creators. Despite some differences in European, Russian, and Middle Eastern repre- sentation, creators are heavily concentrated in the US, China, and Western Europe, with little to no representation in South America or Africa, across modalities. The current Gini coefficient for (Text, Speech, Video) (0.92, 0.86, 0.74), where higher values indicate more concentration.
The importance and progress of representation in AI training data. Diversity and representation in training datasets, and among their creators, are widely acknowledged as essential to building AI
models that are less biased, more useful, and more equitable (Joshi et al. 2020 [ingh et 2024a
Ustün et al. 2024 Adelani et al. 2021 2024 Aakanksha et al. 2024, McMillan Major et al. 2022b
Porgali et al. 2023 Monfort et al. 2019a Sigurdsson et al. (01 Prior work has measured cu cultural,
ideological, geographic and linguistic imbalances in data (Faiaal et al. 2022 Shankar et al.) 2017
McMillan Maajoor et al. 2022a De Vries et al. 2019: Mahadev & Chakravarti 2021). These studies have exposed significant flaws, often in the form of bias and discrimination, stemming directly from poor representation in data (Buolamwini & Gebru 2018b; Birhane et al. 202T). As this problem has now been widely acknowledged for decades, recent efforts have foregrounded sourcing data multilingually and multi-culturally. from native speakers and creators (e.g. ROOTS (Laurençon et al. 2022). the Aya Dataset (Singh et al. (2024a), the SEACrowd Catalogue (Lovenia et al. (024). the Masader Catalogue (Alyafeai et al.] 2022), Common Voice (Ardila et al. 2019), Causal Conversations V2 (Porgali al. 2023) or Moments in Time (Monfort al. 2019a)).

---

Published as a conference paper at ICLR 2025
Measuring geographical and linguistic representation. Naturally, aim use audit to measure the progress of these efforts on geographical and linguistic representation in the Al ecosystem. We measure the progress of two forms of representation: (1) language diversity of text and speech data, and (2) geographical diversity of the creators, in all three modalities. For languages, we use the ISO 639-1 and 639-3 language codes and top-level language families from GGottolog 5.0. In Figure Ma, a. c) we display the cumulative sum of unique languages and countries present across all audited datasets, at each time period since 2013. While these measurements illustrate the absolute rise in diversity, we also hope to measure the relative dispersion, or equality of languages and countries in the distribution. In Figure d), we use the Gini Index (Wilson 1914] [Atkinson et al. 1970), a traditional measure of statistical dispersion, frequently used to quantity inequality. This allows us to understand if the distributions of languages and creators are more representative of the international community over the last decade, or equally concentrated despite apparent efforts at the margins.
Inequality in geographical representation remains very high, with few organizations creating datasets from the Global South. For every dataset, our audit recorded the organizational affiliations of each creator of the dataset These organizations were then manually mapped to the country in which they are headquartered. Occasionally, organizations like BigScience, BigCode, or Masakhane have international or continental representation, and were counted such. In Figure 3 we measure the current state of diversity among these creator organizations-weer indicates highest concentration, and lower values more broad representation. Without taking up the normative question of what truly "fair" score would be, these values provide useful comparisons across modalities and over time. We find that Text dataset developers are particularly homogeneous, with Gini-coefficient of 0.92: followed by Speech, at 0.86 and Video at 0.74, which remain high, but are meaningfully less concentrated. Figure 3 also illustrates that even this limited diversity is still concentrated in North America, Europe, East Asia, and less so in the Global South.
a Gini coefficient of I
In Figure 3 we also compare the distribution of datasets, and of tokens or hours by continent. Dataset creators affiliated with African South American organizations account for fewer than 0.2% of all tokens or hours, in each modality. In contrast, Asian affiliated organizations represent large proportions of the data, particularly for speech (39% of hours, attributed predominantly to YODAS (Li et al., 2023b)). Much of this driven by Chinese, Indian, Russian, and Saudi Arabian creators. Most prominently, the combination of North American and European datasets comprises 93% of text tokens, 61% % of speech hours, and 60% of video hours.
Geographical representation has not significantly improved for over a decade. In Figure (c). we measure the total unique number of countries represented across all dataset creator organizations. While individual creators will have varying ethnic and national affiliation, we treat this as an estimate for the influence of each locale in dataset development. We find that while the number of represented countries has risen steadily each year, for each modality, this represents only an illusion of progress. Empirically, the Gini coefficient for each modality has not significantly changed since the start of the period we examine in 2013. Geographic diversity has increased only among Video datasets, and these increases are not significant at the = 0.05 level. Text and Speech geographical representations appear to remain stable over the last decade of AI development.
Multilingual representation has not improved by most measures. Similar to geographical representation, we measure the cumulative number of ISO 639-1 languages and language families over time, as well as the per-modality Gini-coefficient. Figure 4 a) shows significant increases in the number of languages available for speech and text, especially in 2019. and 2023, with the introduction
of large sets like Flores (Goyal et al. 2022), ) xP3x (Muennighoff et al. 2023). Common Voice (Ardila et al. 2019), and the Aya Collection (Singh et al. 2024a). However, once again, when measuring the cumulative dispersion of these datasets Figure 4(b), only Text language families demonstrate any improvement from 2r1-2013 to the present. Improvements in the Gini coefficient appear to be largely driven by individual large-scale projects like xP3x and Common Voice, both introduced in 2019. Subsequently, newer datasets remain predominantly monolingual, causing measures of concentration in text languages, speech languages, and language families to remain consistently high.
Academia, research non-pro,its, and industry labs continue to drive public dataset development. We manually categorize the organizations creating popular datasets into: Academic Organization
A dataset creator, following Longpre et al. 2024b), is defined as organization associated with the release of the dataset as created for machine learning not any of the upstream sources. More details in Appendix [
8

---

Published as a conference paper at ICLR 2025
Total Language (1S0) Rquessated
1.0
500
Modality Speech Test
Gial (Cumulathe)
0.3
400-
200
200
0.4
Medality
100-
Tex: Largunge (IS))
Text Language Fanily
Speeda Langrage (1S0)
Speech Lagaage Family
-2003
2020
2024
<2013 2013
2015
2017
2018 2019 2020 2001 2012 2024
(a) Total Language Representation
(b) Gini-coefficiem for Language Representation
Toud Couuttio Reprosated
Medaliit
ap-
Speech
Video
Giml (Cumetathe))
0.6
Madally
0.2
Speech
2013
2013
2014
2015
2016
2017
2018
2019
2000
2021
2022
2023
2024
<2013 2003 2014 2015 2016 2017 2018 2019 2000 2001 2002 2025 2014
(c) Total Geographical Representation
(d) Gini-cnefficient Geographical Representation
Figure 4: The cumulative totals (left) of languages and countries represented in the data over time, and the 95% confidence intervals of the gini-coefficient over time (right) to measure the representativeness of these variables. Gini-coefficients inequality. A Gini coefficient of indicates highest concentration, and lower values more broad representation. While the number of represented languages and geographies continue to rise (left), the equality of their distribution has in most cases, not significantly changed.
measure of statistical dispersion, frequently used to quantify
Text
16.4
16.9
15.7
15.4
15.1
10.6
9.3
Speech
47.3
6.0
7.1
15.2
10.3
8.2
6.0
Video
71.1
5.5
8.5
10.4
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Creator Cutegury
Academic
Research Group
Industry Lab
Corporation
Startup
Other
Government
Unspecified
Figure 5: The distribution of creator organizations by modality. Most public speech and video datasets are developed by academic organizations, whereas text datasets are developed by wide mix of academia, non-profit or industry labs, as well as startups.
(e.g., universities), Research Groups (e.g., non-profits like BigScience, EleutherAl and A12), Industry Labs (e.g., Cohere for AI, Google DeepMind), Corporations (e.g. Google, Meta), Startups (e.g.. OpenAI, Anthropic), Governments, Unspecified (owner affiliation not shared), and Other. When dataset is released in collaboration between organizations, we record each organization. In Figure 5 find that universities and other academic organizations account for 16%. 47%, and 71% of all recorded dataset releases, across Text, Speech, and Video respectively. Research groups, industry labs and corporations are also significant contributors, especially for Text datasets, where ecosystem contributors are far more distributed. Academic organizations' greater role in Video and Speech may suggest special impediments (e.g., privacy-related) to commercial releases of these datasets.
4 DISCUSSION
The rise web, social media, and synthetic data may pose greater privacy, copyright, and bias risks. Section 3.1 discusses the rise of web content and particularly social media as primary sources for speech and video. Figure T shows these sources now exceed by at least an order of magnitude more traditional, curated sources like movies, audiobooks, radio, TV, data produced by human participants. Their mostly user- gereratee content makes these websites a natural source for the quantity, freshness, and heterogeneity needed to train ceneral-purposs models (Longpre al.

---

Published as a conference paper at ICLR 2025
2023 Aghajanyan et al. 2023). However, prior work suggests that user- geer-genertted web data is also more challenging to use than curated data, particularly for privacy, copyright, bias, and factuality. Web-based and particularly user-g geer-genertted content is dispropootionntely likely to include personally identifiable information (PIT) & Viviano (021); Subraaani et al. (023); Blazar et al. (2023). and copyrighted content (Meese & Hagedorn| 2019] Lee al. 2023bb) These can be reproduced in the outputs of AI models (Carlini et al.J [022] Chen et al. 2023c). creating privacy and copyright
concerns (Zhang et al. 2023a). Open datasets being used train GPAI often attempt to bil
frequently miss-Pll and copyrighted data (Soldaini et al. 2024, Subramani et al. 2023) (although not all do (Penedo et al. 2023)). Social media, in particular, is also known to have bias, toxicity and factuality issues (Olteanu et al. 2019), which can manifest in trained models, even after alignment (Kotha et al. 2023). Lastly, while synthetic data can help reduce the prevalence of PII, copyright, or bias in data, it comes with its own challenges (Kurakin et al. 2023
Liu et al. 2024a).
Social media sites are a very prominent data source, but their terms often restrict crawling or commercial use. We find 71% of Video and 69% of Speech data to be from YouTube, whose scale, freshness, and multimodality (containing videos, speech, images, and text) have made it a prominent data source (Abu-El-Haij) et al. 2 Aytar et al.] 2018 Chang et al. 2020; [thus et al.) [023) Coats| 2023 ei 2023b). However, You'l Tube is social media platform owned by Google and its Terms Serviee prohibit third-party crawling. While content creators retain ownership of material uploaded to YouTube, the YouTube terms of service also grant Google license to reproduce, modify, display, and use the content for YouTube's "business" (which may include building machine learning models), and forbid crawling by third parties, even if the copyright holder has selected a permissive license. Model developers like Nvidia and OpenAl have been sued in the U.S. by content creators who allege that they unlawfully trained on YouTube videos (Cole) 2024; Skolnik 2024). Large social media sites like Reddit and StackOverflow have also recently adopted restrictive terms, with access being restricted just as these data sources become critical scaling Al systems. The enforceability of these licenses and terms, however, is an open legal question beyond the scope of our work.
Ambiguous and poorly documented use restrictions may significantly inhibit model developers adhering to cautious legal and ethical data sourcing standards. In Section [3.2] we find that significant amount of data carry non- com-comeeraaal restrictions in their sources, rather than on the final dataset, which can contain no license reach 99% of all tokens and hours. These inconsistencies are the result of datasets being iteratively re-packaged and re-licensed, without carrying on documentation (Longpre et al., (024b). While not every developer will employ the same filtering standards, our work shows that separating and identifying appropriate datasets remains difficult across these modalities. Without continued audits and documentation, practitioners may have to either take on avoidable risk or forgo large collections of partially viable data, hampering data scaling laws (Kaplan et al. 2020). We hope our audit will help practitioners apply their own standards and make informed decisions on training data use.
permissive one. For text and video, these restrictions can
The limitations of measures of geographical and linguistic representation. Mcasures of ge- ographic and linguistic representation are imperfect: We have only partial information about the developers' identities (including for privacy reasons), limited transparency into how frequently these datasets are used, and cannot say how far proprietary datasets may fill in representation gaps behind closed doors. Nonetheless, we believe the breadth and rigour of the audit make this the best available empirical measure of representation in public datasets. Indeed, we contend that it necessary to mea- sure representation in Al data to understand progress, or its absence, towards AI systems that fairly serve the broad user community. Figures 3 and 4 show that despite the absolute rise of geographic and linguistic representation, western-centric skew persists across thousands of surveyed datasets. We release all audit materials for transparency, replicability, and further research use.
Conducting representative analyses of an ecosystem comes with assumptions. The Al ecosystem is decentralized. Text datasets, for example, are often hosted on HuggingFace, unlike Speech and post-training, this is
Video. Similarly, while Text data is frequently repackaged for general-purge less true of other modalities. Scoping and dataset selection thus need to be modality rather than a single general protocol unable to capture these nuances. Similarly, we studied modalities of interest to foundation model development (Brooks et al. 2024 Radford et al. (023), but many others remain for future work (e.g., images, 3D shapes, tabular, time series, graphs, and geospatial data).
You'Tube Terms of Service
Reddit User Agreement and StackOverflow Terms of Service
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was conducted by the Data Provenance Initiative, collective of independent and academic researchers volunteering their time to data transparency projects. The Data Provenance Initiative is supported by the Mozilla Data Futures Lab Infrastructure Fund.
A EXTENDED RELATED WORK
Progress in machine learning across modalities from speech (Radford et al. 2023) to vision (Dosovit- skiy et al. 2021) to text (Brown et al. 2020aa Wei et at., 2021) has benefited from advancements in large pre-traintn tre-training and fine-tuning corpora. The development of multimodal corpora has also been key to several recent advances, with CLIP in the image/text domain Radford et al. (20TI) CLAP for audio/text settings Elizalde et al.
(022). and a number other models involving both text and
images, audio video (Radford et al. 2022).
2023 Ramirez et al. 2024 Singer et al. 2022 Ramesh et al.,
The datasets powering these advances are not, however, always well-documented, despite the existence of standards and frameworks for recording and annotating dataset metadata that range from 'data statements' (Bender & Friedman, 2018) 'datasheets for datasets' (Gebru et al.) (021 and others (Mitchell et al. 2019). The key problem is not a deficiency of any particular framework, but rather inconsistent adoption and fragmentation (Longpre al. 2024d). Much prior work has argued for the need to document and audit these datasets (Rogers) 2021] Paullada et al., 2021). motivated by
concerns from reproducibility (Kapoor & Narayanan (022) to interpretability (Longpre al., (2023) to bias and fairness problems that may stem from problematic content in training data (Birhane et al. 2021
There have been several attempts to carry out such audits, with prior work examining pretraining data (Longrre et al. 2024c), general web corpora (Gao et al. 2020; Dodge et al. 202I), instruction fine- tuning datasets (Longrre et al. 2024b), and the documentation fields of the HuggingFace Datasets platform in particular (Ynng et al. 2024). For speech and vision, there has been less work, with many discussions of datasets in the aggregate occurring in survey papers (Schiappa et al., 2023
Chaquet et al., 2013),, research aimed directly at improving model performance Gadre et al. (2023 or
close examinations of questions like bias in small groups of datasets (Buolamwini & Gebru 201 8b
Romanou et al. 2024).
Prior work has also examined the identities, affiliations and national origin of paper authors (Movva ct al. 2024) in AI, but an analogous look at the producers of datasets is lacking. We aim to carry such analyses: replicating those for pretraining and text finetuning datasets video and audio domains, and surveying provenance and legal status. Finally, there has also been significant recent attention to legal questions in the collection and use of Al training data (Sag) [020 Henderson et al. 2023). The complex process involved in preparing these datasets (Lee et al. (003b)) and the ambiguous licensing of inputs, can make understanding the legal status of final output quite difficult.
B DATASET LICENSES & TERMS
Detailed taxonomy We code the legal restrictions placed use of datasets along two First, we identify whether a dataset's license permits commercial use ("Commercial" in Table 3 only non-commercial / academic use ("NC Acad"), does not clearly specify what is permitted ("Unspecified"). The latter category includes datasets for which we were unable to locate a license. Datasets which are in the public domain and not subject license are counted as commercially usable. Second, we annotate the contractual or terms-of-use restrictions placed on dataset use by the source of each dataset. There are four levels, defined in Table Note that the Model Closed status can only apply to datasets that are Al-gen Al-generated, at least in part. Some datasets can carry both Model Closed and Source Closed status, but we count the Model Closed first for simplicity.
Detailed breakdown Tables and present crosstabs of these two dimensions, according to respectively the total amount of content and the number datasets. The most notable finding, as discussed in the main text, is the frequency of clashing restriction status between licenses and terms.
11

---

Published as a conference paper at ICLR 2025
LABEL
DEFINITION
MODEL CLOSED
A model used generale part all of the dataset prohibits using its oulputs commercially, to develop a competing Al model. or in general.
SOURCE CLOSED
The source has a license or terms that prohibits use of the data, either commercially, from being crawled, develop AI, in general. No information can be found relevant to restrictions, or lack thereof, for this source.
UNSPECIFIED
UNRESTRICTED
The souree has commercially permissive license, such CC BY, or explicitly states the data is open for broad use.
Table 2: The taxonomy used to determine
restrictions each dataset source. Each source in a dataset examined and into one of these categories. The dataset Terms are then labelled according to the strictest terms across the sources, with Model Closed and Source Closed considered stricter than Unspecified which turn stricter than Unrestricted.
By amount of content, fully 73.0% of text content, 55.0% of speech content, and 21.6% of video content is subject to license permitting commercial but also terms restrictions forbidding or the reverse. The absolute level of restrictions is also high, with < 0.1 % of text content, 5.4% of speech content, and 0.6% of video content usable for commercial purposes under both licenses and terms.
LICENSE/ / TERMS RESTRICTED UNSPECIFIED
UNRESTRICTED
TOTAL
Text Collections
NC/ACAD UNSPECIFIED
96.0
0.0
0.0
96.0
2.3
0.1
0.0
2.4
COMMERCIAL
1.5
0.0
0.0
1.6
TOTAL
99.8
0.1
0.1
Text Datasets
NC/ACAD
21.1
0.0
0.0
21.2
UNSPECIFIED
5.7
0.1
0.0
5.7
COMMERCIAL
73.0
0.0
0.0
73.1
TOTAL
99.8
0.1
0.1
Speech Datasets
NC/ACAD
23.9
1.4
0.8
26.2
UNSPECIFIED
0.5
0.0
0.4
0.9
COMMERCIAL
54.2
13.3
5.4
73.0
TOTAL
78.6
14.7
6.7
Video Datasets
NC/ACAD
33.7
0.0
0.1
33.8
UNSPECIFIED
43,9
0.1
0.1
44.1
COMMERCIAL
21.5
0.0
0.6
22.1
TOTAL
99.1
0.1
0.8
Table 3: A breakdown of the percentage of license and terms restrictions across datasets, by total tokens or hours of content. The much higher frequency restrictions at the collection level is because consider collection's 's license terms status be the most restrictive of those for its datasets. Note that percentages may not add to exactly 100% because of rounding.
12

---

Published as a conference paper at ICLR 2025
LICENSE / TERMS
RESTRICTED
UNSPECIFIED
UNRESTRICTED
TOTAL
Text Collections
NC/ACAD
84.5
0.0
0.3
84.8
UNSPECIFIED
1.5
7.5
0,0
8.9
COMMERCIAL
1.5
0.2
4.5
6.3
TOTAL.
87.5
7.7
4.8
Text Datasets
NC/ACAD
25.0
0.0
0.3
25.3
UNSPECIFIED
17.3
1.2
0.0
18.5
COMMERCIAL
45.2
6.5
4.5
56.2
TOTAL
87.5
7.7
4.8
Speech Datasets
NC/ACAD
9.5
9.5
13.7
32.6
UNSPECIFIED
6.3
0.0
7.4
13.7
COMMERCIAL
7.4
18.9
27.4
53.7
TOTAL
23.2
28.4
48.4
Video Datasets
NC/ACAD
22.1
0.0
9.6
31.7
UNSPECIFIED COMMERCIAL
23.1
1.0
11.5
35.6
25.0
0.0
7.7
32.7
TOTAL
70.2
1.0
28.8
Table A breakdown of the percentage of license and terms restrictions by dataset count. The much higher frequency of restrictions at the collection level is because consider collection's license or terms status to be the most restrictive of those for its datasets. Note that percentages may not add to exactly 100% because of rounding.
C ADDITIONAL RESULTS
Figures 6 and report the size distributions of the datasets. We measure size differently for different types of datasets: Text datasets are tokens, and audio/video in hours of content. The lack of standard tokenization preprocessing schemes for those modalities makes it simplest to report raw dataset size.
Notably, we find quite different size distributions by modality. The distribution of dataset sizes has the thickest right tail for text, followed by speech and then by video. Most video datasets are short in hour terms, with speech datasets tending to be somewhat longer and text datasets having greater prevalence of both very small and very large datasets relative to the mean size.
Dataset tasks, meanwhile, reflect traditional approaches and research programs for each modality. Classification is the most common task for both text and video, with the video community's 's long- standing interest in captioning also visible in its role the second most common task for video datasets. Q&& occupies a similar role for text, though text datasets have a more balanced distribution over other, increasingly prominent tasks like generation and reasoning. Given selection criteria, all datasets for speech are for ASR tasks, but other tasks like speaker identification and translation also represented.
D DATASETS
This section provides a detailed overview of the datasets we have collected and analyzed. [Table 5 summarizes the text datasets, Table the audio datasets, and Table 7 the video datasets. Each of these
13

---

Published as a conference paper at ICLR 2025
(Tokens)
Sprech (Hours)
Video (Hours)
1-100
0-100
15-18M
104. l0sk
1K-1KK
1K-10k
100M-1a
Ih. mm
10K-100K
106- 103K
10
20
30
10
20
401
30
40
Namber of datasets
Number of dastasets
Namberr of datssets
Figure 6: The distribution of dataset sizes for each modality. Most text data collections between 100M-1B tokens. Speech datasets average 100-1k hours, and video datasets are usually the smallest, commonly less than 100 hours.
Text
Speec
Vidao
Closifiction.
Recogrbtinn
Clouficatin
QLAAA
Speaker
Cagrioiia
Genemiior
Transation
Seymentation
Resoming
langrag tb-
Summrnization
Conrvit
Codo
Speeeh
Action Rloogsiton
Locilzation
QRA
Bias Devetion
Keyword Spoting-
Query Ex-
Eeiraation
Runking
Synthess
0%
5%
10%
15%
20%
25%
0%
20%
40%
50%
80% 1005
0%
10%
20%
30%5
40%
50%
Figure 7: The task distribution of datasets, across modalities. Post-training text and video datasets are predominantly based on classification. For text, generation and reasoning are rising categories. All speech datasets are recognition-base particularly for speaker, language, or in the process of translation.
tables lists broad collections of data, sorted chronological order, and provides information about their properties, sizes, sources and permissions. Each collection include multiple datasets, and they generally reflect the ways dataset creators have grouped their datasets (such as in the same paper). Because of the large number of datasets, provide detailed information about their licenses and original published papers, where applicable, in the supplementary Attribution Card in Appendix F
Annotation Details: Text For post-training text datasets it is common to package many together as collections, such as Flan (Wei et al., (02T) or P3 (Sanh et al.) 202I). This practice is not common to the same extent for speech or video datasets. For much of the text analysis, where possible, we chose to analyze statistics at the collection-level, since practitioners are more likely adopt a collection for general- purpose post-training, than individual dataset within the collection. Also, in dataset-level statistics, metadata for single collection with many datasets can get repeated and overwhelm the statistics unfairly (e.g. the dataset aggregator/creator being repeated hundreds of times). Consequently, our collection-lever analysis of the text modality is reflected Figure I Figure 3 Figure 5 Figure 4 Figure 7 and Figure However, for Figure we draw the distinction between collection and dataset metrics, as practitioners may wish to unpack collections to extract only commercially licensed data. In that Collection inherits the most restrictive license and terms of its constituent datasets.
For annotating creator organizations, we follow prior work's instructions (Longpre et al. 20245). For each dataset they record the affiliations listed on the academic paper GitHub or HuggingFac object in which the dataset was released. This does not include the organizations who created or owned the sources from which the data was derived. For instance, the SQuAD dataset (Rajpurkar et al. 2016a) would be associated with Stanford (the authors' affiliation). but not Wikipedia, which the data was partially derived from. For dataset that has authors affiliated with multiple organizations, the dataset will be counted towards each organization.
Speech In many cases, multiple versions of dataset exist due to datasets being expanded or updated. In these scenarios, we used the release date from the initial version (since release dates for subsequent versions not always clear), but used metadata from the most recently released version for which information was available to offer an overview of the current landscape of data. However, if the dataset versions could not be meaningfully aggregated (e.g. different licenses),
Annotation Details:
14

---

Published as a conference paper at ICLR 2025
or did not appear to be cumulatively designed lnerloperlapping otherwise semantically disjoint data), we maintained separate records. We kept only datasets for which ASR was noted as a primary primarily intended for text-to-speech speaker recognition,
task. For example, if dataset did not keep it even if it could conceivably be repurposed for ASR. When computing hours, we excluded any hours without supervisory transcripts/scripts (unlabeled data), but kept hours with "weak supervision" (e.g. model-generated transcripts from speech audio). We recognize the difficulty in comprehensively covering all relevant datasets.
Annotation Details: Video In video, a single dataset can be re-purposed and annotated to address different tasks Monfort al. (2019a) 202 We consider these two different datasets even if they have the same video source since now they can be used for different computer vision tasks.
15

---

Published as a conference paper at ICLR 2025
Table 5: Alignment tuning (text) collections and properties. Collection properties include numbers of datasets, tasks, languages, and text domains. The SOURCE column indicates whether collection contains human-generaiee web text language model outputs (e)) both (8)) The USE
column indicates whether collection includes data freely usable even for commercial purposes
data usable only for noncommercial purposes or academic research and data whose license status is not specified precisely enough to allow us to determine commercial use permissions that each collection may have different datasets with one, two, all three these statuses. Finally, the OAI column indicates collections which include OpenAl model generations, Datasets are sorted chronologically to highlight trends over time.
Note
COLLECTION
PROPERTY COUNTS
TYPES
PERMISSIONS
YEAR
DATASETS TASKS LANGS DOMAINS SOURCE
USE OAI
RiddleSense
2021
3
MathInstr.
2023
I
3
No Robots
2023
1
8
Nectar
2023
I
MetaMathQA
2023
8
2
MegaWika
2023
50
1
50
MedInstr.
2023
1
MathDial
2023
2
PII-Masking
2023
2
4
Pure-Dove
2023
4
LMSYS-Chat-IN 1M
2023
9
PygmaPionAL-PIPPA
2023
3
HelpSteer
2023
5
SeaBench
2023
9
4
Open Asst, v2
2023
4
19
Feedback Coll.
2023
2
Glaive Code Asst.
2023
2
2
EverythingLMM
2023
8
Bactrian-X
2023
4
COBRA Frames
2023
1
2
UltraFeedback Argilla
2023
9
16
20
ExpertQA
2023
3
ChatDoctor
2023
3
1
Capybara
2023
11
17
2
UlrraCaat-200k
2023
7
CollectiveCognitionn
2023
6
Thai Gen AI
2023
9
11
Deita 10K
2023
2
11
3
SelFee
2023
5
ChatbotArena
2023
4
OpenGPT Healthcare
2023
3
4
Orca-Math
2024
1
3
OpenMathInstr.-I
2024
2
3
WildChat
2024
2
7
10
Magpie-Pro
2024
1
9
Continued next page
16

---

Published as a conference paper at ICLR 2025
Table 5: Alignment tuning (text) collections and properties.
COLLECTION
PROPERTY COUNTS
TYPES
PERMISSIONS
YEAR
DATASETS
TASKS LANGS DOMAINS
SOURCE
USE OAI
10k Prompt Ranked
2024
13
Synth.GGSM8K-RR
2024
3
LongAlign-10K
2024
1
3
Llama2-MedTunedd-nee
2024
4
KIWI
2024
Indic-Instr.
2024
8
7
3
Gretel Text-to-SQL
2024
1
Conifer
2024
8
Cidar
2024
8
Aya
2024
71
7
71
Reasoning
2024
1
4
AgentInstruct
Mult.
3
InstAr
Mult.
24
13
9
Dynosaur
Mult.
1k
21
22
Medical Meadow
Mult.
8
2
Mult.
10
10
36
PMC-LLaMA Instr.
Mult.
7
1
2
COIG
Mult.
18
13
2
22
DialogStudio
Mult.
83
3
Table 6: Audio collections and properties. Collection properties include numbers audio hours (HR), speakers (SPKR), languages (LANG), creator institutions (CREAt), tasks (TASKS), data sources (SRC), and topics (TOPICS). The number of datasets is not listed because all collections include only one dataset, except for M2ASR which has four. The US column indicates datasets from or partly from the United States, the Ac column datasets created by academic institutions, and the IND column datasets created industry. Note that dataset can have all of these, none of them, or any combination of them. The USE column indicates whether a collection includes data freely usable even for commercial purposes data usable only for noncommercial purposes or academic research and data whose license status is not specified precisely enough to allow us to determine commercial use permissions Note that each collection may have different datasets with one, two, all three of these statuses. Datasets are sorted chronologically to highlight trends over time.
COLLECTION
PROPERTY COUNTS
CATEGORY
PERM
YEAR
HR
SPKR LANG CREAT TASKS
TOP US AC IND
USE
TIMIT
1990
5
630
3
3
7
Switchboard
1992
250
543
1
1
1
70
African Ace. French
2003
22
232
1
7
CSJ
2003
661
1k
2
Fisher
2004
2k
12k
1
1
36
CSLU 22 Langs.
2005
84
21
7
AMI
2005
100
2
CSLU 1.2
2007
25
5k
1
1
1
ALLSSTAR
2010
86
140
27
3
Continued next page
17

---

Published as a conference paper at ICLR 2025
Table 6: Audio collections and properties.
COLLECTION
PROPERTY COUNTS
CATEGORY AC IND
PERM
YEAR
HR
SPKR LANG CREAT TASKS SRC TOP
US
USE
TED-LIUMB
2012
452
2k
2
NST Norwegian
2013
540
870
1
7
NST Danish
2013
500
1
1
1
7
NST Swedish
2013
300
7
Vystadial
2014
56
3
THCHS-30
2015
35
40
1
LibriSpeech
2015
1k
2k
106
THUYG-20
2015
20
371
2
3
VCTK
2016
44
110
Spoken Wikipedia
2016
1k
960
3
1
AISHELL-1
2017
520
400
2
2
11
LJSpeech
2017
24
ClarinPL
2017
56
317
1
7
AISHELL-2
2018
1k
2k
2
2
8
Regional Af, Am. Lang.
2018
159
222
8
Crowd Sourced Speech
2018
1k
3k
5
Korean
2018
96
181
1
7
RTVE
2018
691
7
OpenSTT
2019
20k
6
MuST-C
2019
4k
2k
16
2
2
4
M-AILABS
2019
1k
8
33
MAGICDATA
2019
755
1k
Common Voice 17
2019
1k
330k
124
3
CoNASE
2019
154k
6
Nigerian English
2019
6
7
Norwegian Parl. Speech
2019
140
309
7
120h Spanish Speech
2019
120
17
7
DiDiSpeech Czech Parliament
2020
800
6k
2
2020
444
212
7
CoVoST-2
2020
3k
78k
22
1
KSC
2020
332
5
Basq Cat. and Gal.
2020
34
132
3
2
KsponSpeech
2020
969
2k
6
Samromur
2020
145
8k
5
Multiling. LibriSpeech
2020
50k
6k
8
33
MaSS
2020
160
8
1
FT SPEECH
2020
2k
434
2
2
2
Eng. Aec. in Brit. Isles
2020
31
120
1
4
Highland Puebla Nahuatl
2021
156
3
7
QASR
2021
2k
11k
2
7
Multiling. TEDx
2021
765
9
3
7
Minds14
2021
25
14
1
7
Golos
2021
1k
3
6
Continued on next page
18

---

Published as a conference paper at ICLR 2025
Table 6: Audio collections and properties.
COLLECTION
PROPERTY COUNTS
CATEGORY PERM
YEAR
HR
SPKR LANG CREAT TASKS SRC TOP
US
Ac IND
USE
MASC
2021
1k
14k
1
3
15
LaboroTVSpeech
2021
2k
2
2
7
KeSpeech
2021
2k
27k
2
1
1
1
JTUBESPEECH
2021
1k
2
7
GigaSpeech
2021
10k
1
9
24
VoxPopuli
2021
2k
4k
16
SPGISpeech
2021
5k
50k
1
2
West Afr. Radio
2021
142
10
2
2
3
AISHELL-4
2021
120
61
6
West Afr. Virt. Asst.
2021
2
49
3
2
2
2
MediaSpeech
2021
40
4
5
12
1
People's Speech
2021
30k
1
7
14
1111 Hours Hindi
2022
108
5
Shrutilipi
2022
6k
12
2
2
WenetSpeech
2022
10k
1
4
4
10
Samromur Children
2022
131
3k
5
SDS-200
2022
200
4k
3
2
aidatatang
2022
200
600
1
1
7
Fleurs
2022
1k
102
3
3
11
OLKAVS
2022
1k
1k
1
2
2
14
Norwegian Parl.
2022
140
267
1
2
2
MAMCC
2022
180
663
1
15
Kathbath
2022
2k
1k
12
2
2
3
Hebrew Kan
2022
9
3
Hebrew Coursera
2022
36
7
Bloom Speech
2022
428
56
5
5
8
English- Vietnamese
2022
508
2
1
1
7
Earnings-22
2022
119
125
1
2
YODAS
2023
370k
149
3
AFRISPEECH-200
2023
200
2k
20
14
6
Aalto Finnish Parl.
2023
3k
449
1
2
ReazonSpeech
2023
35k
2
2
EdAcc
2023
40
120
8
Rix Vox
2023
5k
1
2
Japanese Anime Speech
2023
110
7
Snow Mountain
2023
273
11
14
2
1
Samromur Milljon
2023
967
17k
1
5
Bud500
2024
500
1
1
4
Vibra Vox
2024
18
200
1
1
1
M2ASR
Mult.
448
655
4
3
9
19

---

Published as a conference paper at ICLR 2025
Table 7: Video collections and properties. Collection properties include numbers of hours of video, datasets, creator institutions, countries of creator institutions, and data sources. The USE column indicates whether collection includes data freely usable even for commercial purposes usable only for noncommercial purposes academic research and data whose license status is Note that
data
not specified precisely enough to allow determine commercial use permissions each collection may have different datasets with one, two, or all three of these statuses. Finally, the
AVAIL column indicates whether dataset is available online (V) or has been taken down, usually for legal reasons (X). Datasets are sorted chronologically to highlight trends over time.
COLLECTION
PROPERTY COUNTS
PERMISSIONS USE AVAIL
YEAR HOURS DATASETS COUNTRIES CREATORS SOURCES
HOLLYWOOD2
2009
20
Collective
2009
HMDB
2011
7k
UCFI01
2012
26
YouCook
2013
1k
50 Salads
2013
40
StoryGraphs
2014
Hollywood Ext.
2014
Breakfast
2014
77
2
Sports-IM
2014
106k
THUMOS
2014
254
2
4
VideoStory
2014
743
SumMe
2014
2
3
TVSum
2015
4
Volleyball
2015
ActivityNet
2015
849
MovieQA
2015
381
Mars
2016
NTU RGB+D
2016
74
MSR-VTT
2016
41
Charades
2016
VTW
2016
213
Youtube-8N.
2016
350k
Narrated Instr. Vid.
2016
TGIF
2016
MuliTHUMOS
2017
30
Imd
2017
PKU-MMD
2017
50
2
20BN-SOMETHING
2017
121
YouCook2
2017
176
VoxCeleb
2017
2k
Davis
2017
2
QFVS
2017
20
2
DiDeMo
2018
275
SOA
2018
2k
Charades-Fgg Ego
2018
69
EPIC-KITCHENS
2018
100
3
3
MovieGraphs
2018
94
How2
2018
2k
Continued next page
20

---

Published as a conference paper at ICLR 2025
Table Video collections and properties.
COLLECTION
PROPERTY COUNTS
PERMISSIONS
YEAR HOURS DATASETS COUNTRIES CREATORS SOURCES
USE AVAIL
VLOG
2018
336
VaTeX
2019
115
2
2
20BN-jester
2019
13
HowTo100M
2019
134k
2
4
COIN
2019
476
2
MMAct
2019
100
2
2
HACS
2019
833
3
CrossTask
2019
376
5
Moments in Time
2019
833
TRECVid
2019
1k
MSA
2019
516
Toyota Smarthome
2019
269
TITAN
2020
VIOLIN
2020
582
AareAct
2020
21
VIRAT
2020
100DOH
2020
5k
Oops!
2020
50
OmniSource- Web
2020
13k
Condensed Movies
2020
1k
MovieScenes
2020
250
BEV
2020
370
Movie-Net
2020
3k
FineGym
2020
708
HAA500
2020
5
4
LEMMA
2020
11
2
HVU
2020
96k
3
5
Apes
2021
36
3
WebVid
2021
13k
2
2
VideoL.T
2021
14k
4
HOMAGE
2021
30
2
UAV-Human
2021
18
2
HD-VILA-
2021
372
M-MiT
2021
833
2
Mimetics
2021
Spoken Moments
2021
417
QuerYD
2021
207
MAD
2022
FERV39K
2022
CDAD
2022
215
MVBench
2023
VidProm
2024
240k
ShareGPT4Video
2024
3k
OpenVid-IM
2024
52k
2024
3k
Disney Vid. Gen.
2024
Continued on next page
21

---

Published as a conference paper at ICLR 2025
Table Video collections and properties.
COLLECTION
PROPERTY COUNTS
PERMISSIONS
YEAR HOURS DATASETS COUNTRIES CREATORS SOURCES
USE AVAIL
Kinetics
Mult.
4k
3
2
Ego4D
Mult.
5k
2
2
MPI
Mult.
110
3
2
Project-Aria
Mult.
1k
2
1
Ava
Mult.
146
2
2
LSMDC
Mult.
316
2
10
E
CONTRIBUTIONS
Here we break down contributions to this work. Contributors are listed alphabetically, except for team leads who are placed first.
Text Datasets Shayne Longpre (lead), Jad Kabbara (lead), Ahmad Anis, Deividas Mataci- unas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Kun Qian, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Nayan Saxena, Niklas Muen- nighoff, Naana Obeng-Mamen Marnu, Robert Mahari, Seonghyeon Ye, Seungone Kim, Shayne Longpre, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, William Brannon, Xuhui Zhou, Yizhi Li, An Dinh, Caroline Chitongo, Christopher Klamm, Da Yin, Damien Sileo, Ariel Lee
Reviewing Text Dataset Metadata Jad Kabbara (lead), Shayne Longpre (lead), Robert Mahari, Damien Sileo, Niklas Muennighoff, William Brannon,
Data Explorer Features Shayne Longpre (lead), Christopher Klamm, Vu Minh Chien,
Speech Datasets Nikhil Singh (lead), Manuel Cherep (lead), An Dinh, Minnie Liang, Shrestha Mohanty
Video Datasets Kush Tiwary (lead), Joanna Materzynska (lead), Vivek Sharma, Shayne Longpre, Robert Mahari, Jad Kabbara, William Brannon, Tobin South, Shrestha Mohanty, Nikhil Singh, Manuel Cherep
Data Analysis Shayne Longpre (lead), Nikhil Singh (lead), Manuel Cherep (lead), Kush Tiwary (lead), Joanna Materzynska (lead), Naana Obeng-Mann (lead), William Brannon (lead),
Writing Shayne Longpre (lead), Jad Kabbara (lead), Nikhil Singh, Manuel Cherep, Kush Tiwary, Joanna Materzynska, Robert Mahari
Legal Analysis Robert Mahari (lead), Luis Villa
Visualizations & Visual Data Analysis Nikhil Singh (lead), Manuel Cherep (lead), Kush
Tiwary (lead), Joanna Materzynska (lead), Naana Obeng-Marnu (lead), William Brannon (lead), Shayne Longpre (lead), Ariel Lee, Hamidah Oderinwale, Camphell Lund
Senior Advisors Stella Biderman, Sara Hooker, Jad Kabbara, Sandy Pentland, Luis Villa, Caiming Xiong
F ATTRIBUTION CARD
Here we provide detailed information about the licenses of each data collection and constituent datasets, and cite all of the papers (455 in all) which introduced datasets consider. Text datasets are laid out in Table audio datasets in Table and video datasets in Table 10 Because of the large number of references, we include second bibliography after the tables (named 'Attribution Card References') ) with numbered citations in this section referring to that second bibliography.
22

---

Published as a conference paper at ICLR 2025
Table 8: References and licenses for alignment-tunin (text) dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphahetically for ease of dataset lookup.
Collection
Licenses
Cite
10k Prompt Ranked AgentInstruct
Unspecified
Unspecified, CC BY 4.0, MIT Li- Shridhar et al. cense
(2021 ;
Yao et al.
(2023)
Liu al.
(2023c)
Zeng et al.
(2023
Deng et al. (2023
Aya
Apache License 2.0
Singh et al.
2024b)
Bactrian-X
CC BY-SA 3.0, CC B.-
Li al.
(023a
COBRA Frames
BigScience OpenRAlI.-M
Zhou et (023b)
COIG
Various
Zhang et al. (202bb)
Bai et al.
(2024a)
Capybara
Various
ChatDoctor
Unspecified
Li al. 2023d
ChatbotArena
CC BY 4.0, CC BY-NC 4.0
Zheng al.
(2023
Cidar
CC N.0
Alyafeai
(2024)
CollectiveCognition
MIT License
Conifer
Apache License 2.0
Sun et 2024
Deita 10K
Apache License 2.0, CC BY-NC 4.0
Liu et al.
2024b)
Continued on next page
23

---

Published as a conference paper at ICLR 2025
Table 8: References and licenses for alignment-tuning (text) dataset collections presented this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
DialogStudio
Various
Chen al. (2021a); Wei al. (2018); Lin et al. (2021b); Chawla al. (222 ) He al. (2018) Mrk- Sic et al. (201 7): Qian et al. 2022)
Liu et al. (2021); El Asri al.
(2017); Quan et al. (2019); Chen et al. (2019) 3
Eric & Man-
ning 2017|; Zang lyminov et al. (2019); Martin et al.
(2020): Sha-
(2020): Peskov et al. (2019); Eric et al. (2019): Moon et al. (2019) Rastogi
(2020); Mosig et al. (2020); Chiu et al. (002); Shah
(2088); Byrme et al. (2019)
Mrk3ie & Vulic (201 8p; [Shang al.
(2018): Rameshkumar & Bailey
(0020;
;; Fabbri et al. (2021); Chen
(021cc;: Mukherjee et al. Shang al. (2018); Zhu (2021): Zhong et al. (2021); Gliwa al. (2019); Chen
al.
(2022)
ct al.
al.
(2022a); Feigenblat et al. (2021 al. (2019c); Dinan al. (2019a) Rashkin et al. (2019); Bai et al. (2022); Chen al. (2023 Kim et al. (2022); Myers et al. (2020); Reddy al. (2019): Yu et al. (2019a); Talmor & Berant (2018)
Nan et al. (2021 2022); Gu et al. (2021); Chen et al. (020b) Gupta ct al. (2018a): Li ct al. (2021a); Talmor et al. 2021) Yu et al. (2019c); lyyer et al. (2017 7 Yu et al. (001bb) Parikh al. (2020); Yih
et al. (2016); Zhong et al. (2017);
Pasupat & Liang (2015): Komeili et al. (2022); Dinan et al. (001bb);
Hemphill et al. (1990); Casanueva et (2020) Zhhng et al. (2022b);
Larson et al. (2019): Rastogi et al.
(2020); Liu et al.
2019 2013);
Coope et al. (2020):
Coucke et al. (2018b)
(2018); Gupta et al.
Continued on next page
24

---

Published as a conference paper at ICLR 2025
Table 8: References and licenses for alignment-tuning (text) dataset collections presented this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
Dynosaur
Various
Adlakha (2022);
Agarwal
et al. 20021 Akyürek et al. (2022))
Amini et al. (2019): Ardanuy et al.
(2020); Austin et al. (2021): Azer-
bayev et al. (2023); Bai ct al.
(2022); Bajaj et al. (2018); Balakr
ishman et al. (2019) ; Bartolo et al. (2020); Bisk et al. 2019)) Boratko
et al. (2020); Botha et al. (2018);
Boudin & Gallina [22 Bravo
et (2015); Brown et al. (2020b);
Byme al. (2019); Cao & Wang
[202];; Cao et al. ; ; asanueva
(2020); Cetoli et al. 2019):
Chalkidis et al. (2019h 2021
Chan et al. (0022 Chapuis et al. (202 Chen et al. (2220a); Cheng al. (2022); Chouldechove (2017);
Christmann et al. (2019); Clark et al.
(2019) (0019|2018]] (201
Cobbe et al. (2021); 1);
Coucke al. (2018); Dankers al.
(2022); Dasigi et al. (2019); De
varaj et al. (2021): De Young et al. (2021): Diggelmann et al. (202
Emelin al. (2020); Fabbri et al. (2019); Faruqui & Das (2018); Feng
al. (2021); Gallina al. (2019))
Ganesan et al. (2010); Gazzola et al. (20T9): (20T9)/[Georg & Mamidi (2019);
al. (2019); Gorrell et al. (2018); Gu al. (2022); Gupta et al. (2021) Ha & Eck (2017): Haagsma et al. (2020)|Hazoom Hazoom et al. (2021) Hen
Geva et al. (2019); Gliwa
derson et al. (2022); Hendrycks et al.
(2021): Huang
(2019 202
Huang (2022): Irwin et al. 2020
Ivgi al. (2022) Iyer et al. (2017
Jiang et al. (2020) 202 ); Jin et al.
(2019); Joshi al. (201 7): Juraska
al. (201 Jurezyk al. (2016);
Kanade et al. (2020); Kaushik al.
(2020); Khot al. (2020) (018):
Kim et al. (2088); Kornilova Ei delman (2019); Kury al. (2020);
Lai et al. (2017); 7>; Lake Baroni (2018); Lebret al. (016)) Lewis 2022 2019a);
et al. (2017) et al.
Lin al. (2020a)
Continued page
25

---

Published as a conference paper at ICLR 2025
Table 8: References and licenses for alignment-tuning (text) dataset collections presented this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
Dynosaur (cont'd)
Various
Lin al. (202bb 2019 2022 Ling
et al. (2017): Liu et al. (2019);
Louis et al. (2020); Lowe al. (2016); Malo et al. (2013); Mar- tin et al. (2018); Merity et al. (2016); MMihaylov et al. (2018) 8); Mishra et al. (2023); Moniz & Torgo (2018); Mostafazadeh et al. (2020); Nan et al. (2021); Narayan al.18) Nguyen al. (2021)
Nie et al. (2020): Novikova et al. (2017) Paik (2021) 1 Pakhomov ct al. (2010); Pa Pang & Lee (2003);
Pavlichenko et (2021) Pedersen et al. (2007): Perez- P Beltrachini et al. (2019); Petroni et al. (2019); Pham et al. (2023); Rajani et al. (2019); Rajpurkar et al. (l.((21)0)))00 (2016b); Rameshku mar & Bailey 2020); Rashkin al. Rastogg et al. (2020); Royer
al. (2018); Rush al. (2015). Rust et al. (2023); Saeidi et al. (2018); Saha et (2018); Sak- aguchi et al. (2019) g Sanh et al. (2022); [Sap al. (211)); Schulz et al. (2020); See et al. (201)) Sharma (2019) Shriberg al. (1998); Sileo & Moens (2023)
Soleimani et al. (2021) (tolckk et al.
(2000); Tafjord et al. (2019) 2018).
Talmor et al. (2019); Tandon et al. (2019) Tang et al. (200)) Thawani et al. al. 2021h: horne et all (2018). Tyleček & Sára 2213 Ullrich et al. (2023); Ushio et al. (2023) Wang
al. (2.02a 2020c 2019 2023a)
Warstadt et al. (2023); ; Welbl et al.
(2018) 2017): Weller al. (2020); Weston et al. (0015) Williams et al. (2020); Wu et al. (2018); Xiong al. (2019a) Yang et al. (2018) Yu
al. (2019b); Zellers et al. (2019)
Zhang et al. (016 2023c 2019
Zhou et al. (2019 2023a): Zhu et al. (0022
EverythingL.M
MIT License
ExpertQA
MIT License
Malaviya et al. (024)
Feedback Coll.
MIT License
Kim et al. 2024
Continued on next page
26

---

Published as a conference paper at ICLR 2025
Table 8: References and licenses for alignment-tuning (text) dataset collections presented this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
Glaive Code Asst.
Apache License 2.0
Gretel Text-to-SQL
Apache License 2.0
HelpSteer
CC BY 4.0
Wang et al. (002bb)
Indic-Instr.
Various
Gala ct al.
(2024)
InstAr
Various
Hu et 2020
Einea
2019
Mozannar et al. (2119) Pratapa et al.
Chouikhi |
et (2024); Ab-
bas et al. (201 Abdelghany et al.
(2020); Orabi et al. (2020); ElSa
har & El-Beltagy (2015): Elnagar &
Einea (2016); Pieri et al.
(2024); AI
ghamdi et al. (2022) Abdallah al.
(2024); Biltawi et al.
(2020) Aloui
al. (2024); El-khain
(l.24))
(2016)
KIWI
BY-SA 4.0
Xu et (2024a)
LMSSS-Chat-IM
LMSYS-Chat-II 1M Dataset License,
Z.heng et al. (024a))
Anthropic, Llama
Llama2-MedTuned
CC BY-NC 4.0
Rohanian al.
2023
LongAAign-
Anthropic, Apache License 2.0
Bai et al.
2024b
Magpie-Pro
Meta Llama3 Community License
Xu et al.
2024b
MathDial
CC BY-SA 4.0, MIT License
Macina et al. (2023
MathInstr.
MIT License
Yue et (2023)
MedInstr.
Unspecified
Zhang et al. (0024
Medical Meadow
Various
Han et al.
(023):
Wang et al. (2020); Savery
(2020b): Jin et al.
al. (2020)
Mega
CC BY-SA 4.0
Barham al. 2023
MctaMathQA
MIT License
Yu ct al. 2023
Nectar
Various
No Robots
CC BY-NC 4.0
Open Asst. v2
Apache License 2.0
Köpf et al. (023)
Open-Platypus
Various
Sawada et al. (2023): ; Dettmers et al.
(2023);
Lightman al. (2233) Yu
et al. 2020); ; Wang et al.
2024
Lu
al.
2022): Chen
2023b
OpenGPT Healthcare
Unspecified, OGL 3.0
OpenMathInstr--1
Custom, MIT License, Apache Li- Toshniwal et al.
2024
cense 2.0
Orca-Math
Various
Mitra et al.
2024
PIl-Masking-200k
Non Commercial
PMC-LLaMA Instr.
Unspecified, Apache License 2.0
Wu et al.
2023 ; Jin et al.
2019
Pure-Dove
Apache License 2.0
PygmalionAl-PIPPA
Apache License 2.0
Gosling et al.
2023
Continued on next page
27

---

Published as a conference paper at ICLR 2025
Table 8: References and licenses for alignment-tuning (text) dataset collections presented this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
Reasoning
Apache License 2.0
RiddleSense
MIT License
Lin et al. (02
SeaBench
Apache License 2.0
Nguyen et al.
2023
SelFee
MIT License
Ye et al. 2023
Synth--GSM8K-2
Meta Llama3 Community License
Thai Gen AI
Various
UltraChat-200k
CC BY-NC 4.0
Ding et al.
2023
UltraFeedback Argilla
Various
WildChat
A12 ImpACT License Low Risk
Zhao et al.
2023
28

---

Published as a conference paper at ICLR 2025
Table 9: References and licenses for audio dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
1111 Hours Hindi
Custom
Bhanushali et al.
2022
120h Spanish Speech
CCO 1.0
AFRISPEECH-200
CC BY-NC-SA 4.0
Olatunji et al.
2023
AISHELL-1
Apache 2.0
Bu et al.
2017
AISHELL-2
Unspecified
Du et al
2018
AISHELL-4
CC BY-SA 4.0
Fu et al.
2021
ALLSSTAR
CC BY 4.0
Bradlow
(2010)
AMI
CC BY 4.0
Carletta et al. (2006)
Aalto Finnish Parl.
Custom
Virkkunen et al.
2022
African Acc. French
Apache 2.0
Basq., Cat. and Gal.
CC BY-SA 4.0
Kjartansson al.
2020
Bloom Speech
Various
Leong al.
2022
Bud500
Apache 2.0, CC BY-NC-SA 4.0
CSJ
Custom
Mackawa
2003
CSLU 1,2
CSLU Agreement
Lander, T
2007
CSLU 22 Langs.
CSLU Agreement
Lander,
2005
ClarinPL
CC BY 4.0
Korzinck et al.
2017
CoNASE
Custom
Coats 2019
CoVoST-2
CCO 1.0
Wang al. (020a)
Common Voice 17
CCO 1.0
Ardila et al. (2020a)
Crowd Sourced Speech
CC BY-SA 4.0
Kjartansson et al.
(2018)
Czech Parliament
CC BY 4.0
Kratochvil et al.
2020
DiDiSpeech
Unspecified
Guo et al. 2021
Earnings-22
Unspecified
Del Rio et al.
(2022)
EdAcc
CC BY-SA 4.0
Sanabria et al.
2023
Eng. Acc. in Brit. Isles
CC BY-SA 4.0
Demirsahin et al. (2020)
Inglish-Vietnamen
CC BY-NC-ND 4.0
Nguyen al. 202
FT SPEECH
Custom
Kirkedal et al. 2020
Fisher
LDC User Agreement
Cieri et al. 2004
Fleurs
CC BY 4.0
Conneau et al. 2022
GigaSpeech
Apache 2.0
Chen et al.
2021b
Golos
Custom
Karpov ct al. 2021
Hebrew Coursera
Unspecified
Hebrew Kan
Unspecified
Highland Puebla Nahuatl
CC BY-NC-SA 3.0 Unspecified
Shi al. (2021
JTUBESPEECH
Takamichi et al.
2021
Japanese Anime Speech
CCO 1.0
KSC
CC BY 4.0
Khassanov et al.
(021)
Kathbath
CCO 1.0
Javed al. 2022
KeSpeech
Custom
Tang et al.
2021
KsponSpeech
Unspecified
Bang et al.
2020
Continued on next page
29

---

Published as a conference paper at ICLR 2025
Table 9: References and licenses for audio dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
LJSpeech
Public Domain
Ito & Johnson
(0117)
LaboroTVSpeech
Custom
Ando & Fujihara
(202)
LibriSpeech
CC BY 4.0 Custom
Panayotov et al.
2015
M-AILABS
Solak (024))
M2ASR
Unspecified
Shi et al. 2017): 7) Mamtimin et al.
(2023) Zhi et al.
(2021):
Li et al.
2017
MAGICDATA
CC BY-NC-ND 4.0
MASC
CC BY 4.0
Al-Fetyani et
2023
MaSS
Unspecified
Boito et al.
2020
MagicData-RAMCC
CC BY-NC-ND 4.0
Yang et al.
2022
MediaSpeech
CC BY 4.0
Kolobov al.
(2021
Minds14
CC BY 4.0
Gerz al. 2021
MuST-C
CC BY-NC-ND 4.0
Di Gangi et al. (2019)
Multiling. LibriSpeech
CC BY 4.0
Pratap et al. 2020
Multiling. TEDx
CC BY-NC-ND 4.0
Salesky al.
(022)
NST Danish
CCO 1.0
NST Norwegian
CCO 1.0
NST Swedish
CCO 1.0
Nigerian English
CC BY-SA 4.0
Norwegian Parl.
CCO 1.0
Solberg Ortiz
2022
Norwegian Parl. Speech
CCO 1.0
Solberg Ortiz
2022
OLKAVS
Custom
Park et al. 20223
OpenSTT
CC BY-NC 4.0
Andrusenko et al.
2020
People's Speech
Various
Galvez al. (202
QASR
Unspecified
Mubarak et al.
(021
RTVE
Custom
ReazonSpeech
CDLA-Sharing-1A.
Yin et al.
2023
Regional Af. Am. Lang.
CC BY-NC-SA 4.0
Rix Vox
CC BY 4.0
SDS-200
Custom
Plüss et al. 2022
SPGISpeech
Custom
O O'Neill et al. (2021)
Samromur
CC BY 4.0
Mollberg et 2020
Samromur Children
CC BY 4.0
Hernandez Mena et al.
2022
Samromur Milljon
CC BY 4.0
Shrutilipi
CCO 1.0
Bhogale et al.
2022
Snow Mountain
CC BY-SA 4.0
Raju al. (2033)
Spoken Wikipedia
CC BY-SA 4.0
Baumann et al. (2019
Switchboard
LDC User Agreement
Godfrey et al. (192
TED-LIUM3
CC BY-NC-ND 3.0
Hernandez et al. (2018
THCHS-30
Apache 2.0
Wang & Zhang
2015
THUYG-20
Apache 2.0
Rozi et al. (2015)
Continued on next page
30

---

Published as a conference paper at ICLR 2025
Table 9: References and licenses for audio dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
TIMIT
LDC User Agreement
Garofolo, John S. et al.
1993)
VCTK
CC BY 4.0
VibraVox
CC BY 4.0
VoxPopuli
CCO 1.0
Wang et al. 2021
Vystadial
CC BY-SA 3.0
Korvas et al.
(2014)
WenetSpeech
BY BY 4.0
Zhang al.
(202aa)
West Afr. Radio
CC BY-SA 4.0
Doumbouya
(0021
West Afr. Virt. Asst.
CC BY-SA 4.0
Doumbouya et
2021
YODAS
CC BY 3.0
Li et al.
2023c
Seroth-Korean
CC BY 4.0
aidatatang
CC BY-NC-ND 4.0
31

---

Published as a conference paper at ICLR 2025
Table 10: References and licenses for video dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
100DOH
Custom
Shan et al.
2020
SOB-TIOM
Custom
Goyal ct al. (2017)
20BN-jester
Custom
Materzynska et al.
(2019)
50 Salads
CC BY-NC-SA 4.0
Stein & McKenna
(2013)
ActivityNet
MIT License
Heilbron et al.
(2015)
Apes
Unspecified
Alcazar ct al
(022
Ava
CC BY 4.0
Roth al. 2019
Gu et al. 2018
Breakfast
CC BY 4.0
Kuchne et (2014)
CDAD
Unspecified
Xiang et al.
(0022)
COIN
Custom
Tang et al.
2019)
Charades
Custom
Sigurdsson al.
(0016b)
Charades-Ego
Custom
Sigurdsson et al.
2018
Collective
Unspecified
Wongun Choi et al.
2009
Condensed Movies
CC BY 4.0
Bain et al. (2020)
CrossTask
Unspecified
Zhukov et
(2019
Davis
Custom
Perazzi et al.
2016
DiDeMo
BSD 2-Clause License
Hendricks et al.
2018
Disney Vid. Gen.
Apache 2.0
EEV
CC BY 4.0
Sun 202
EPIC-KITCHENS
CC BY-NC 4.0
Damen et al. (2018)
Ego4D
Custom, MIT License
Grauman et al
(2022)
FERV39k
CC BY-NC 4.0
Wang et al. (202bb)
FineGym
CC BY-NC 4.0
Shao al. 2020
Video
CC BY 4.0
HAA500
Unspecified
Chung et al.
2021
HACS
Custom
Zhao et al. 2019
HD-VILA-100M
Custom
Xue al. 2022)
HMDB
CC BY 4.0
Kuchne et al. (201)
HOLLYWOOD2
Unspecified
Marszalek et al.
(0009
HOMAGE
Unspecified
Rai et al. 22021
HVU
Custom
Diba al.
2020
Hollywood Ext.
MIT License
Bojanowski al.
2014
How2
Various
Sanabria et al.
2018
H0WTo100M
Unspecified
Miech al.
(0099)
ImageNet- Vid
CC BY-NC
Russakovsky al.
2015
Kinetics
Unspecified
Kay et al. (2017) )
Carreira et al.
(2018) ; Smaira al. (2020)
LEMMA
Unspecified Custom, MIT License
Jia et 2020
LSMDC
Rohrbach et al.
2016a); Sharma
al. (2020)
M-MiT
Unspecified
Monfort et al.
2021b
Continued next page
32

---

Published as a conference paper at ICLR 2025
Table 10: References and licenses for video dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
MAD
Custom
Soldan et al. (022)
MMAct
Custom
Kong ct al. 2019)
MPII
Unspecified, Custom
Rohrbach et al. 2016b
2015
MSA
Unspecified
Xiong et al.
(2019b)
MSR-VT VTT
Unspecified
Xu et al.
(016
MVBench
MIT License
Li al.
(022))
Mars
Unspecified
Zheng al. (2016)
Mimetics
Unspecified
Weinzaepfel
Rogez
202
Moments in Time
Custom
Monfort et al.
2019bb
Movie-Net
Unspecified
Huang et al.
2020
MovieGraphs
Custom
Vicol
2018
MovieQA
Unspecified
Tapaswi et al. (2016)
MovieScenes
Unspecified
Rao et al. (020
MuliTHUMOS
CC BY 4.0
Yeung al.
(201 7
NTU RGB+D
Custom
Shahroudy
(2016
Narrated Instr. Vid.
MIT License
Alayrac al.
20
OmniSource-We Web
Apache License 2.0
Duan et al. 2020
Oops!
CC BY-NC-SA 4.0
Epstein et al. (2020)
Vid-Vid
CC-BY-4.0
Nan al.
(2024)
PKU-MMD
Unspecified
Liu et al.
2017
Project-Aria
Apache License 2.0 Unspecified
Pan et al.
(2023);
Lv ct al. 2024
QFVS
Sharghi al.
(2017
QuerYD
Unspecified
Oncescu
(202)
RareAct
Unspecified
Miech et al.
2020
SOA
Unspecified
Diba et al.
2020
ShareGPT4Video
Citibution-NonCommmnn 4.0 In- ternational
Chen et al.
2024
Spoken Moments
Custom
Monfort et al.
2021 a
Sports- 1M
CC BY 3.0 Unspecified
Karpathy et al 22014
StoryGraphs
Tapaswi et 2014
SumMe
Unspecified
Gygli ct al. 2014
TGIF
Custom
Li et al.
2016)
THUMOS
Custom
Idrees et al. (2017
TITAN
Non Commercial
Malla et al.
2020
TRECVid
CC BY-NC-SA 4.0
Awad al. 2020
TVSum
CC BY 3,0
Yale Song et al.
(0015
VIRAT
Unspecified
Demir al.
2020
Toyota Smarthome
Custom
Das (2019)
UAV-Human
Custom
Li et
202
UCF101
Unspecified
Soomro et al. (0012
VIOLIN
Unspecified
Liu et al. 2020
VLOG
Custom
Fouhey et al. (0177)
Continued page
33

---

Published as a conference paper at ICLR 2025
Table 10: References and licenses for video dataset collections presented in this paper. Collections containing material under more than three distinct licenses are marked as having "Various" licenses, and we refer readers to our raw data for the full details. Datasets are sorted alphabetically for ease of dataset lookup.
Collection
Licenses
Cite
VTW
Unspecified
Zeng et al.
2016)
VaTeX
CC BY 4.0
Wang et al. 2220dd
VidProm
CC-BY-NC 4.0
Wang & Yang
20024b
VideoLT
Non Commercial
Zhang ct al. (0021
VideoStory
Unspecified
Habibian et al.
2014
Volleyball
Unspecified
Ibrahim al. 2016
VoxCeleb
Custom
Nagrani al.
(0018
WebVid
Custom
Bain et al.
(2022)
YouCook
Unspecified
Das et al.
(0013)
YouCook2
MIT License
Zhou et al.
2017
Youtube-8M
Unspecified
Abu-El-Haija et al.
(0166))
REFERENCES
Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb- Tarrant,, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. The multilingual alignment prism: Aligning global and local preferences to reduce harm, 2024. URL /tttpps://rxxi...///// org/ abs 18682)
Mourad Abbas, Kamel Smaïli, and D. Berkani. Evaluation of Topic Identification Methods on Arabic Corpora. Journal of Digital Information Management, 9(5), October 2011. URL https
/ //ww.dlingg info/ /
AbdeIrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. ArabicaQA: A Comprehensive Dataset for Arabic Question Answering, March 2024. URL http:://aaxiv.oorg//e/ LtXiv:2403.17844 [cs].
Ahmed Abdelghany, Hammam Abdelaal, Abdulrahman Kamr, and Passent Elkafrawy. Doc2Vec: An approach to identify Hadith Similarities. Australian Journal of Basic and Applied Sciences, pp. 46-53, December 2020. doi: 10.22587/ajbas. 2020.......0..1444 14.12.5.
Sami Abu-El- Haijaa,,,,, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vjjaynnaraiimhn. YouTube-8M: A Large-Scale Video Clas- sification Benchmark, September 2016a. URL http://aaxiv..orrrab//eeeeee arXiv:1609.08675 [es].
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaanarasimman. Youtube-8m: large-scale video classification benchmark. arXiv preprint 1rXXiv1609.000 (66600 2016b.
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constan- Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 9.1116-1131, 2021.
tine Lignos, Chester
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He. Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kebede Guge, and Pontus Stenetorp. Irokobench: A new benchmark for african languages in the age of large language models, 2024. hRLhttss:
https: /ttps:///rxxiee wttp://wwwwwgra/////eeeeeee 03368
34

---

Published as a conference paper at ICLR 2025
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. Topi- OCQA: Open-domain Conversational Question Answering with Topic Switching, February 2022.
URL htRptt:///aax...oo org/ abs/2110. 00768 arXiv:2110.00768 [cs].
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanee Language Model Pre-training, March 2021. URL
http://arxiv. org/ www./bssli.. /2010. 12688 arXiv:2010. 12688 [es].
Armen Aghajanyan, Lili Yu. Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative modell language models. In International Conference on Machine Learning, pp. 265-279. PMLR, 2023.
Alëna Aksënova, Daan van Esch, James Flynn, and Pavel Golik. How might we create better benchmarks for speech recognition? In Kenneth Church, Mark Liberman, and Valia Kordoni (eds.), Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pp. 22-34, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 11t333/1/22211pppppp bppf-11.4 1.4. URL
bpppf-14
Ekin Akyürek, Tolga Bolukbasi, Frederick Frederik Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards Tracing Factual Knowledge in Language Models Back to the Training Data, Octoher
2022. URI. ptttt://arxiv.coo
bs 2205. 1182 arXiv:2205. 1482 [cs].
Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. MASC: Massive Arabic Speech Corpus. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 1006-1013, Doha, Qatar, Jamuary 2023. IEEE. ISBN 7779-8-3003-966 doi: 10/gtsqzj. URL htttp::/eeeexpooeeeee./ ieee. org/ www.ddoouttnnnooeeoe wtt.uummenn1022252////
Jean-Baptiste Alayrae Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised Learning from Narrated Instruction Videos, June 2016. URL
09215 arXiv: [cs].
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXh::2402.16827 2024.
Juan Leon Alcazar, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbelaez, Bernard Ghanem, and Fabian Caba Heilbron. APES: Audiovisual Person Search in Untrimmed Video, June 2021. URL /ttt::///aaxiiii org abs 01 667 urXiv:2106.0166 [es].
Reem Alghamdi, Zhenwen Liang, and Xiangliang Zhang, ArMATH: Dataset for Solving Arabic Math Word Problems. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 351-362, Marseille, France, June 2022. European Language Resources Association. URL httph://///ccleeee http:///aalaantooog..ooooo wwwphhh/l.gg..../222..... rre-
37
Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 101 Billion Arabic Words Dataset, April 2024. URL http::::/aaa....rr///b////////// 01590 arXiv:2405.01590 [es].
Zaid Alyafcai, Maraim Masoud, Mustafa Ghaleb, and Maged S Al-shaibani, Masader: Metadata sourcing for arabic text and speech data resources. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6340-6351 2022.
Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran A. Q Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged S. Al-Shaibani. CIDAR: Culturally Relevant Instruction Dataset For Arabic, February 2024. URL
Settp://arxiinn org/ abs//4422 03177 arXiv:2402.03177 [es].
Aida Amini. Saadia Gabriel, Peter Lin, Rik Heanell-edzzioski. Yejin Choi, and Hannanch Hajishirzi. MathQA: Towards Interpretable Math Word Problem Solving with Operation- -Bssed Formalisms, 13319 arXiv:1905.1333 [cs].
May 2019. URL UtR.Lnttte//arrii..rr//aa///////
35

---

Published as a conference paper at ICLR 2025
Shintaro Ando and Hiromasa Fujihara. Construction of Large-seale Japas ASR Corpus on TV Recordings, March 2021. URL httptttpp//ai...ooot
abs wtwpg//ab//200....2
1rX2v:2103.14336
[cs, cess].
Andrei Andrusenko, Aleksandr Laptev, and Ivan Medennikov. Exploration of End-to-End ASR for OpenSTT Russian Open Speech-to-Tens Dataset. In Lecture Notes in Computer Science, volume 12335, pp. 35-44 Springer, Cham, 2020. doi: 10. http: /ttp:///axiv..../ hwwpg//bb//0u 2000 08274 arXiv:2006.08274 [es, eess].
URL
Mariona Coll Ardanuy, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Katherine McDonough, Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. Living Machines: A study of atypical animacy, November 2020. URL http://aax.vvoor//abb arXiv:2005. 1140 [es].
2005. 11140
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. arXiv preprint 19Xiv:
2019.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common Voice: A Massively- Multilingual Speech Corpus, March 2020a. URL http: / http:://aaxxvv.orr////////////// arXiv:1912.06670 [es].
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: massively- multilingual speech corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno. Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218 42222, Marseille, France, May 2020b. European Language Resources Association. ISBN 979-10-95546-344 URL https //aclanchology org. www.p//.ww...........0.......... 1rec-1.520)
Anthony B Atkinson et al. On the measurement of inequality. Journal of economic theory, 2(3): 244-263, 1970.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL http:///axxi...r//bbbbb stXiv22188.07322 [cs].
George Awad, Asad A. Butt, Keith Curtis, Yooyoung Lee, Jonathan Fiscus, Afzal Godil, Andrew Delgado, Jesse Zhang, Eliot Godard, Lukas Diduch, Alan F. Smeaton, Yvette Graham, Wessel Kraaij, and Georges Quenot. TRECVID 2019: An Evaluation Campaign Benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & Retrieval, September 2020. URL (RRL/ttp://aatt
org/oogg//a.... 2009. C9984 arXiv:2009.0900 09984 [cs].
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang. and Nando de Freitas. Playing hard exploration games by watching youtube. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https::/ fi #1ee3350922eeebee333cccaaaaa2223
fi Files/paper/2211 Pap.erdd-ppppppppppp.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathe- matics, February 2023. [cs].
abs / 2302. 12433 arXiv:2302.12433
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia. Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Von Platen, Yatharth Saraf, Juan Pino, et al. XIs-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint 2rXiv:2111.092290 2021.
36

---

Published as a conference paper at ICLR 2025
Yuelin Bai, Xinrun Du, Yiming Liang. Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang. Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Min Yang, Shiwen Ni, and Ge Zhang. COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning, March 2024a. URL httt:://axii org/ abs 2403. 18058 arXiv:2403. 18058 [es] version: 1.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauma Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022. URL http: /wwpiv/aaivvv//aa org/ abs 2204 05862 arXiv:2204.05862 [cs].
Yushi Bai, Xin Lv, Jiajie Zhang. Yuze He, Ji Qi. Lei Hou, Jie Tang, Yuxiao Dong. and Juanzi Li. LongAlign: A Recipe for Long Context Alignment of Large Language Models, January 2024b. URL httpthttp///iii// org/ /tabs///01. 1....11800 18058 arXiv:2401.18055 [es].
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed Movies: Story Based Retrieval with Contextual Embeddings, October 2020. URL http:::/aaxxi.or//baboo. 04208 arXiv:2005.042: 04000 [cs].
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval, May 2022. URL httt:://aaxxiit org /b/////////// 2104 00650 arXiv:2104.00650 [cs].
Payal Bajaj. Daniel Campos. Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenherg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, October 2018. URL http:ttt////xii.orggg/. [es].
09268 arXiv:1611.09268
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. Constrained Decoding for Neural NLG from Compositional Representations in Task- Oriented Dialogue, June 2019.
906. C7220 1eXiv:190.007220 [cs].
Jack Bandy and Nicholas Vincent. Addressing "documentation debt" in machine learning research: retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241, 2021.
Jeong-Uk Bang, Seung Yun, Seung-Hi Kim, Mu- Yeol Choi, Min-Kyu Lee, Yeo-Jeong Kim, Dong- Hyun Kim, Jun Park, Young-Jik Lee, and Sang-Hun Kim. KsponSpeech: Korean Spontaneous Speech Corpus for Automatic Speech Recognition. Applied Sciences, 10(19):6936, January 2020. ISSN 2076-3417. doi: 10/gtwwek. Utt / //www wwwwwwwwpp.... mdpi. com 201/207666 6- 3417/10/19/ 6936
Number: 19 Publisher: Multidisciplinary Digital Publishing Institute.
Samuel Barham, Orion Weller, Michelle Yuan, Kenton Murray, Mahsa Yarmohammadi, Zheng- ping Jiang, Siddharth Vashishtha, Alexander Martin, Anqi Liu, Aaron Steven White, Jordan Boyd-Graber, and Benjamin Van Durme. MegaWika: Millions of reports and their sources across 50 diverse languages, July 2023. URI. http::///axxii..o//bbbbbbs arXiv:2307.07045 [cs].
Max Bartolo, Alastair Roberts, Johannes Welbl, Sehastian Riedel, and Pontus Stenetorp. Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension. Transactions of the Association for Computational Linguistics, 8:662 678, December 2020. ISSN 2307-387X. doi: 10/gizgwj. URL URL/http://arxx http::///aawww.o.//aabooao// /200 00293. ar arXiv:2002.00299 [cs].
Timo Baumann, Ame Köhn, and Felix Hennig. The Spoken Wikipedia Corpus collection: Harvesting, aligmment and application to hyperlistening. Language Resources and Evaluation, 53(2): 303-329, June 2019. ISSN 1574-0218, doi: 10/gg5xdf. URL https: //doi. org/10. 1007/ - 94100y
37

---

Published as a conference paper at ICLR 2025
Emily M. Bender and Batya Friedman. Data statements for natural language processing: To- ward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604, 2018. doi: 10.1162/tacl a_00041.
URL https:
//aclantho warechhooooy....oo www.ooggggo1/1
Anish Bhanushali, Grant Bridgman, Deekshitha G, Prasanta Ghosh, Pratik Kumar, Saurabh Kumar, Adithya Raj Kolladath, Nithya Ravi, Aaditeshwar Seth, Ashish Seth, Abhayjeet Singh, Vrunda Sukhadia, Umesh S. Sathvik Udupa, and Lodagala V. S. V. Durga Prasad. Gram Vaani ASR Challenge on spontaneous telephone speech recordings in regional variations of Hindi In Inter- speech 2022, pp. 3548- 3552. ISCA, September 2022. doi: 10gtsqzz. URL httpst// /ww isca-archi ve. org/ www.inneeerreeeeee itt http://erreeeeeeeee//////aaaeeee http://hhnnushlii22_naa
html
Kaushal Santosh Bhogale, Abhigyan Raman, Tahir Javed, Sumanth Doddapaneni, Anoop Kunchukut- tan, Pratyush Kumar, and Mitesh M. Khapra. Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages, August 2022. URL
http: org/ abs/ 2208 12666 arXiv:2208. 12666 [cs, cess].
Mariam Biltawi, Arafat Awajan, and Sara Tedmori. Arabic Reading Comprehension Benchmarks Created Semiautomatically. In 2020 st International Arab Conference on Information Technology (ACIT), pp. 1-6, Giza, Egypt, November 2020. IEFF. ISBN 978-1-72818-855- doi: 10/g6k6b8.
URL https: http::://tteeepeeeeee/ /tte:://eeeeeppreeeeoo leee org/ http://dccummee//30000.0
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01663, 2021.
Abeba Birhane, Vinay Prabhu, Sang Han, Vishnu Naresh Boddeti, and Alexandra Sasha Luccioni. Into the laions den: Investigating hate in multimodal datasets. arXiv preprint arX1v:2311.03449, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language, November 2019. URL http:://axxv..orr 1911. arXiv:1911. 1161 [cs].
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. URL
fttps://arxiv. http:///aaxxvv....aa.... /2111.111111111111
Marcely Zanon Boito, William N. Havard, Mahault Garnerin, Eric Le Ferrand, and Laurent Be- sacier. MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible, February 2020. URL 12Xv:1990.112895 [cs].
1907. 12895
Piotr Bojanowski. Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef' Sivic. Weakly Supervised Action Labeling in Videos Under Ordering Constraints, July 2014. URL 1208 arXiv:1407. 1208 [cs].
http::/arxiv. www.orrg//b.... abargg/ab//ii http::///arxlv.ooor///////10
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index, 2023.
Michael Boratko, Xiang Lorraine Li, Rajarshi Das, Tim O'Gorman, Dan Le, and Andrew McCallum. ProtoQA: Question Answering Dataset for Prototypical Common-Senss Reasoning, October 2020. URL http: /tttp://axxiv.. org/ abs 2005.00000000 C0771 arXiv:2005.0077 [cs].
Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. Learning To Split and Rephrase From Wikipedia Edit History, August 2018 http:httt//axxiii wwwp/iiiorrg//bb.. 1808. 09468 MxXiv:880.00946 [cs].
Florian Boudin and Ygor Gallina. Redefining Absent Keyphrases and their Effect on Retrieval Effectiveness. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4185-4193, Online, June 2021. Association for
38

---

Published as a conference paper at ICLR 2025
Computational Linguistics. doi: 10/g6k64d. URL htt.:tss//aalaatttooooooo/////// maaci-main.2 330
A.R. Bradlow. ALLSSTAR: Archive of LI and L2 Scripted and Spontaneous Transcripts And Recordings, 2010. URL attpp://speechboxx wwt..:ntis...cceee....... www.:usss..tttttttee.....
11/2goto=allssee
Alex Bravo, Janet Pinero, Nuria Queralt-Rosinaee Michael Rautschka, and Laura I. Furlong. Extrac- tion of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC Bioinformattics, 16(1 1651)555, February 2015. ISSN 1471-2105. doi: 10/f7kn8s. URL htt.htttss:/ce //co: wrg. 118.11166686 12011166/1189-81 -15-0472-3
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL httpss:////// com/ htmm/researchh
(eGeeo-generatinn -onealoonnnnmmsssss as-wo wo-worree ld simulators
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., nwwrineurrccoooo ccwpape files//aper/2o20/
2020a. URL htttp::/pproceeeiiiii Fiee2455cccbrr033571188333222222 pappaaaeee.cccceeeeeeeaeeeeeeeee
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Vosss.. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020b. URL org/ abs /2005.205.0000 14165 arXiv:2005.1 14165. [cs].
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. AISHELL-I: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline, September 2017. URL http:://axiy arXiv:1 NDw:1709.0555 709.05522 [cs].
org/abs/17e 1709. 05522
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.). Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pp- 77-91. PMLR, February 201 8a. URL htRl:tts:://// htt.ttttp:s/prroeeeiingg mlr press/v8) v83ssvv81. /uuu amwi ni 8aa. html
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. Sorelle A. Friedler and Christo Wilson (eds.). Proceedings of the Ist Conference Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pp. 77-91. PMLR, February 2018b. URL httptttt/////tt /tt/https//pprrcedings mlr press/vee /t..///1//// 1/0///u www.mmmnii.... 18a. html
Bill Byrne, Karthik Krishmamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. T Toward a Realistic and Diverse Dialog Dataset, September 2019. URL arXiv: 1w..vv1999.00. 05358 [cs].
abs/ 1909. 05358
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base, June 2022. URL htttt:://rxxv..oo/ wwtpxiii.org/as///oo 2007. 03875 MXXv:2007.03875 [cs].
39

---

Published as a conference paper at ICLR 2025
Shuyang Cao and Lu Wang. Controllable Open-ended Question Generation with A New Question Type Ontology, June 2021. URL Attp://arxx...o2g///b///200 00152
arXiv:2107.00152 [es].
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, lain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. The AMI Meeting Corpus: A Pre-annen are-announcemen. In Steve Renals and Samy Bengio (eds.), Machine Learning for Multimodal Interaction, volume 3869, pp. 28-39. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006. ISBN 978-3--40-33254999 doi: 10.1007/1 1677482_3 URL http: Series Title: Lecture Notes in Computer
Wrinkksppringe Science.
10. /ttp:/1111100011777818100101
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. February 2022.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. OpenReview, 2023a.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Schwag, Florian Tramèr, Borja Balle, Daphne Ippolito. and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, Anaheim, CA, August 2023b. USENIX Association. ISBN 978-1-999113-337-3, URL /www/
org/
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A Short Note about Kinetics-600, August 2018. URL http:::://axxiv.rrppp/. a808 01340 arXiv:1108.01 003400 [es].
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient Intent Detection with Dual Sentence Encoders. In Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kumar, lñigo Casanueva, and Rushin Shah (eds.), Proceedings of the 2nd Workshop Natural Language Processing for Conversational AI, pp. 45. Online, July 2020. Association for Computational Linguistics. doi: 10/gjhzzs. URL MtXiv:2003.0480 [cs].
attps://accaatthlogyy wrg. wtlpgg/222.com
2020 wwwp::/w...ccc.oooooomo........e
Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii- -hsii-Oshhih,
Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. arXiv preprint arXiv:2103.12028, 2021.
Sarah Huiyi Cen, Aspen Hopkins, Andrew Ilyas, Aleksander Madry, Isabella Struckman, and Luis
Videgaray Caso. AI Supply Chains, April 2023. URL http::////ok.ooioorr11001....11
ssrn. 4789403
Alberto Cetoli, Mohammad Akbari, Stefano Bragaglia, Andrew D. O'Harne, and Mare Sloan. Named Entity Disambiguation using Deep Learning on Graphs. volume 11438, pp. 78-86. 2019. doi: (007/978-3-030-1777 arXiv:1810.0916. [cs].
10. URL UBUlstto://arxxv.orr//bbb. wtt.:/orrraas//111000 09164
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural Legal Judgment Prediction in English, June 2019a. URL [es].
02059. arXiv:1906.022
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. Large-Scale Multi-Label Text Classification on BU Legislation, June 2019b. URL http::://aaxxivorggg abs/1905. 02192 T.XXv:1906.02199 [cs].
Ilias Chalkidis, Manos Fergadiotis, Nikolaos Manginas, Eva Katakalou, and Prodromos Malakasiotis. Regulatory Compliance through Doc2Doc Information Retrieval: A case study in EU/UK legis- lation where text similarity has limitations, January 2021. URL http:: /twxiviivv.///////v/ org/ /bs/
21 10726 11Xiv:210.10722 [cs].
40

---

Published as a conference paper at ICLR 2025
Jun Shern Chan, Michael Pieler, Jonathan Jao, Jérémy Scheurer, and Ethan Perez. Few-shot Adapta- tion Works with UnpredicTTable Data, August 2022. URL http:: www.awwwww....aaaaooooaaaoaeeraa wwt./ba//aa../ooooo/e//..eeeeabe abs/ 2208 01009 arXiv:2208.0100 [cs].
Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4283-4294 Curran Asso- ciates, Inc., 2020. [22p://1|22222222000a2333333e112
httpa//pppppppppppeeeeeee//epppp files/paper/
Xinyu Chang. Gender bias in hiring: An analysis of the impact of amazon's 's recruiting algorithm. Advances in Economics, Management and Political Sciences, 23:134-140, 09 2023. doi: 10.54254/ 2754- 1 11//6////////220667
Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, and Chloe Clavel. Hierarchical Pre-training for Sequence Labelling in Spoken Dialog, February 2021. URL http://arxiv org/ abs / 2009. 11152 arXiv:2009.1 152 [es].
Jose M. Chaquet, Enrique J. Carmona, and Antonio Femmnder-Cahhallero. A survey of video datasets for human action and activity recognition. Computer Vision and Image Understanding. doi: 00.1016/..viu.2213.1..... URL http:
117(6)6333-659, June 2013, ISSN 1077-3142.
//dx.doi.org 1016 cwwu. http:///1...111/.......211....0. 01.
Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3167-3185, Online, June 2021. Association for Computational Linguistics. doi: 10/gtsqxV. URL. httppsss/aclaaahoooooooooo 202rg/2221..0 naw.l-aaallaaam
254
Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tammoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3002-3017, Online, June 2021 a. Association for Computational Linguistics. doi: 10/gtsqxt. URL W.R.httt... httph:::t/........... haaaaaaaaaaaoocccaaacaaao ogy org 2021 naac 1 ma in. 239
Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuai- jiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, Zhao You, and Zhiy- ong Yan. GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours Transcribed Audio, June 2021b. URL http: eess].
URLhttp: /tap:://arxivvi httpixxxivv.///.i// wbw.gg/ab//221...m 06909 arXiv:2106.06909 [cs,
Hannah Chen, Yangfeng Ji, and David Evans. Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory, November 2020a. URL http://rxxa org/ axg/abs//e 2011. 01856 arXiv:201 1.01856 [cs].
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, June 2024. URL http: //arxiv. /larxiv.org/ablell. abs////////// www.aab//24.... 04325 arXiv:2406.04325 [cs].
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tuu Tur. PLACES; Prompting Language Models for Social Conversation Synthesis, February 2023a. URL arXiv:2302.03269 [es].
03269
41

---

Published as a conference paper at ICLR 2025
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. SummScreen: A Dataset for Abstractive Screenplay Summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8602-8615, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10/gtsqxx. URI /tsss://alaattttt ogy. wtw..orgg/222..cl1111ege.
589
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention, June 2019. URL lattp://aaxiaa org/ / /105 12866 19Xiv:1005.1200 [cs].
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang. and William Yang Wang. HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1026-1036, Online, November 2020b. Association for Computational Linguis- tics. doi: 10/gpmd4x. URL httl:htts:///ee http:p:://cclaaatooooooooor/r/// wtw.:020...iimm Windings-emtn
91
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: Theorem-driven Question Answering dataset, December 2023b, URL http::/arxiv. org. abs 2305. 12524 arXiv:2305. 12524 [cs].
Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can language models be instructed to protect personal information? 2023c.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A Real-Life Scenario Dialogue Summarization Dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 5062-5074, Online, August 2021c. Association for Computational Linguistics. doi: 10/gtsqxs. URL https
//aclanthologe htwphhhhooooyy.o/2221111111 wwt.ooggg.2oo... f Eindings-acl... 449
Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, and William Yang Wang. KETOD: Knowledge- Enriched Task-Oriented Dialogue, May 2022b. URL http://
arxiv. www.vv/orggabb//20200 05589 arXiv:2205.0555 [cs].
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation, March 2022. URL arXiv:2108.06 0088 12 [cs].
06712
Ssu Chiu, Maolin Li, Yen-Ting Lin, and Yun-Nung Chen. SalesBot: Transitioning from Chit-Chat to Task-Orienter Dialogues. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 6143-6158, Dublin, Ireland, May 2022, Association for Computational Lin- guistics. doi: 10/gtsqxw. (tthtttss:// /tap://///laa/hhoooo www.y.org/2222. -lng. 425.
Hasna Chouikhi, Manel Aloui, Cyrine Ben Hammou, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. GemmAr: Enhancing LLMS Through Arabic Eestuuction-Tunngg July 2024. URL
httt:://axiv. org/ abs 2407 02147 arXiv:2407.02147 [cs].
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism 01103.0330005
prediction instruments, February 2017. URL
a0X [cs, stat].
Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, and Gerhard Weikum. Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Ju- dicious Context Expansion. In Proceedings of the 28th ACM International Conference on In-
formation and Knowledge Management, pp. 729-738, November 2019. doi: 10/gkz233. URL http://arxiv> o abs/ abs/1910 03262 arXiv:1910.0322 [cs].
Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang, Yu-Wing Tai, and Chi- Tang. HAA500: Caman-Cennric Atomic Action Dataset with Curated Videos, August 2021. URL http:// arxiv. wwxiv.or//bb.. 2009 05224 arXiv:2009.05224 [cs, eess).
42

---

Published as a conference paper at ICLR 2025
Christopher Cieri, David Miller, and Kevin Walker. The Fisher Corpus: Resource for the Next Generations of Speech-to-Texx In Maria Teresa Lino, Maria Francisca Xavier, Fátima Ferreira, Rute Costa, and Raquel Silva (eds.), Proceedings of the Fourth International Conference Language Resources and Evaluation (L.REC"04), Lishon, Portugal, May 2004. European Language Resources Association (ELRA). URL ht http: wttp://www.. 1 rec- conf wttg/pocceeeooeccooooooooooeoe ngs/
Lxec2004/pdE/67.poo
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, May 2019. arXiv: 1199900000044 [cs].
URL http:t//aaxi....oom org/ abs 1905 10044
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge, March 2018. URLhttp: wttp://aaxvvvvv.. www.wwwwwwww.../aaaaaaaa abtp///bb///// 05457 arXiv:1833.05457 [es].
Steven Coats. A Corpus of Regional American Language from YouTube. In Proceedings of the Digital Humanities in the Nordic Countries 5th Con-
ference, 2019. Baneall-neeeiiannnnaaaaaaaaa From-YouTubeeeoots:
URL hntpss://www A-Corpus-of Regii -eelll-mmeerii
Steven Coats. Dialect corpora from youtube. Language and linguistics in a complex world, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL http: 14168 arXiv:2110.4416 [es].
//arxiv.orq//////
Samantha Cole. Nvidia sued for scraping youtube after 404 media investi- gation. 404 Media, August 2024.
URL https:// 4medmaa /o/
www.cia-suedd-oo-scrrpppp
for- http:///ooooooooooo-oo---o-o---- wtt.raeeeerr----eee------eeeeeor
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: Few-shot Learning Evaluation of Univer- sal Representations of Speech, May 2022. URL arXiv:2205.1 12446 [cs, eess) version: 1.
12446
Sam Coope, Tyler Farghly, Daniela Gerz, Ivan Vulić, and Matthew Henderson. Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations, July 2020. org/ abs/ /205 08866 arXiv:2005.08886 [cs].
URL http:
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. Snips Voice Platform: an embedded Spoken Language Understanding system for private- by-design voice interfaces, December 2018. URL httpt/p:/// attp//aaivvvv. wwwp:////g.......//////////..o 1rXiv:1805.10193 [cs].
abs
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi. Bryan Catanzaro, and Wei Ping. Nylm: Open frontier-class multimodal 1lms. arXiv preprini, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling Egocentric Vision: The EPIC-KITCHENS Dataset, July 2018. URL
abs/1800 02748 arXiv:1804.022 [cs].
Verna Dankers, Christopher Lucas, and Ivan Titov. Can Transformer Too Compositional? Analysing Idiom Processing in Neural Machine Translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 3608-362 Dublin, Ire- land, May 2022. Association for Computational Linguistics. doi: 10/g6k6xn. URL https? 7/aclane ogy. ogw.oggg.......... acl long. 252
43

---

Published as a conference paper at ICLR 2025
Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A Thousand Frames in Just Few Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In 2013 IEEE Conference Computer Vision and Pattern Recognition, pp. 2634 2641, Portland, OR. USA, June 2013. IEEE. ISBN 978-0-7695-49887 doi: 10/gtsqzr. URL.ttt /ttp:///eeeepllee lore
Words:
leee. org http///cccmmttt6611/.3.3
Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota Smarthome: Real-World Activities of Daily Living. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 833-842, Seoul, Korea (South), October 2019, IEEE. ISBN 978-1-72814-8033 doi: 10/ghfje7. URL https://
leeexplore leee. org/ doc.ment//////cccc/oooo / 90081357
Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. Quoref: A Reading Comprchension Dataset with Questions Requiring Coreferential Reasoning, September 2019. URL
http:://rxiis http://rxvvv..g/gb//sass// www.ooggg////....o.. 908 05803 arXiv: 1ete.9008.00 008803 [es].
Emilia David. Ai image training dataset found to include child sexual abuse imagery. The Verge, December 2023. URL httpht::/:/// //w.:::::/ww../../ www. theverge com 2023
generative-ae image-immme---eee aaion csam wwwpooooetgalllleieeeeeeoeeeeeee srooggle-stablllty--ttttaaee PST.
7:57 AM
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition workshops, pp. 52-59, 2019,
Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra. Earnings-22: A Practical Benchmark for Accents the Wild, March 2022. httttt::/aarziv.og//aattt 2203. 15591
arXiv:2203.155 15591 [cs].
Ugur Demir, Yogesh S. Rawat, and Mubarak Shah. TInAVIRAT: Low-resolution Video Action Recognition, July 2020. URL http:://axiiv/
07355 arXiv:2007003355
[cs, eess].
Isin Demirsahin, Oddur Kjartansson, Alexander Gutkin, and Clara Rivera. Open-source Multi- speaker Corpora of the English Accents in the British Isles. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 6532-6541, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-1 34-4. URL pttpptttsa//acaatttoooac owg. 1.w.011.e111000
Xiang Deng, Yu Gu, Boyuan Zheng. Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web, December 2023. URL http:
6070 #xXiv:2306.06070 [cs].
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyrr. QLoRA: Efficient Fine- tuning of Quantized ILMss, May 2023. URL. httt:://axx.w.orr//bbb/// 14311000..111.... 14314 arXiv:2305.1433 [es].
Ashwin Devaraj. lain J. Marshall, Byron C. Wallace, and Junyi Jessy Li. Paragraph--evel Sim- plification of Medical Texts, April 2021. URI. htt.:t:://aa.i..oor/a////////// arXiv:2104.05767 [es].
Jay De Young, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. MS2: Multi- Document Summarization of Medical Studies, November 2021. URL abs/2104. 06486 arXiv:2104.066 06486 [cs].
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST-C: Multilingual Speech Translation Corpus. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2012-20122 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10/gtsqzk. URI a antho logy org/N1 9- 1202
44

---

Published as a conference paper at ICLR 2025
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jurgen Gall, Rainer Stiefelhagen, and Luc Van Gool. Large Scale Holistic Video Understanding, December 2020. URL ltttp://axxi erg/abs 1904. 11451 arXiv:1904.1 1451 [cs].
Thomas Diggelmann, Jordan Boyd-Graben, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippoid. CLIMATI--FEVER: A Dataset for Verification of Real-World Climate Claims, January 2021. URL http://arxxii wtg/abs//////eooooooe/.re.aaaaaa abb 2012. 00614 arXiv:2012.0611 [cs].
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. The Second Conversational Intelligence Challenge (ConvA12), January 2019a. http:////aaxxi org/abs/1902. 00098 #rXiv:1902.00099 [cs].
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of Wikipedia: Knowledge-Powernd Powered Conversational agents, February 2019b. URL URLhttp: http://arxiv
orq/ axq/abs/1811. 01241 arXiv:1811.0124 [cs].
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing Chat Language Models by Scaling High-quality Instructional Con- versations, May 2023. URL http:://axxi..rrr/////// abtp://bs////a 1 14233 arXiv:2305.1423 14233 [es].
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale, 2021.
Moussa Doumbouya, Lisa Einstein, and Chris Piech. Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Users, April 2021. URL http:://arxt www.orggbbb.... 2114 13083 A.iv:2100413088 [cs].
Jiay Du, Xingyu Na, Xuechen Liu, and Hui Bu AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale, September 2018. URL http://arxiv.orgg/
10583 arXiv:1808 10583 [es].
Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin. Omni-sourced Webly- supervised Learning for Video Recognition, August 2020. Utt. 2003.13042 mtXiv:2003.13044 [cs].
Omar Einea, Ashraf Elnagar, and Ridhwan Al Debsi. SANAD: Single-label Arabic News Arti- cles Dataset for automatic text categorization. Data in Brief. 25:104076, August 2019. ISSN 23523409. doi: 10/666cnn. URL fittps://lime nighub. el sevier wtw.eeiiie..ooeeeeeeeeeeeeeeeeee
p p11/525234091533025
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mchrotra, and Kaheer Suleman. Frames: a corpus for adding memory to coal-oriented dialogue systems. In Kristinna Jokinen, Manfred Stede, David De Vault, and Annie Louis (eds.), Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 207-219, Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi: 10/gtsqx4. URL https
//aclanthology wttp:hhhooooooo.o/1171.0 wwt/w17-gw11111 5526
Ibrahim Abu El-khair. 1.5 billion words Arabic Corpus, November 2016. URL http:://arxi org/ arg/abs/De aba///a////// /161l. 04033 arXiv:1611 1.04033 [cs].
Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? In The Twelfth International Conference on Learning Representations, 2023.
45

---

Published as a conference paper at ICLR 2025
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision, 2022.
Ashraf Elnagar and Omar Einea. BRAD 1.0: Book reviews in Arabic dataset. In 2016 JEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). pp. 1-8, Agadir, Morocco, November 2016. IEEE. ISBN 978-1-5090-4320-0
doi: 10/g6k6jm. URL http:
//ieeexplore. ieee leee org/ cocument
Hady ElSahar and Samhaa R. El-Beltagy. Building Large Arabic Multi-domain Resources for Sentiment Analysis. In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, pp. 23-34, Cham, 2015. Springer International Publishing. ISBN 978-3-319-188117-2 doi: 10/g6k58r.
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences, December 2020. URL
15738 arXiv:2012.15738 [es].
Dave Epstein, Boyuan Chen, and Carl Vondrick. Oops! Predicting Unintentional Action in Video. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 916- 926, Seattle, WA, USA, June 2020. IEEE. ISBN 978 1-72817-168-55 doi: 10/ghbckh. URL
https: //ieeep/ wre. ore. leee. org/ /t1pe/mmnen9115500
Mihail Eric and Christopher D. Manning. Key-Value Retrieval Networks for Task-Oriented Dialogue, July 2017. Utttttt: org abs abs/1705 05414
000.1055.51144 MrXiv:1705.0544 [cs].
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur. MultiWOZ. 2.1: : A Consolidated Multi- Domain Dialogue Dataset with State Corrections and State Tracking Baselines, December 2019.
URL /RLAEpp////aXX...rrr/////////111 01669 2Xiv:1907.01661 [es].
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content guided video synthesis with diffusion models, 2023. URL https:/7
arxiv.org/abs/ 2302 03011
Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Conversation Summarization Benchmark and Improved Abstractive Sum- marization with Argument Mining. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1thh International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp- 6866-68888 6880, Online, August 2021. Association for Computational Linguistics. doi: 10/gmf9qs. URL htthtt:::// http:://aaaaanttttcace
Radev. ConvoSumm:
http otg.://g..r....22// acl -oonn-nn----- 535
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. Multi-News: a Large- Scale Multi-Documeet Summarization Dataset and Abstractive Hierarchical Model, June 2019. URL httt:://axxiv.oot htsp/bbb/19900 00..00001799 01749 arXiv:1906.01749 [es].
Fahim Faisal, Yinkai Wang, and Antonios Anastasopoulos. Dataset geography: Mapping language data to language users. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp.
2022.
Manaal Faruqui and Dipanjan Das. Identifying Well-formed Natural Language Questions, August abs 1808.. 0111118888119888 arXiv: 1808. 008.09419 [cs].
2018. URL htttttt/////xiv..t.
Guy Feigenblat, Chulaka Gunasekara, Benjamin Sznajder, Sachindra Joshi, David Konopnicki, and Ranit Aharonov. TWEETSUMM November 2021. WtRLhttp:// arwii.iiiiv....a. org hwwpg////////a 2111. 11894 arXiv:21 111. 1. 1894 [cs].
Dialog Summarization Dataset for Customer Service,
Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. MultiDo:2Dial: Modeling Dialogues Grounded in Multiple Documents. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. pp. 6162-6177 2021. doi: 10/g6k938. URL http: http:://axxi: /arxiv brg/abss/100 110.e..159999 12595 arXiv:2109. 12595 [cs].
46

---

Published as a conference paper at ICLR 2025
David F. Fouhey, Wei-cheng Kuo, Alexei A. Efros, and Jitendra Malik. From Lifestyle Vlogs to Everyday Interactions, December 2017. htttpp:///aaxi wwwp:/iwworr//b///////0 02310 arXiv:1712.02310 [es].
Yihui Fu, Luyao Cheng, Shubo Lv, Yukai Jv, Yuxiang Kong, Zhuo Chen, Yanxin Hu, Lei Xie, Jian Wu, Hui Bu, Xin Xu, Jun Du, and Jingdong Chen. AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario, August 2021. URL htt:: htttt:://axxitt http//aaxvvvvv/ org abs 2204 00111144330000 03603 arXiv:2104.03603 [cs, eess].
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt. Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mchdi Cherti, Ranjay Krishna, Pang Wei W Koh, Olga Saukh, Alexander J Ratner, Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beau- mont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 27092-271 Curran Associates, Inc., 2023. oc URL httpps:/pproocedigggs.eeeer/r//p w httppp/ppppccccccaaaa paper files paper/app//////// 2223///22///fff///////// and Benchmarks.
pdf
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Aswanth Kumar M. Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M. Khapra, Raj Dabre, Rudra Murthy, and Anoop Kunchukuttan. Airavata: Introducing Hindi Instruction- tuned LLM, February 2024. URL http::/arxiv.org./ abs 2401 15006 arXiv:2401. 15006 [cs].
Ygor Gallina, Florian Boudin, and Beatrice Daille. KPTimes: Large- Scale Dataset for Keyphrase Generation on News Documents, In Kees van Deemter, Chenghua Lin, and Hiroya Takamura (eds.), Proceedings of the 12th International Conference on Natural Language Generation, pp. 130-135, Tokyo. Japan, October 2019. Association for Computational Linguistics. doi: 10/g6664k. URL Cettp:://alaathhllese
org/ www.//////////0/1 8617
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage, November 2021.
URL battp://arxivy org/ aatsass//1ll.. 09344 MxXv:2211.09944 [cs, stat].
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. Opinosis: A Graph Based Approach Ab- stractive Summarization of Highly Redundant Opinions. In Chu-Ren Huang and Dan Jurafsky (eds.), Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pp. 340-348, Beijing, China, August 2010. Coling 2010 Organizing Committee. URL attps::/accLantiiooor..ooo// 1039
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.001 2020.
Garofolo, John S., Lamel, Lori F., Fisher, William M., Pallett, David S., Dahlgren, Nancy L., Zue, Continuous Speech Corpus, 1993. Victor, and Fiscus, Jonathan G. TIMIT edu/LDCC331 Artwork Size: 715776 KB Pages: URL https://cata lgg. ldc. soc..upenn.. 715776 KB.
Murilo Gleyson Gazzola, Sidney Evaldo Leal, and Sandra Maria Aluísio. Predição da complex- idade textual de recursos educacionais abertos português. Symposium in Information and Human Language Technology- STIL, 2019. URL WRRL/htts:://// /ttttpss//eppsssoooiii.oooooo usp bz /item/
002 00297140 2
47

---

Published as a conference paper at ICLR 2025
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12)) (1): 86-92, 2021.
Elizabeth Jasmi George and Radhika Mamidi. Conversational implicatures in English dia- logue: Annotated dataset, November 2019. RRLL:ttp.//aaxiv..rrr///////1111 arXiv:1911. [cs].
Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michat Lis, Eshan Singhal, Nikola Mrkšić, Tsung-Hsien Wen, and Ivan Vulié. Multilingual and Cross-Lingual Intent Detection from Spoken Data, April 2021. http:/tt:://aaa wttp://.rrrgaa////..///// 08524 200o5522 arXiv:2104.08524 [es].
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. DiscoFuse: A Lcrge-Scale Dataset for Discourse- Rissourse-Based Sentence Fusion, March 2019. URL [ttp:://aaax htt.:/iwi..rgg.aa/aoaaaaaaaoaas
10526 arXiv:1902. 10526 [cs].
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum Corpus: A Human- annotated Dialogue Dataset for Abstractive Summarization. In Lu Wang, Jackie Chi Kit Cheung. Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10/gmjggr. BtLhhtts::// https///ccaaathology... wtwp://ooo..../11--5-00--000
J.J. Godfrey, E.C. Holliman, and J. McDaniel. SWITCHBOARD: telephone speech corpus for research and development. In [Proceedings) ICASSP-922 1992 IEEE International Conference
on Acoustics, Speech, and Signal Processing, pp. 517-520 vol.1, San Francisco, CA, USA, 1992.
IEEE. ISBN 97880-7700-- doi: 10/fp48kw. URL http:::/eeeexplooreee... iee.....e.e..................... org/
2occument2225855
Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. RumourEval 2019: Determining Rumour Veracity and Support for Rumours, September 2018. httptp:://axxi....o// org/abs /acaaa/ssa18100 11899.6899.68133
[cs].
Tear Gosling, Alpin Dale, and Yinhe Zheng. PIPPA: A Partially Synthetic Conversational Dataset, August 2023. URL /tlL:ttpp//axx..ooot
www.::iww...o.aaaaaaaaaaoaeaaaae org. abs 2208. C5884 arXiv:2308.0588 [cs].
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyúska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Freettg,, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense, June 2017. URL http::ttt///c www.aaaiivvv....... org/ abs/ arXiv:1706. 042611 [cs].
1706. 04261
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier. Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang. Wenqi Jia. Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4D: Around the World in 3,000 Hours of Egocentric Video, 07058. arXiv:21 10.07058 [cs].
March 2022. URL URLpttpp: //tp///riiiii http///axxxv....aaa//a/
48

---

Published as a conference paper at ICLR 2025
Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susamna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: A Video Dataset of S temporally Localized Atomic Visual Actions, April 2018. URL http:://axii..rgg//////
1705. 08421 arXiv: 1rXiv:1705.0844 100.08821 [cs].
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases. In Proceedings of the Web Conference 2021, pp. 3477-3488, April 2021. doi: 10/gnnfbt. URL ht5//bs/221111 07743
arXiv:201 1.00743 [es].
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Transactions on Computing for Healthcare, 3(1):1-23, January 2022. ISSN 2691-1957, 2637-8051. doi: 10/gmkjjx. URL http::ttp::.:ttt. wtt.aatwwii..aaaaaaaoooooa abs 2007 15779
ar arXiv:2007.15779 [cs].
Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, and Xiangang Li. DiDiSppeeh: Large Scale Mandarin Speech Corpus, February 2021. URL htttp:://axxi..t..t.tt org/ abs//b001 09275 arXiv:2010.09275 [eess].
Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui. Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering, June 2021. URL 04016 1XXi:22166.04016 [cs].
http: wwtp:///gggbbbbbbbob/oooes
Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. MMQA: A Multi-domain Multi-lingual Question-Answeriin Framework for English and Hindi In Nicoletta Calzolari, Khalid Choukri. Christopher Cieri, Thierry Declerck. Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018a. European Language Resources Association (ELRA). URL URL MRLLLttts://aaaantooooo wwt.:/thww..oyy../111111111 1440,
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. Semantic Parsing for Task Oriented Dialog using Hierarchical Representations, October 2018b. URL http: //arxiv
07942 arXiv:1810.07942 [cs].
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating Summaries from User Videos. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 20/4, volume 8695, pp. 520- Springer International Publishing, Cham, 2014. ISBN 978-3- 978-3-319-10583-3 10111 link. tirkk.springer in Computer Science.
URL/ntt:: // 10584 33 Series Title: Lecture Notes
David Ha and Douglas Eck. A Neural Representation of Sketch Drawings, May 2017. URL http:// org/ abs/1704 03477 arXiv:1704.03477 [es, stat].
Hessel Haagsma, Johan Bos, and Malvina Nissim. MAGPIE: A Large Corpus of Potentially Id- iomatic Expressions. Nicoletta Calzolari, Frédérie Béchet, Philippe Blache, Khalid Choukri, Christopher Cicri, Thierry Declerck. Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceed- ings of the Twelfth Language Resources and Evaluation Conference, pp. 279-287, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34 URL https: httpss//aclanthhlooo
org/ 2020. 1rec-
Amirhossein Habibian, Thomas Mensink, and Cees G.M. Snoek. VideoStory: A New Multimedia Embedding for Example Recognition and Translation of Events. In Proceedings of the 22nd ACM international conference Multimedia, pp. 17-26, Orlando Florida USA, November 2014. ACM. ISBN 978- 1503333003333 doi: 10/ggs25n. URL https: //11. acm. org/ doi/10
0ww.0/....655999 11 1145/2647868 2654 913
49

---

Published as a conference paper at ICLR 2025
Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. MedAlpaca An Open-Source Col- lection of Medical Conversational AI Models and Training Data, October 2023. URL |
//arxiv.org http://bb//33440 08247 arXiv:2304.88247 [cs].
Moshe Hazoom, Vibhor Malik, and Ben Bogin. Text-to-SOL in the Wild: A Naturalyy-Occurring Dataset Based on Stack Exchange Data, June 2021. URL
05006 1XXi:2206.05006 [cs].
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang, Decoupling Strategy and Generation in Negotiation Dialogues, August 2018. URL arXiv: 1tt::1888.000 808.09637 [es].
09637
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. NetivityNet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961-977 Boston, MA, USA, June 2015. IEEE. ISBN 978- 4-633-696----- doi: 10/gfsvdw. URLhttp: http::/heeeepploree wwweexxploreieeee... ieee. org
document/1298632
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS Spoken Language Systems Pilot Corpus. In Speech and Natural Language: Proceedings ofa a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990, 1990. doi: 10/cz3442. URUhhtpps://aelatthhoooe
org ATr//9900 1901
Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. Pile of Law: Learning Responsible Data Filtering from the Law and 256GB 2207 00220
Open-Source Legal Dataset, November 2022. URL StRl:htt:://// htttt:://axii..rg////oo MtXiv:2207.002200 [cs].
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark Lemley, and Percy Liang. Foundation models and fair use. arXiv preprint arXiv:2303.15715 2023.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell, Lo- calizing Moments in Video with Temporal Language. In Ellen Riloff, David Chiang. Julia Hocken- maier, and ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Nat- ural Language Processing, pp. 1380-1390, Brussels, Belgium, October 2018. Association for Com- putational Linguistics. doi: 10/gtsqzq. URL htt.:pp:///aaaattoooooooo///////
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD; An ExperttAnanttt NLP Dataset for Legal Contract Review, November 2021. URL http:://aaravvv.cxxaaa//// 06268 ar 1rXiv:2103.06268 [cs].
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020. URL htttps:/arxiii http://rrgaabb////oooo// 2000///000000017701
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adapta- tion. In Lecture Notes Computer Science, volume 11096. pp. 198-208. Springer. Cham, htt.:tt:::///wwwoo/////////
2018. doi:
04699
1rXiv:1805.044: 04699 [es).
Carlos Daniel Hernandez Mena, David Erik Mollberg, Michal Borsky, and Jon Gudnason. Samrómur Children: An Icelandic Speech Corpus. In Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings the Thirteenth Language Resources and Evaluation Conference, pp. 995 1002, Marseille, France, June 2022. European Language Resources Association. URL hitps: htttps://aclanthert http//aaaaaaaaaaaaaaaaaaaaoooola wrw.occ....rg/2220
the
Trec-1 10105
50

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXtv:2203. 2022.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual General- ization, September 2020. URL http:tttpppp........aaaaal [es].
11080 arXiv:2003.11080
Albert Huang. "Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities, February 2022. URLhtt:://arxii wwwpavv..rggg..eaoeeeeeeo wttp///a/g///////e [es].
arXiv:2202.15581
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning, September 2019. URL http://
0177 arXiv:1909.00277 [es].
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient Attentions for Long Document Summarization, April 2021.
arXiv:2104.02 02..000221122 12 [es].
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. MovieNet: A Holistic Dataset for Movie Understanding, July 2020. UKL/http.//arrxx..0///////000::0 arXiv:2007.10937 [es].
Moustafa Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A Hi- erarchical Deep Temporal Model for Group Activity Recognition, April 2016. URL http 1511: [cs].
//axxiv.org/aat abs/ / 1511 11511.00101001
Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS Challenge on Action Recognition for Videos "in the Wild". Computer Vision and Image Understanding, 155:1 -23, February 2017. ISSN 10773142. doi: 604 06182 arXiv:1604.0618% [cs].
10/19rwnr. URL
John J. Irwin, Khanh G. Tang, Jennifer Young, Chinzorig Dandarchuluun, Benjamin R. Wong, Munkhzul Khurelbaatar, Yurii S. Moroz, John Mayfield, and Roger A. Sayle. ZINC20- Free Ultralarge-Seat Chemical Database for Ligand Discovery. Journal of Chemical Information and Modeling, 60 (0:0166666667773 December 2020. ISSN 1549-9596. doi: 10/gmjg8b. htttst
//doi wr.. 10. 0 1 021//cc jcim. OcC0675 Publisher: American Chemical Society.
Keith Ito and Linda Johnson. The LJ Speech Dataset, 2017. URL httlatts///iiititt..comm
LJ - Si--Speehh-Daaasse
Maor Tvgi, Uri Shaham, and Jonathan Berant. Efficient Lext Underssniing with Short- Text Models, December 2022. httc:p////artv..rrr//////22000.0 arXiv:2208.0077 [cs].
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a Neural Semantic Parser from User Feedback, April 2017. URL http:: //arxiv arg/abs/1700 08760 arXiv:1704 704.0800 [es].
Mohit Iyyer, Wen-tau Yih, and Ming-Was Chang. Search-based Neural Structured Learning for Sequential Question Answering. In Regina Barzilay and Min-Yer Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1831-1831, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10/gf6nx8. URL pestps://aclantthee wow.ooy....o........... wtt.hhooooooyooooo11/11111
Tahir Javed, Kaushal Santosh Bhogale, Abhigyan Raman, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. IndicSUPERB: A Speech Processing Universal Performance Bench- mark for Indian languages, December 2022. TRLhhtp::/// http://wxv...o./a/////////////// arXiv:2208. 1 1761 [cs, eess).
51

---

Published as a conference paper at ICLR 2025
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities, July 2020. URL httl:t://aa//x..r abs/ abs/2007.o 2007 15781 arXiv:2007.15781 [es].
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. Neural CRF Model for Sentence Alignment Text Simplification, August 2021. URL 2005 02324 2Xiv:20055022244 [cs].
/bs/
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Mancesh Singh, and Mohit Bansal. Ho A Dataset for Mop Fact Extraction And Claim Verification, November 2020. URL http://arxiv org/ abs 2011 03088 TrXiv:2011.03085 [es].
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams, September 2020. URL http://axxiv.rrg/ arXiv:2009. [cs].
13081
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA:
Dataset for Biomedical Research Question Answering, September 2019.
org/abs abs/ 06146 arXiv:1909.06 1466 [cs, q-bio].
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, May 2017. URL http
//a //arxiv..org www..ii.rrr/abb//70055 03551 arXiv:1705.0355 [cs].
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095, 2020.
Juraj Juraska, Kevin K. Bowden, and Marilyn Walker. ViGGO: A Video Game Corpus for Data-To To-
Text Generation in Open-Domain Conversation, October 2019. URL http://arxiv..rrg/
abs/1910, 12129 arXiv:1910. 12129 [cs].
Tomasz Jurczyk, Michael Zhai, and Jinho D. Choi. SeIQA: A New Benchmark for Selection- based Question Answering, October 2016. URL arXiv:1606.08513 [cs].
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and Evaluating Contextual Embedding of Source Code, August 2020. URL (RL::tt:///aat www.iiiorg//...oooooooooo.ooooot abs 2201 00059 arXiv:2001.0000 [cs].
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0833 2020.
Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in ml-based science. arXiv preprini 22t::22077000048 2022.
Andrej Karpathy, George Toderici. Sanketh Shetty, Thomas Leung. Rahul Sukthankar. and Li Fei-Fei. Large-Scare Video Classification with Convolutional Neural Networks. In 2014 IEEE Conference Computer Vision and Pattern Recognition, pp. 1725-1733 1732, Columbus, OH, USA, June 2014. doi: 10/gf4hdn. URI. pttttp:://ieeepplrrst leee. org/
IEEE ISBN 978- 479--5118-5
document/ /900me11699999999
Nikolay Karpov, Alexander Denisenko, and Fedor Minkin. Golos: Russian Dataset for Speech Research, June 2021. URL htlt:::/aaxxv..org//bb//// [eess].
arXiv:2106.100 10161
Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the Difference that Makes a Difference with caaerffcttally-Auugmeeee Data, February 2020. URL http:// http:://axxii org/abs/ 9009. 12434 1909 12434 [cs, stat].
52

---

Published as a conference paper at ICLR 2025
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The Kinetics Human Action Video Dataset, May 2017. URL http://arxiv..rr
1705. 06950 1XXiv:1005.0650 [cs].
Yerbolat Khassanov, Saida Mussakkooayyea, Almas Mirzakhmetov, Alen Adiyev, Mukhamet Nurpei- issov, and Huseyin Atakan Varol. A Crowdsourced Open-Source Kazakh Speech Corpus and Initial Speech Recognition Baseline, January 2021. URL
20092 10334
EXXiv:2009.10333 [cs, eess].
Tushar Khot, Ashish Sabharwal, and Peter Clark. SciTaiL: A Textaal Entailment Dataset from Science Question Answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32 (1), April 2018. ISSN 2374-3468, 2159-5399,
doi: 10/grm22d. URL httts::/ojsssaaa aaai
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A Dataset for Question Answering via Sentence Composition, February 2020. URL arXiv: 1910. 1473 [cs].
abs/ /2s0/.91.... 11473
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. ProsocialDialog: Prosocial Backbone for Conversational Agents, October 2022. URL httptttt: wwtp:/iwwwwggaaaaaoaoaaaaaaooaaa /0005/2220..0 200p:/2222.1188 arXiv:2205.1266 [cs].
Seonhoon Kim, Inho Kang, and Nojun Kwak. Semantic Sentence Matching with Densely Recurrent and Co-attentive Information, November 2018. URL
1805. 11360 arXiv: 1360 [cs].
Seungone Kim, Jamin Shin, Yejin Cho. Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing Fine- grained Evaluation Capability in Language Models, March 2024. URL htttp:://arxvv.oz// abs/ 221502310.. 08491 11.00:2310008491 [cs].
Andreas Kirkedal, Marija Stepanović, and Barbara Plank. FT Speech: Danish Parliament Speech Corpus, October 2020. URL http: http::://aaxiv..e
12368 arXiv:2005.12368
[cs, eess].
Oddur Kjartansson, Supheakmungkol Sarin, Knot Pipatsrisawat, Martin Jansche, and Linne Ha. Crowd-Sourced Speech Corpora for Javanese, Sundanese, Sinhala, Nepali, and Bangladeshi Ben- gali. In 6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018), pp. 52-55. ISCA, August 2018. doi: 10/gtwwbs. URL hal.htts:: /wwww. scaa archi ive org situ (011///jartaaa.
altu.html
Oddur Kjartansson, Alexander Gutkin, Alena Butryna, Isin Demirsahin, and Clara Rivera. Open- Source High Quality Speech Datasets for Basque, Catalan and Galician. In Dorothee Beermann, Laurent Besacier, Sakriani Sakti, and Claudia Soria (eds.), Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourcee Languages (CCURL... pp. 21-27, Marseille, France, May URL https
2020. European Language Resources association. ISBN 979- 979-10-95546-35-1.
7/aclanthology org s.t..l.tu....
Kevin Klyman. Acceptable use policies for foundation models, 2024. URL pictps://arxtn brg/abs/2409 09041
Rostislav Kolobov, Olga Okhapkina, Olga Omelchishina, Andrey Platunov, Roman Bedyakin, Vy- acheslav Moshkin, Dmitry Menshikov, and Nikolay Mikhaylovskiy. MediaSpeech: Multilanguage ASR Benchmark and Dataset, March 2021. URL /ttl://tc.//aarrv.orr////////// arXiv:2103.16193 [es, eess].
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-Augmentedd Dialogue Generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 2460-8478, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10/gr75db. URL
https: ///s:///clll antho ogy. ogy.orrg/022..oo -log. 579
53

---

Published as a conference paper at ICLR 2025
Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. MMAct: A Large-Scale Dataset for Cross Modal Human Action Understanding. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 8657-86666 October 2019. doi: 10/ ghfhxx. URL /ttpss//eeeeeeereeee leee org/ http://occmee///////5///0 ISSN: 2380- 7504.
Anastassia Kornilova and Vlad Eidelman. BillSum: A Corpus for Automatic Summarization of US Legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 48-56, 2019. doi: 10/gtwwdd URL http: http:://wwwxvv..g/a/aaaa//aa [cs].
1rXiv:1910.00523
Matëj Korvas, Ondřej Plátek, Ondřej Dušek, Lukáš Žilka, and Filip Jurčiček. Free English and Czech telephone speech corpus shared under the CC-BY-SA -SA 3.0 license. In Nicoletta Calzolari, Khalid
Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno,
Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL CRLLhttp:/// www. 1rec-conf..oo org/ proceedings httpe/iiiiinseeeee//////a/////// Papp_aaappppp___e pdf
Language Resources and Evaluation (LREC" 14), pp.
Danijel Korzinek, Krzysztof Marasek, Lukasz Brocki, and Krzysztof Wolk. Polish Read Speech Corpus for Speech Tools and Services, June 2017. URL 00245 MrXiv:1706.0022 [cs].
/106
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. arXiv preprint arXiv:2309.10105 2023.
Jonáš Kratochvil, Peter Polák, and Ondiej Bojar. Large Corpus of Czech Parliament Plenary Hearings. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 6363-6367, Marseille, France, May 2020. European Language Resources Association. ISBN 979- 10-95546--44444 URL https: /atc//accccc httpa/aaaannttooyy.oooooooo
2020.1rec-1.77 181
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orhikkh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, al. Quality a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50-72, 2022.
H. Kuehne, H. Jhuang, Garrote, T. Poggio, and Serre. HMDB: large video database for human motion recognition. In 201 International Conference on Computer Vision, pp. 2556-2563, Barcelona, Spain, November 2011. IEBE. ISBN 978-14457-1110-2. htttp://ieexppLee ieee ieee. org www.deot /12mmme4e66126553...
doi: 10/fxpf8k. URL
Hilde Kuehne, Ali Arslan, and Thomas Serre. The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 780-787, Columbus, OH, USA, June 2014 4. IEEF. ISBN 978-1-4799-55 18 eee. org/ /occmmmttt/t9905000 eeexp o re.
5. doi: 10/gqdc3v. URL https: /ttpess///eeeeppple
Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Avik Bhattacharyya, Mitesh M Khapra, Pratyush Kumar, et al. Ai4bharat-indicnle corpus: Monolingual corpora and word embeddings for indic languages. arXiv preprint 000002005000085 2020.
Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing large-language models to generate private synthetic text. June 2023.
Fabrício Kury, Alex Butler, Chi Yuan, Li-heng Fu, Yingcheng Sun, Hao Liu, Ida Sim, Simona Carini, and Chunhua Weng. Chia, large annotated corpus of clinical trial eligibility criteria. Scientific Data, 7(1)2811, August 2020. ISSN 2052-4463, doi: 10/gr4ftn. / /ttp::///wwm..ttee Comm/articcleess4415999-0-0
Publisher: Nature Publishing Group.
54

---

Published as a conference paper at ICLR 2025
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnv Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant Conversations - Democratizing Large language Model Alignment, October 2023. URL UR.nnttp://arriiv.org////////// 2000 [cs].
arXiv:2304.07327
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding Comprehension Dataset From Examinations, December 2017. URL http://arxiv.org
abs/1704. arXiv: 1tt::1000 700404683 [cs].
Brenden M. Lake and Mareo Baroni. Generalization without systematicity: On the compositional recurrent networks, June 2018. URL http::/aax.v..rrttt abs/1711. 00350 arXiv:1711.00350 [cs].
skills of
Lander, T. CSL.U: 22 Languages Corpus, November 2005. URL https://catall:: 1dc
Lander, T. CSLU: Foreign Accented English Release 1.2, May 2007. URL BtLptttss://caaa.oo
Idc. www.upennnn... edw www.LDc2072L00 Artwork Size: 1468006 KB Pages: 1468006 KB.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An Evaluation Dataset for Intent Classification and Out-of-Scopp Prediction. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1311-1316, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10/grfvv.. URL
oxg/D19-113)
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafcai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dizs -Doos Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The bigscience roots corpus: 1.6tb composite multilingual dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 31809-31826, Curran Associates, Inc., 2022. URL /ttp://ttss//rrrceddnn.....eeeee wwtprrppppcpppppp...ee
cc/ 022p://0002112222222222222223332 http://ses__uuurrrccrceoepeeaeae
and Benchmarks pdf
Remi Lebret, David Grangier, and Michael Auli. Neural Text Generation from Structured Data with Application to the Biography Domain, September 2016. URL http::::/aa//i.oo////aaa// [cs].
1603. 07771 arXiv:1603.077
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of 1lms. NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a.
Katherine Lee, A Feder Cooper, and James Grimmelmann. Talkin"bout ai generation: Copyright and the generative-ai supply chain. arXiv preprint arXiv:2309.0813: 2023b.
Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, and Daniel Whitenack. Bloom Library: Multimodal Datasets in 300+ Languages for Variety of Downstream Tasks, October 2022. URL http:://t////c //axxi///rriv.... org/ abs abs /2100 14712 1Xiv:22100147122 [cs].
55

---

Published as a conference paper at ICLR 2025
Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. Deal No Deal? End-to-Ena Learning for Negotiation Dialogues, June 2017. URL htLhhtt://aaxiiii org. /bs 7706.05124 urXiv:1706.000 0215 [cs].
Guanyu Li, Hongzhi Yu, Thomas Fang Zheng, Jinghao Yan, and Shipeng Xu. Free linguistic and speech resources for Tibetan. In 2017 Asia-Pucific Signal and Informution Processing Association Annual Summil and Conference (APSIPA ASC), pp. 733-736, Kuala Lumpur, December 2017. IEEE. ISBN 978- 1-5386-154444 1542-3. doi: 10/gtsqzh. URL http:://ieee/ppppeeee// wew.eeeeeeooorreee...oooo. document /28
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-X: Muliiiiguaa Replicable InsllwitionnFoolloinng Models with Low-Rank Adaptation, October 2023a. URL
wttp://wwww////b///./oo// 1 15011 arXiv:2305. 15001 [es].
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark. In Paola Merlo, Jorg Tiedemann, and Reut Tarraty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2950-2962, Online, April 2021a. Association for Computational Linguistics. doi: 10/gtsqxq.
URL https://
aclanthology, org. http:/2222eeeee1111111 maacl-main.250
Hongyu Li, Seohyun Kim, and Satish Chandra. Neural Code Search Eyaluation Dataset, October 2019a. http:/ttpp//aa..v.ovoooo
09804. arXiv: 1908.09804 [cs].
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark, May 2024. URL http:::p/pp.....o////eeaao [cs].
1ete1111770000 17005 arXiv:2311.17005
Sheng Li, Zhiqiang Tao, Kang Li, and Yun Fu. Visual to text: Survey of image and video captioning.
IEFE Transactions on Emerging Topics in Computational Intelligence, 3(4):297-312, 2019b. doi: 10.1
Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, and Zhiheng Li. UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles, August 2021b. URL
00946 arXiv:2104.00946 [es].
Xinjian Li, Shinnosuke Takamichi, Takaaki Sacki, William Chen, Sayaka Shiota, and Shinji Watanabe. Yodas: Youtube-oriented dataset for audio and speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023b.
Xinjian Li, Shinnosuke Takamichi, Takaaki Saeki, William Chen, Sayaka Shiota, and Shinji Watan- abe. Yodas: Coutube-Oriented Dataset for Audio and Speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8, Taipei, Taiwan, December 2023c. iew.eeeeopoooeeeee.oooo.o.o
IEEE. ISBN 7-779-9-300---
doi: 10/gtsqze. URL
4ocument/10389955
Yu Li, Kun Qian, Weiyan Shi, and Zhou Yu. End-to-End Trainable Col-Collaborative Dialog System, November 2019c.
10742 1rX1v:1911.10742 [cs].
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' A'tumme Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukeuoglu, and Oriol Vinyals. Competition- Code Generation with AlphaCode. Science, 378(6624) December 2022. ISSN 0036-8075, 1095-9203. doi: 10/grggxf. URL http:: /ttp://rxxivi org.oog///ao.... 2203/ 22011200131244 07814 arXiv:2203.07814 [cs].
Yuncheng Li. Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. TGIF: A New Dataset and Benchmark on Animated GIF Description, April 2016. URL
http::/arxi: http://axxvv..r//bbbbb/o
wrw. hwwpg///////// 6044 02748 1rXiv:1604.02748 [cs].
56

---

Published as a conference paper at ICLR 2025
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-Al (LLaMA) Using Medical Domain Knowledge, June 2023d. URL TRLLhtpp:// http///axvvvv/vaa//aaa wwt. org/ abs/////////00 14070 arXIv:2303.14070 [es].
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step, May 2023. URL 20050 arXiv:2305.200 20050 [es].
http::/aaxiv..org//22..
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models, September 2020a.
URL ottpap/pp//aaa.....ooo/////////0
wttpo//g...///oooee 00622335.0000333 00683 arXiv:2005.00683 [es].
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: Constrained Text Generation Challenge for Generative Com- monsense Reasoning, November 2020b. URL http://arxiar http://twwi..orr/a///11111.1.11/
abs
1:X1v:1911.03705 [cs].
Bill Yuchen Lin, Ziyi Wu, Yichi Yang. Dong-Ho Lee, and Xiang Ren. RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge, July 2021a. URL
org. abs 210021111111 00376 arXiv:2101.00376 [es].
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner, Reasoning Over Paragraph Effects in Sit- 908 05852 WiXiv11908.0585.
uations, December 2019. URL http:////////:tiv..t [cs].
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human 07 758 arXiv:2109.07958
Falsehoods, May 2022. URL [cs].
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale Fung. BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling, June 2021b. URL http: http:///www.....////////....... 02787 arXiv:2106.02787 [cs].
Wang Ling. Dani Yogatama, Chris Dyer, and Phil Blunsom. Program Induction by Rationale Generation Learning to Solve and Explain Algebraic Word Problems, October 2017. URL htttp://arxiv..rg/aaet
04146 arXiv: 041 46 [es].
Chunhui Liu, Yueyu Hu, Yanghao Li, Sijie Song, and Jiaying Liu. PKU-MMD: A Large Scale Benchmark for Continuous Mudai-Modal Human Action Understanding, March 2017. URL [cs].
http::/arxti //tp:///arvvv.... org/ abs / 703. 07475
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved basclines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. Asgard: A portable architecture for multilingual dialogue systems. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8386-8390, Vancouver, BC, Canada, May 2013. IEEE. ISBN 978-1-4799- 0356-6. doi: 10/gtsqxn. URL /RLltttt//eeeeeporrr .erooreeeeeeeeee..e. org /
www.eeee...//occmmnnnteeeeo dwwpmmnt/cm///oo//// 0eeeett/6333300 6639301/
Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. VIOLIN: A Large-Scale Dataset for Video-and-Language Inference, March 2020. http
//arxiv. httpxii..rrg//b/////00
11618 arXiv:2003. [cs].
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M Dai. Best practices and lessons learned on synthetic data. April 2024a.
57

---

Published as a conference paper at ICLR 2025
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What Makes Good Data for Alignment? A Comprchensive Study of Automatic Data Selection in Instruction Tuning, April 2024b. URL httpttt/aaxii...oo org. 2ttp/311..1.. 15685 MrXiv:2312.15668 [cs].
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents, October 2023c. URL arXiv:2308.c S1V:22308.03688 [cs].
arg/
2308. 03688
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. Benchmarking Natural Language Understanding Services for building Conversational Agents, March 2019. httt://///axx
org/abs/1903 05566 arXiv:1903.0555 003.05566 [cs].
Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18004 18041, 2024c.
Zeming Liu, Haifeng Wang, Zheng- Yu Niu, Hua Wu, and Wanxiang Che. DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4335-4347, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10/gtsqxr. URL httpss:/acclathoo/ httphhhoooooo....222/o2// emnlp-main. 356
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's 's guide to training data: Measuring the effects of data age. domain coverage, quality, & toxicity, 2023.
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, et al. The responsible foundation model development cheatsheet: A review of tools & resources. arXiv preprint arXiv:2406. 16746, 2024a.
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marma Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi (Alexis) Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, and Sara Hooker. A large-scale audit of dataset licensing and attribution in Al. Nature Machine Intelligence, 6(8): 975- 987, August 2024b. doi: 10/gt8fSp.
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. Consent in crisis: The rapid decline of the ai data commons. arXiv preprint arXiv:2407. 14933, 2024c.
Shayne Longpre, Robert Mahari, Naana Obeng- William Brannon, Tobin South, Katy Gero, Sandy Pentland, and Jad Kabbara. Data authenticity, consent, & provenance for ai are all broken: what will it take to fix them? arXiv preprint (XXi:22444.12691, 2024d.
Annie Louis, Dan Roth, and Filip Radlinski. "T' rather just go to bed": Understanding Indirect Answers, October 2020. URL htRphttp::/// httt::t////txi/// org abs /t..b//2000 201 C. 03450 arXiv:2010.03450
[cs].
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno Kampman, et al. Seacrowd: A multilingual multimodal data hub and benchmark suite for southeast asian languages. arXiv preprint arXiv:2406. /0/ 18, 2024.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems, February 2016. URL http:://axiiv org/ abs /1505 08909 T.X.vv1506.089000 [cs].
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, October 2022. URL hhtp: http://axxiv... org abs htwpabb....22.22.2 09513 McXiv:2209.09513 [es].
58

---

Published as a conference paper at ICLR 2025
Alexandra Sasha Luccioni and Joseph D Viviano. What's in the box? preliminary analysis of undesirable content in the common crawl corpus. May 2021.
Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, Kiran Somasundaram, Luis Pesqueira, Mark Schwesinger, Omkar Parkhi, Qiao Gu, Renzo De Nardi, Shangyi Cheng, Steve Saarinen, Vijay Baiyya, Yuyang Zou, Richard Newcombe, Jakob Julian Engel, Xiaqing Pan, and Carl Ren. Aria Everyday Activities Dataset, February 2024, URL http: wttpvv.iiivv.//... org/ abs 2402. 1432.13329. arXiv:2402. 13349 [es].
13449
Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems, October 2023. URL /ectp://axxic org/bbs
2305. 4536 arXiv:2305.14 14536 [cs].
Kikuo Mackawa. Corpus of spontaneous Japanese: its design and evaluation. In Proceedings of the ISCA/IEEE Workshop on Spontaneous Speech Processing and Recognition, pp. paper MMO2, 2003.
URL https:://// hwtp://wwww./. www.ecaarcciiee.e org/ sapr 2003 2003/maekawa03 sspr html
Rohan Mahadev and Anindya Chakravarti. Understanding gender and racial disparities in image recognition models. arXiv preprint arXi::2107.09211, 2021. 2021.
Robert Mahari and Shayne Longpre. Discit ergo est: Training data provenance and fair use. Robert Mahari and Shayne Longpre, Discit ergo est: Training Data Provenance And Fair Use, Dynamics of Generative Al (ed. Thibault Schrepel Volker Stocker), Network Law Review, Winter, 2023.
Robert Mahari, Longpre Shayne, Lisette Donewald, Alan Polozov, Alex 'Sandy' Pentland, and Ari Lipsitz. Comment to US copyright office on data provenance and copyright, 2023.
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- pertQA: Curated Questions and Attributed Answers, April 2024. URL httt:://ama org/ abs 2309. 07852 arXiv:2309.0785 [cs].
Mishaim Malik, Muhammad Kamran Malik, Khawar Mehmood, and Imran Makhdoom. Automatic speech recogmition: a survey. Multimedia Tools and Applications, 10:9411-9457, 2021.
Srikanth Malla, Behzad Dariush, and Chiho Choi. TITAN: Future Forecast Using Action Priors. In
2020 IEEF/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 11183- 11193, Seattle, WA, USA, June 2020. IEFF. ISBN 978- 17771--72177-16811
doi: 10/gg99rg. URL
ieee org/ Soocmeett/915550
Pekka Malo, Ankur Sinha, Pyry Takala, Pekka Korhonen, and Jyrki Wallenius. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts, July 2013. URL http://arxiv
org/abs/1307. 5336 arXiv: [cs, q-fin].
Ikram Mamtimin, Wenqiang Du, and Askar Hamdulla.
A Free Kirghiz Speech Database and Accompanied Baselines. Information, 14(1):55, January 2023. ISSN 2078-2489. doi: 10/gtsqzm. URI. https:t///// www. www. mdpi com /0//2/00782299////5555
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale, 2023. URL https://
arxiv. ww.. abs 2309. 04564
Marcin Marszalek, Ivan Laptev, and Cordelia Schmid. Actions in context. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 978- 14444-3992- doi: 10/d5bs7p. URL httllattss///eexplpeet
Miami, FL, June 2009. IEEE, ISBN ieee. . axg/document/
52065577
Lara Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O. Riedl. Event Representations for Automated Story Generation with Deep Neural Nets. Proceedings of the AAAI Conference Artificial Intelligence, 32(1), April 2018. ISSN 2374-3468, 2159-5399. doi: 10/g6k72p. URL http: /ttp:::///xii....//. wwwp/b//aaaaaeaee.eeeeee 01331 176.01700 [cs].
59

---

Published as a conference paper at ICLR 2025
Scott Martin, Shivani Poddar, and Kartikeya Upasani. MuDoCo: Corpus for Multidomain Coreference Resolution and Referring Expression Generation. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck. Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings the Twelfth Language Resources and Evaluation Conference, pp. 104-111, Marseille, France, May 2020. European Language Resources Association. ISBN 10-95546- 34-4. L wwtp:///ccaaaatoooooooooooo org 2 2020. 1ew.0000...eeeeeee 1rec-1.13
Joanna Matcrzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The Jester Dataset: Large-Scale Video Dataset of Human Gestures. In 2019 JEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 2874-2882, Seoul, Korea (South), October 2019.
IEEE. ISBN 978 1-72815--2-2---9 doi: 10/gh5k47. /tll/tttps://eeellorreeeeee
document/9022235
Cecily Mauran. What was Sora trained on? Creatives demand answers. com/
https://nashable sora ai ai-video we---iid-ggeeeeeee gwweratteerrrrr trainn data 2024. [Ac- cessed 28-09-2024).
Angelina McMillan- Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilić, et al. Documenting geo- graphically and contextually diverse data sources: The bigscience catalogue of language data and resources. arXiv preprint arXiv:2201. 10066, 2022a.
Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji. Suzana Ilić, Nurulaqilla Khamis, Colin Leong, Maraim Masoud, Aitor Soroa, Pedro Ortiz Suarez, Zeerak Talat, Daniel van Strien, and Yacine Jermite. Documenting geographically and contextually diverse data sources: The bigscience catalogue of language data and resources, 2022b. URL https: httpp:://axxxi.coom /ttp:i::///axvv.o//o/o wwtp:///.rgga/a//otaaoaa...a.s 2201 10066
James Meese and Jennifer Hagedorn. Mundane content on social media: Creation, circulation, and the copyright problem. Social Media+ Society, 5(2):2056355119833 19839190, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models, September 2016. URL [cs].
07843 1.Xiv:1609.077
Antoine Miech, Dimitri Zhukov, Jean-Baptiss Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowToo10M Learning Text- Video Embedding by Watching Hundred Million Narrated Video Clips, July 2019. URL [es].
org wttporgg/aab.aooeeeeeeeeeeeeeeee 03327 arXiv:1906.03322
Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Zissermann RareAct: A video dataset of unusual interactions, August
01018 arXiv:2008.0100 [cs].
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, September 2018. httht
//arxiv. wttp://wwgw.bbba//...o. 02789 arXiv:1809.027 0278999999999. [cs].
Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in a nonparametric datastore. In NewrlPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A Unified Benchmark for Mathematical Reasoning, March 2023. URL /ttp//aiivv...////// wwwp:ivvorrraaooom abs abs/2210 2210 1rXiv:2210.17511 [es].
17517
Margaret Mitchell. Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman. Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
60

---

Published as a conference paper at ICLR 2025
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-Math: Unlocking the potential of SLMs in Grade School Math, February 2024. URL http:ttt:///xxx..//////// 2402. 14830 arXiv:2402.14883 [cs].
Daniela Moctezuma, Tania Ramírez-delReal Guillermo Ruiz, and Othón Sonzález-Chávez Video captioning: comparative review of where we are and which could be the route, 2022. URL
https://arxe wttp://ogg/////.o/o/// 05976
David Erik Mollberg, Olafur Helgi Jonsson, Sunneva Thorsteinsdottir, Steinthor Steingremsson, Eydis Huld Magnusdottir, and Jon Gudnason. Samrómur: Crowd-soureing Data Collection for Icelandic Speech Recognition. In Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri. Christopher Cieri, Thierry Declerek. Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 3463-3467, Marseille, France, May 2020. European Language Resources Association. ISBN 10-95546--66444 URL https //aclanthologys wttp::y..rg0000....000
1.425
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):502 508, 2019a.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, and Aude Oliva. Moments in Time Dataset: one million videos for event understanding, February 2019b. URL
org/abs/ abs/ 1801. 03150 arXiv:1801 .53150 [cs].
Mathew Monfort, YouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken Moments: Learning Joint Audio Visual Representations from Video Descrip- tions, May 2021a. URL httpt:ttttttii.///p/////a// org abs 2105. 04489 arXiv:2105.0444 [cs, eess].
Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogerio Feris, and Aude Oliva. Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding. September 2021b. URL htttp://arxiv.a http:aawwwvv.orra////.a///////// awt.aassslll 00232 191i:19911000233 [cs, eess].
Nuno Moniz and Luís Torgo. Multi-Souree Social Feedback of Online News Feeds, January 2018.
URL httthh/pp///xiiv/ org. abs//8asl.11.. 07055 1eXivv1801.07000 [cs].
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. OpenDialKG:
Explainable Conver- sational Reasoning with Attention- based Walks over Knowledge Graphs. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 845-854, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10/ggt4wm. URL httpss://ccaaatholoogoooooo
/tw.or////............ PP1-1081
Frank Monton-Park, Licensed to learn: Mitigating copyright infringement liability of generative ai systems through contracts. Notre Dame Journal on Emerging Technology, 5:64, 2023.
Johannes E. M. Mosig, Shikib Mehri, and Thomas Kober. STAR: Schema-Guider Dialog
Dataset for Transfer Leaminng October 2020. arXiv:2010.1 1853 [es].
Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll GLUCOSE: CeeneaaLized and COntextualized Story Explanations, October 2020. URL
00w.009900777.. arXiv:2009.07756 [cs].
Rajiv Movva, Sidhika Balachandar, Kenny Peng, Gabriel Agostini, Nikhil Garg, and Emma Pierson. Topics, authors, and institutions in large language model research: Trends from 17k arxiv papers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1223- 1243, 2024.
61

---

Published as a conference paper at ICLR 2025
Hussein Mozannar, Karl EI Hajal, Elie Maamary, and Hazem Hajj. Neural Arabic Question Answer- ing, June 2019. URL
wwtp/wggg/////ooo////////... abs www.s./90....0 05394 arXiv:1906.05394 [cs].
Nikola Mrksic, Diarmuid Seaghdha, Tsung- Ssing Wen, Blaise Thomson, and Steve Young. Neural Belief Tracker: Data-Driven Dialogue State Tracking, April 2017. URL ptstt://arxiv..o arXiv: [cs].
abs/16055 20ss/606.000 03777
Nikola Mrkšić and Ivan Vulić. Fully Statistical Neural Belief Tracking, May 2018. URL http 11350 arXiv: 11350 [cs].
Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. QASR: QCRI Aljazeera Speech Resource- Large Scale Annotated Arabic Speech Corpus, June 2021. URL
org/ abs/ 2106. 13000 arXiv:2106. 100006..3000 13000 [cs, eess].
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven
Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15991-16111, 2023.
Rajdeep Mukherjee. Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde, Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, and Pawan Goyal. ECTSum: New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10893-10906 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10/gtsqxz. URL. htttt://p///l antho www.nooooooo......ooooooooooyoom ogy. wwg.gyorrr/202.... emn 1omnlp-mainn 748
Will Myers, Tyler Etchart, and Nancy Fulda. Conversational Scaffolding: An Analogy- based Ap- proach to Response Prioritization in Open-domain Dialogs:. In Proceedings of the 12th In- lernational Conference on Agents and Artificial Intelligence, pp. 69-78, Valletta, Malta, 2020. SCITEPRESS Science and Technology Publications. ISBN 978-989-758-3958 gtsq86. http::::wwwwww..itppeee... 10. 5220/22/0000 0122000000939900500 wtt.d//www.........aaaaaaaaaaaa. aspx?doi
doi: 10/
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb large-scale speaker identifica- tion dataset, May 2018. URL httt:://erxiv..rrraba/////0 08612 1eXiv:1706.086 [es].
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Open Vid- IM: A Lcage-Scale Sugaityalityy// Dataset for Text-to-video Generation, August 2024. URL http:://aa/x
2200 arXiv:2407.02371 [cs].
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadía Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. DART: Open- Domain Structured Data Record to Text Generation. In Kristina Toutanova, Anna Rumshisky, Luke Zeutemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 432-447, Online, June 2021. Association for Computational Linguistics. doi: 10/gnh49f. | https \/aclantho wrg.y...r///00 2221 naacl -asacclmmingg mmain.ai.... 37 arXiv:2007.022 [cs].
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryšciński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma. Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQA: Free-form Table Question Answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022. doi: 10/gtsqx3. httpps/::: cotehannthhoooooo logy Place: Cambridge, MA Publisher: MIT Press.
org/2022. tacl- 1.3,
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't Give Me the Details, Just the Summary! Tware-Aware Convolutional Neural Networks for Extreme Summarization, August 2018. URL
http::/arxi: v. www./agg......o /8808 08745 wwXiv:180..0877 [cs].
62

---

Published as a conference paper at ICLR 2025
Linh The Nguyen, Nguyen Luong Tran, Long Doan, Manh Luong, and Dat Quoc Nguyen. A High- Quality and Large-Scale Dataset for English- Vietnamese Speech Translation, August 2022. URL httt::/arxiv.. org/ abs /ta/s//200 2208 04243 arXiv:2208.0422 042443 [es].
Tuan-Phong Nguyen, Simon Razniewski, and Gerhard Weikum. Advanced Semantics for Common- sense Knowledge Extraction. In Proceedings of the Web Conference 2021, pp. 2636-2647, April 2021. doi: 10/gnnffn. URL http:://arx....rg/abbb//o// 00905 TrXiv:2011.0002 [cs].
Suan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang. Chaoqun Liu, Hang Zhang. and Lidong Bing. SeaLLMs Large Language Models for Southeast Asia, December 2023. URL
2312. 00738 #rXiv:2312.0073: [es].
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A New Benchmark for Natural Language Understanding, May 2020. hRtt:/t:///ax:
orq axq/abs/1910. 14599 arXiv:1910.1 19t.99990.44999 [cs].
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The E2E Dataset: New Challenges
For End-to-End Generation, July 2017. URL http::/arxiv...e/////// org. 1rXiv:1706.0990 706.09254 [cs].
Tobi Olatunji, Tejumade Afonja, Aditya Yadavalli, Chris Chinenye Emezue, Sahib Singh, Bonaventure P. Dossou, Joanne Osuchukwu. Salomey Osei, Atnafu Lambebo Tonja, Naome Etori, and Clinton Mbataku. AfriSpeech-200 Pan-African Accented Speech Dataset for Clinical and General Domain
ASR, September 2023. URL httLhttt://aat /arxiv org/ abs 2310. 00274 arXiv:2310.00274 [es].
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıeıman. Social data: Biases, method- ological pitfalls, and ethical boundaries. Frontiers in big data, 2:13, 2019.
Andreca-Marte Oncescu, João F. Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanic. QuerYD: A video dataset with high-quality text and audio narrations, February 2021. URL 11071 arXiv:201 1.1 1071 [cs].
htttp://axxiv.org//bbb
Patrick K. O'Neill, Vitaly Lavrukhin, Somshubra Majumdar, Vahid Noroozi, Yuekai Zhang, Oleksii Kuchaiev, Jagadeesh Balam, Yuliya Dovzhenko. Keenan Freyberg, Michael D. Shulman, Boris Ginsburg, Shinji Watanabe, and Georg Kucsko. SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition, April 2021. URL http:://rxii org/ htg/abs// 2104.11.. 02014 arXiv:2104.02014 [cs, eess].
OpenAI. Hello gpt-4o: We're announcing gpt-4o, our new flagship model that can reason across audio, vision, and text real time., 2024. URL htttts:///peeee hwmppeeai..comimiieemmeeeee ww..mmminnee hello-gpt-4o//
Mariam Orabi, Hozayfa El Rifai, and Ashraf Elnagar. Classical Arabic Poetry: Classification based on Era. In 2020 JEEE/ACS 17th International Conference on Computer Systems and Applications
(AICCSA), pp. 1-6, Antalya, Turkey, November 2020. IEEE. ISBN 77778-1172818-777777 doi: ieee wttp://///oomeeee///55555555555.
10/g6k6b2. URL
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.0211 2022. URL https://
arxiv. org/bbs.ggbbb.... 2203. 02155
Cory Paik, Stéphane Oueel-Oe,,,,,te, Alessandro Roncone, and Katharina Kann. The World of an Octopus: How Reporting Bias Influences w Language Model's Perception of Color, October 2021.
URL http::ttp//aiiia org/ Balgg//bs/// abs/a/ss1110 08182 arXiv:2110.08/ [cs].
Serguei Pakhomov, Bridget McInnes, Terrence Adam, Ying Liu, Ted Pedersen, and Genevieve B.
Melton. Semantic Similarity and Relatedness between Clinical Terms: An Experimental Study.
AMIA Anmual Symposium Proceedings, 2010:52--57 2010. ISSN 1942-597X. (tthtts
www. ncbi n mmm. .onm.it ww....i..hhgg.mmmmmm http:///w../////////////tttttaaa http://:aaaaaiiceeee..c00000.11.
63

---

Published as a conference paper at ICLR 2025
Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Carl Yuheng Ren. Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception, June 2023. URL http:/ arwiv. wwt//bs///aaaaaooooo 06362
ar 2XXix2336606366 [cs].
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur: Librispeech: An ASR corpus based on public domain audio books. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, South Brisbane, Queensland, Australia, April 2015. IEEE. ISBN 978-1-4673--997-5 org/ dww.ment /1tpeettt77789////
doi: 10/gfv84w. URL http:://eeexpporre ieee
Bo Pang and Lillian Lee. Sceing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, June 2005. URL http://arxiv. www.///gg/bb/ae//ee hstp://aa/sssssse/e/e /tt../oorr///a//////0000777 0506075 arXivics/050607..
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTo: A Controlled Table-To-Texx Generation Dataset. In Bonnie Webher, Trevor Cohn, Yulan He, and Yang Liu (. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173-1186, Online, November 2020. Association for Computational Linguistics. doi: 10/gm3nmg. URL httpps://claatthloogg
org/2020 wwweemmlpp...mmm emnle-main..n..mm 89
Jeongkyun Park, Jung-Wood Wook Hwang, Kwanghee Choi, Seung-Hyun Lee, Jun Hwan Ahn, Rae-Hong Park, and Hyung-Min Park. OLKAVS: An Open Scale Korean Audio- Visual Speech Dataset, January 2023.
06375 arXiv:2301. 106375 [cs].
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Data, data everywhere: A guide for pretraining dataset construction. arXiv preprint 2407.06380, 2024.
Panupong Pasupat and Percy Liang. Compositional Semantic Parsing on Semi-Structured Tables. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguisties and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10/gfz98s. URL http::t/t///llll// http///caannholoogyy.
1oc/P15-1142
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11), 2021.
Nikita Pavlichenko, Ivan Stelmakh, and Dmitry Ustalov. CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription, October 2021. URL http::t/axxii..///
abs/2101 01091 arXiv:2107.01m [cs, eess].
Ted Pedersen, Serguei V. S. Pakhomov, Siddharth Patwardhan, and Christopher G. Chute. Mea-
sures of semantic similarity and relatedness in the biomedical domain. Journal of Biomedi- cal Informatics, 40(3):288-22) 299, June 2007. ISSN 1532-0464. doi: 10/fghjwr. URL https
/ wwww.sccieneedrrec.e.. com/ htt.a//aww...........o////aaaaaa
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The Reb dataset for falcon LLM: Outperforming curated corpora with web data, and web data only. June 2023.
Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, al. Reproducing whisper-style training using an open-source toolkit and publicly available data. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
F. Perazzi, Pont- Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- Sorkine-Hornung A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation. In 2016 IEEE
64

---

Published as a conference paper at ICLR 2025
Conference on Computer Vision and Pattern Recognition (CVPR), pp. 724-732, June 2016. doi: 10/ ggdmmw. URL 6919.
ieee. .rg/ wwtp://ggonoooeo.. dww.ment///////o// /7800t77774444 ISSN: 1063-
Laura Perez-Beltrache Beltrachini, Yang Liu, and Mirella Lapata. Generating Summaries with Topic Templates and Structured Convolutional Decoders, June 2019, httl:tttp//aii..oooo 04687 WeXiv:1906.0465 [es].
6966
Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. Multi-Domaie Goal- -rien-Oreette Dialogues (MultiDoGO): Strategies toward Curating and Annotating Large Scale Dialogue Data. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methwds in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp- 4526 4536, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10/gkr9hj. URL
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language Models Knowledge Bases?, September 2019. URL http: arXiv: 1ete:1999.00660 [cs].
//arxiv. //axxiv.ogg/abs// 909. 01066
Thang M. Pham, Seunghyun Yoon, Trung Bui, and Anh Nguyen. PiC: A Phrase-in-Contex/ Dataset for Phrase Understanding and Semantic Search, February 2023. URL http://arxiv.org abs/ 2207.2277.22 09068 Nlw:2207.09006. [cs].
Sara Pieri, Sahal Shaji Mullappilly, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Timothy Baldwin, and Hisham Cholakkal, BiMediX: Biliinuull Medical Mixture of Experts LLM, February 2024. URL http::tttttttiit
htrggabbs/44000
3253 arXiv:2402.1322 13253 [cs].
Michel Plüss, Manuela Hürlimann, Marc Cuny, Alla Stöckli, Nikolaos Kapotis, Julia Hartmann, Malgorzata Anna Ulasik, Christian Scheller, Yanick Schraner, Jain, Jan Deriu, Mark Cicliebak, and Manfred Vogel. SDS-200: A Swiss German Speech to Standard German Text Corpus, May 2022. httt:ttt://aate orq abs 02005. 09501 arXiv:2205.099
[cs].
Bilal Porgali, Vitor Albiero, Jordan Ryda, Cristian Canton Ferrer, and Caner Hazirbas. The casual conversations v2 dataset, 2023. URL httpss:/aaxxvvv.rr.em /2 04838
Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. Goodtriever: Adapiie toxicity mitigation with retricval- -aggeettd models, 2023. URL
wtt.a/xwiiwr...a//////aaaaaaaa// 223/110
07589
Rohit Prabhavalkar, Takaaki Hori, Tara N Sainath, Ralf Schlüter, and Shinji Watanabe. End-to- end speech recognition: A survey. JEEF/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert, MLS: A Large- Scale Multilingual Dataset for Speech Research. In Interspeech 2020, pp. 2757-2761, October 2020. doi: 10/grk6mp. URL http: [cs, eess).
03411 arXiv:2012.03411
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi,, et al. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research. 25(97):1-52, 2024.
Adithya Pratapa, Rishubh Gupta, and Teruko Mitamura. Multilingual Event Linking to Wikidata, July 2022. URL httt::tttaxiiiii org/ abs 221. 06535 arXiv:2204.0653. [cs].
Kun Qian, Satwik Kottur, Ahmad Beirami, Shahin Shayandeh, Paul Crook, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. Database Search Results Disambiguation for Task-Oriented Dialog Systems. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1158-1177 1173, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10/gtsqx2. URL https 7/aclanthe ogg. 2022. naagl -mai i. 85
65

---

Published as a conference paper at ICLR 2025
Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. GECOR: An End-to-Enn -ndd Generative Ellipsis and Co-reference Resolution Model for Oriented Dialogue. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNN.P-JJCNLP) pp. 4547-4557, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10/gk3btq. URL https:://acLantooooggy
Empirical
org/D19- 1462
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00023 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492-28518, PMLR, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint ar arXiv:2305. 18290, 2023.
Nishant Rai, Haofeng Chen, Jingwei Ji, Rishi Desai, Kazuki Kozuka, Shun Ishizaka, Ehsan Adeli, and Juan Carlos Niebles. Home Action Genome: Cooperative Compositional Action Understanding. May 2021. URL http: wwtp::iw..wi.aaaaaaaoeeoaaa /2105 05226 arXiv:2105.05221 [cs].
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain Yourself!
Leveraging Language Models for Commonsense Reasoning, June 2019. /Rt/ttp:: http::/taaattt
httg/bs/11900 02361 arXiv:1906.02366 [cs].
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 05250, 201
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text, October 2016b.
05250 arXiv:1606.0522 [cs].
Kavitha Raju, Anjaly Ryan Lish, and Joel Mathew. Snow Mountain: Dataset of Audio Recordings of The Bible in Low Resource Languages, May 2023. URL httlltttpp/axx..cooc h/s////a///// 2r200////aa 2206 [cs, eess].
01205 arXiv:2206.0
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv: arXiv:2204.06132 April 2022.
Revanth Rameshkumar and Peter Bailey. Storytelling with Dialogue: A Critical Role Dungeons and Dragons Dataset. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5121-5134, Online, July 2020. Association for Computational Linguistics. doi: 10/gtsqxp. URL
acl-main. 459
Francis McCann Ramirez, Luka Chkhetiani, Andrew Ehrenberg, Robert McHardy, Rami Botros, Yash Khare, Andrea Vanzo, Taufiquzzaman Peyash, Gabriel Oexle, Michael Liang, et al. Anatomy of industrial scale multilingual ast. arXiv preprint arXiv:2404.00 401.0444 2024.
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter, 2022. URL httttsss///axiii..rrrr wttp///rgg/aabb/..../oo//
Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. A Local-to-Global Approach to Multi-modal Movie Scene Segmentation, April 2020. URL
http://arxivv http:///xxvvvo//o orgg abs 2aabs/2004 2004 02678 arXiv:2004.02678 [cs].
Hannah Rashkin, Eric Michael Smith, Margaret Li. and Y.-Lan Boureau. Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset, August 2019. http
7/ Tarxiiv.cre wwwp...wr//.b//11111111
00207 arXiv:181 11110002007 [cs].
66

---

Published as a conference paper at ICLR 2025
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards Scalable Multi-domain Conversational Agents: The Schema- Saneead-Guiedd Dialogue Dataset, January 2020.
P.Xiv:1909.0550 [cs].
Siva Reddy, Danqi Chen, and Christopher D.Maniin CoQA: A Conversational Question Answering
Challenge, March 2019. URL htt.::ttp///xii...tt abs httpb///8888.0000 07042 MoXiv:188.00700 [cs].
Eric P Robinson and Yicheng Zhu. Beyond "i agree": Users' understanding of web site terms of service. Social media+ society, 6(1):2056305 198973211 2020.
Anna Rogers. Changing the world by changing the data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
on Natural Language Processing (Volume I: Long Papers), pp. 2182-2194, Online, August 2021. Association for Computational Linguistics. doi: 10. 10..1865314 (0.118633/11/221..oo hww.:oog....l-lo..117.. URL
https: httpaa//cccaaattooooooooggg2//..
Omid Rohamian, Mohammadmahdi Nouriborji, and David A. Clifton. Exploring the Effectiveness, of Instruction Tuning in Biomedical Language Processing, December 2023, URL
org/aba/ 2401. 00579 arXiv:2401.00577 [cs].
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. Dataset for Movie De- scription, January 2015. URL http:://axxv..orgg//////o/////// abw/aaas..111..... 02530 arXiv:1501.02530 [cs].
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie Description, May 20. URL http://///aaii org/abs/1600 03705 arX.v:1605.03705 [cs].
Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data. International Journal of Computer Vision 11963)33463313 337333,,,, September 2016b. ISSN 0920-5691, 1573-1405. doi: 10/f8wókp. BRRLhttp:2 http/aaaivvv//. org/ abs 1502 06648 8
1rXiv:1502.06644 [cs].
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shiv- alika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fer- nando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzeminski, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzich Fadace, Sara Hooker, and Antoine Bosselut, Include: Evaluating multilingual language understanding with regional knowledge, 2024. URL wttps:t/t///........ httpivi///axxv...o/ org/ abs 1t/.//111.1111111111111111111111 19799
Joseph Roth, Sourish Chaudhuri, Ondrej Klejeh, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, and Caroline Pantofaru. AVi-ActtieSpeeaker: An Audio Visual Dataset for Active Speaker Detection, May 2019.
URL
abs/ wwwpsii..rrg/ab///99000 01342 arXiv: 1901:1901.00 101 342 [cs, eess).
org
Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, and Kevin Murphy. XGAN: Unsupervised Image-to- Image Translation for Many-to-Many Mappings, July 2018. URL [cs].
abs 05139 arXiv:1711.05133
Askar Rozi, Dong Wang, Zhiyong Zhang, and Thomas Fang Zheng. An open/free database and Benchmark for Uyghur speaker recognition. In 2015 International Conference Oriental COCOSDA
67

---

Published as a conference paper at ICLR 2025
held jointly with 2015 Conference Asian Spoken Language Research and Evaluation (O- COCOSDA/CASLRE), pp. 81 85, Shanghai, Chima, October 2015, IEEE. ISBN 978-1-4673-8279933 doi: 10/grh5rd. URL http: http::://eeeeppooee ieee httgg//cccmmm//3355785e58
Alexander M. Rush, Sumit Chopra, and Jason Weston. A Neural Attention Model for Abstractive Sentence Summarization, September 2015. URL //ta////......aaaoooooooooooo///
arXiv:1509.00685 [es].
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211 252, December 2015. ISSN 1573-1405, doi: 10geegkww. URL https
//doi.org/ 0. 1007/>11263
0816-Y
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. Language Modelling with Pixels, April 2023. URL http:://axxiv..rg/ abs/ 2207/2207...0 06991 arXiv:2207.066: 2.0999 [cs].
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guil- laume Bouchard, and Sebastian Riedel. Interpretation of Natural Language Rules in Conver- sational Machine Reading, August 2018. URL
1rXiv:1809.01499 [cs, stat].
Matthew J. Sag. The new legal landscape for text mining and machine learning. In Journal of the Copyright Society of the USA, 2020.
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension, October 2018. URL lhttp://arxis org/ abs/ 804 07927 arXiv:1804.07927 [cs].
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale, November 2019. URL http://arxiv
org/abs/1907 907. 10641 arXiv:1907. 10641 [es].
Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, and Matt Post. The Multilingual TEDx Corpus for Speech Recognition and Translation, June 2021. URL arXiv:2102.01757 [cs].
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "Everyone wants to the model work, not the data work": Data cascades high- stakes AI. In CHI, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. doi: 10.1 14/////11164.44555551
ISBN 9781450380966.
htRLhhtps:://dde wot.ddi..r//..........1
3411 1411764.200 3445518
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. How2: A Lcrge-scale Dataset for Multimodal Language Understanding, December 2018. http::/t:: http///rxxiv..r//aba./a// 1811 111111100000 C0347 181 .00347 [cs].
Ramon Sanabria, Nikolay Bogoychev, Nina Markl, Andrea Carmantini, Ondrej Klejch, and Peter Bell. The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR, March 2023. URL http: /ttt:://arxinn orq/ abs 2abbs/2233.1880 arXiv:2303.188 [es. cess].
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Scao, Arun Raja, et al. Multitask prompted training enables zero- shot task generalization. ICLR 2022, 2021. URL
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnadd Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Unmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen,
68

---

Published as a conference paper at ICLR 2025
Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask Prompted Training Enables Zero-Shot Task Generalization, March 2022.
arXiv:21 10.08207 [cs].
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SociallQA: Common- sense Reasoning about Social Interactions, September 2019. URL /BLLtttp://ax....o/////// [cs].
1904. 09728
Max Savery, Asma Ben Abacha, Soumya Gayen, and Dina Demner- Fushman. Question-Driven Summarization of Answers to Consumer Health Questions, May 2020. URL http::/araxi
org/abs/2005 09067 arXiv:2005.0000 [cs].
Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kramias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. ARB: Advanced Reasoning Bench- mark for Large Language Models, July 2023. URLhttp::/arxxtt org. http:////aaaaa..orraaaa/2211111 13692 MtXiv:2307.13692 [cs].
Madeline C. Schiappa, Yogesh S.
Rawat, and Mubarak Shah. Self-supervised learning for videos: survey, ACM Computing Surveys, 55(13s): July 2023. ISSN 1557-7341. doi: 10.1 145/ 3577925. URL http://///d dowp//xxxxoii..... wtw.://oo.oooggg........o 1114/35575999
Claudia Schulz, Josh LevyeKramer, Camille Van Assel, Miklos Kepes, and Nils Hammerla. Biomedical Concept Relatedness - large EHR-based benchmark, October 2020. URL
http://arxiv. wr..rrg/.o abw.aa///122/ 20000 16218 arXiv:2010.161 162118 [es].
Abigail See, Peter J. Liu, and Christopher D. Manning. Get To The Point: Summarization with Pointer-Generate Networks, April 2017. URL htttp::/arx....rgrabb///////s//s arXiv:1704.04366 [es].
Pararth Shah, Dilek Hakkani-Tüir Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. Building Conversational Agent Overnight with Dialogue Self-Play, January 2018. URL
org/ abs////ss11111 1801. 04871 arXiv: 111111114880 104870871 [cs].
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis, April 2016. URL URL http:::/aaxx...og///eee./// abs wwtp://rrraaasss.....o. arXiv:1604.02808 [cs].
Igor Shalyminov, Sungjin Lee, Arash Eshghi, and Oliver Lemon. Few-Shot Dialogue Generation Without Annotated Data: A Transfer Learning Approach, August 2019. URL http://arxivv org/abs/ 1908. 05854 arXiv:1908. 058554 [cs].
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. Understanding Human Hands in Contact at Internet Scale, June 2020. URL cttp://aaxiv.orr//ae//0066 06669 arXiv:2006.06669 [cs].
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazir- giannis, and Tean-Pierre Lorré Unsupervised Abstractive Meeting Summarization with Multi- Sentence Compression and Budgeted Submodular Maximization. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers). pp. 6646774 Melbourne, Australia, July 2018. Association for Com- putational Linguistics. doi: 10/gm8jvb. TtRLbttps:/// httpss////aaaaatooooooorr/r/////
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint 17X 2017.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. FineGym: A Hierarchical Video Dataset for Fine- grained Action Understanding, April 2020. (KLL:ttpp//aarrv..rr//////2...0. AXXi:20004006044 [cs].
Aidean Sharghi, Jacob S. Laurel, and Boqing Gong. Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach, July 2017. URL htttp://arxiv.oor/ abs/ tos/1707..m 04 960 arXiv: 1 [cs].
69

---

Published as a conference paper at ICLR 2025
Eva Sharma, Chen Li, and Lu Wang. BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization, June 2019. URL arXiv:1906.03741 [es].
03741
Vivek Sharma, Makarand Tapaswi, and Rainer Stiefelhagen. Deep Multimodal Feature En- coding for Video Ordering, April 2020. URL arXiv:2004.02205 [es].
02205
Jiatong Shi, Jonathan D. Amith, Xuankai Chang, Siddharth Dalmia, Brian Yan, and Shinji Watanabe. Highland Puebla Nahuatl Speech Translation Corpus for Endangered Language Documentation. In Manuel Mager, Arturo Oncevay, Annette Rios, Ivan Vladimir Meza Ruiz, Alexis Palmer, Graham Neubig, and Katharina Kann (eds.), Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pp. 53 63, Online, June 2021, Association BtRLhttps:/// att.a/aaaaoloy.oooo////// for Computational Linguistics. doi: /tw.w/////////////////o//ooooooo
Americasnlp-11
Ying Shi, Askar Hamdullah, Zhiyuan Tang, Dong Wang, and Thomas Fang Zheng. free Kazakh speech database and speech recognition baseline. In 2017 Asia-Pacific Signal and Information
Processing Association Annual Summit and Conference (APSIPA ASC), pp. 745-748, Kuala Lumpur, December 2017. IEEE. ISBN 978-1-5386-1542-
doi: 10/tsqzzf. URL http://
leeexplore ieee. org/ dww.ment/////occcccocccc...c wtt.e/mmmmnnt/222232/ 8282133/
E. Shriberg, R. Bates, A. Stolcke, P. Taylor, D. Jurafsky, K. Ries, N. Coccaro, R. Martin, M. Meteer, and C. van Ess-Dykema. Can prosody aid the automatic classification of dialog acts in conver- sational speech? Language and Speech, 41 ( Pt 3-4):443 1998. ISSN 0023-8309. doi:
101117/02233099804000110
Mohit Shridhar, Xingdi Yuan,
Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, March 2021. URL
org.
wwtggaab/. abs / 2010 03768 arXiv:2010.03768 [cs].
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding, 2016a. URL
http::////xx...ooot////////// 01753
Gunnar A. Sigurdsson, Güi Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.
Hollywood Homes: Crowdsourcing Data Collection for Activity Understanding. July 2016b. URL httttp://axxi..or/////
01753 arXiv:1604.011 1604.01753 [cs].
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and Observer: Joint Modeling of First and Third -erron Videos, April 2018. URL http://
/184. 09627 arXiv:1804.0900 808.0004.6277 [cs].
Damien Sileo and Marie-Francine Moens. Probing neural language models for understanding of words of estimative probability, June 2023. URL http: wwwpiiioorg/abb.... http://aaa/211.1333333
arXiv:2211.03358 [cs].
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Vidd Text-to- Video Generation without Text- Video Data. arXiv: 2tti::2009.149922 September 2022.
URL http::tppp/:ii///// org/ abs 2209 14792
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Het- tiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Mucnnighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadacc, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024a.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Het- tiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh
70

---

Published as a conference paper at ICLR 2025
Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff. Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadace, and Sara Hooker. Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning, February 2024b. URL htttp://axxiv.org// wwtp://wwwwwg../b/ob////oo /20///as.//222..2 06619 arXiv:2402.06119 [es].
Sam Skolnik. Openai sued over using youtube videos without creators' consent.
Bloomberg Law, 2024. URL hag.a//.......tttt.os////os/ooar
-penai- org.a//ne...............oooonaaa
Lucas Smaira, João Carreira, Eric Noland, Ellen Claney, Amy Wu, and Andrew Zisserman. Short
Note on the Kinetics-700-20020 Human Action Dataset, October 2020. URL http:://axtt
oxg/abs/2010 2010 10864 arXiv:2010. [cs].
Imdat Solak. The M-AILABS Speech Dataset caito, January 2024. URL BtLLhttps::///tt htttpp://www.com
de/2013/01/03/5mm m - dataset/
Per Erik Solberg and Pablo Ortiz. The Norwegian Parliamentary Speech Corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 1003-1008, Marseille, France, June 2022, European Language Resources Association. URL https: /ttt::://allanthology.o owwphhoooooorrrg/222220 org/2022 1rre-1..110
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024.
Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba Heilbron, Chen Zhao, Silvio Giancola, and Bernard Ghanem. MAD: A Scalable Dataset for Language Grounding in Vidcos from Movic Audio Descriptions, March 2022. URL http:://arae www.oorgg/bb./...oooo....o..ee/t 2tw.b////1.....0 00431 arXiv:2112.004p3 [cs].
Amir Soleimani, Christof Monz, and Marcel Worring. NLQuAD: A Non-Factoid Long Question Answering Data Set. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pp. 1245-1255 Online, 2021. Association for Computational Linguistics. doi: 10/g6k6th. URL htt.:tte.//acllattoooooooo.ooo//
eaac- main. 106
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A Dataset of 101 Human Actions Classes From Vidcos in The Wild, December 2012. 1212. 0402 ar2.v:1212.0402 [cs].
Sebastian Stein and Stephen McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, UbiComp '13. pp. 729-738, New York, NY. USA, September 2013. Association for Computing Machinery. ISBN 778-1-450331770-2 doi: 10/gtwv9t. URL httpt::tp//... //doi. .or.orrggg /00. 1145/
93442]
A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, Taylor, R. Martin, C. Van Ess-Dykema, and M. Meteer. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3):339-373 September 2000. ISSN 0891-2017, 1530-9312. doi: 10/dqmv4j. URL http:://aatttii org/ htt.b//aa/0000// arXivics/000602..
Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. Detecting personal information in training corpora: an analysis. In Proceedings of the 3rd Workshop Trustworthy Natural Language Processing (TrustNLP 2023), Stroudsburg, PA, USA, 2023. Association for Computational Linguistics.
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang. Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving Complex Constrained Instruction Instruction-Followinng Following Ability of Large Language Models, April 2024. TTRL.ttpp://// htt///xxiv../ hhtp:://arxiv..rr/aaaa//aaa /404. C2823 arXiv:2404.02825 [cs].
71

---

Published as a conference paper at ICLR 2025
Jennifer J. Sun, Ting Liu, Alan S. Cowen, Florian Schroff. Hartwig Adam, and Gautam Prasad. EEV: A Large-Scale Dataset for Studying Evoked Expressions from Video, February 2021. URL
org/ abs 2001 05488 a arXiv:2001.05488 [es].
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships, November 2018. URL
http://arxiv/ wwtp://.ggg////oe/oo/o/ooooose/s
08048 181 [es].
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions, September 2019. URL http:://aaxi...gg////aaaaaa
03553
[cs].
Shinnosuke Takamichi, Ludwig Kürzinger, Takaaki Saeki, Sayaka Shiota, and Shinji Watanabe. JTube- Speech: corpus of Japanese speech collected from YouTube for speech recognition and speaker veri- fication, December 2021. URL /tttp::///axivvi org/ abs 2212.0922223 arXiv:2112.09323
[cs, eess].
Alon Talmor and Jonathan Berant. The Web as Knowledge-Base for Answering Complex Questions. Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume (Long Papers), pp. 641-651, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10/gk22k6. URL
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@A:: A Question Answering Challenge Targeting Commonsense Knowledge, March 2019. URL httth://a///xii
org/abs/1810 00937 arXiv:1811.00933 [cs].
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannanch Hajishirzi, and Jonathan Berant. MultiModalQA: Complex Question Answering over Text, Tables and Images, April 2021. URL arXiv:2104.00039 [cs].
Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. WIQA: dataset for "What if..." reasoning over procedural text, September 2019. URL httth://ap//xi
org/abs 04739 arXiv:1909.04739 [cs].
Raphael Tang, Rodrigo Nogueira, Edwin Zhang, Nikhil Gupta, Phuong Cam, Kyunghyun Cho, and Jimmy Lin. Rapidly Bootstrapping Question Answering Dataset for COVID-19, April 2020. URL
arXiv:2004. [cs].
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie
Zhou. COIN: Large-scale scale Dataset for Comprehensive Instructional Video Analysis, March 2019.
URL http::t/axxii...../// org/ abs abs/1903. 02874 arXiv: N9w.11903.087.. [cs].
Zhiyuan Tang, Dong Wang, Yanguang Xu, Jianwei Sun, Xiaoning Lei, Shuaijiang Zhao, Cheng Wen, Xingjun Tan, Chuandong Xie, Shuran Zhou, Rui Yan, Chenjia Lv, Yang Han, Wei Zou, and Xiangang Li. KeSpeech: An Open Source Speech Dataset of Mandarin and Its Eight Subdialects. In Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks, volume December 2021. URL http:://darr.aa...-eeecmaaaaaaet htat//333ccaa0055522244333333333
neuziss......u.....o............ ec wttpcppp/p//pp/////peepepp Abst round2. html
Makarand Tapaswi, Martin Bauml, and Rainer Stiefelhagen. StoryGraphs: Visual- izing Character Interactions as Timeline. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 827-834, 2014. btttps://openaccess theevf. wwwceoo/cccieeeeee content cvpr 2014/tttl/ta// /aaaassiaaiiiii StoryGraphs Visualizing htw.yyrraapsssisssliiiiinng Character 2014 CVPR paper.html html
URL
Makarand Tapaswi, Yukum Zhu, Rainer Steffelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding Stories in Movies through
September 2016.
htRt:htt:://axt wttpeivv.wrga/oo/iiooeeae
902 arXiv:1512.02902 [es].
72

---

Published as a conference paper at ICLR 2025
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint wtw.:2332.111000. 2023.
Avijit Thawani, Jay Pujara, and Filip Ilievski. Numeracy enhances the Literacy of Language Mod- els. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6960-6967, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10/g6k6w5. URL https: attt:a://aclannthooooo..ooor
mimmpp-main.55
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for Fact Extraction and VERification, December 2018. URL http:
brg/abs/1800 05355
arXiv: 8035553555555 [cs].
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. OpenMathinstruuttI:: A 1.8 Million Math Instruction Tuning Dataset, February 2024. URL
http://arxiv http://aaxvivov.oeeaaa/// abs 2402 10176 arXiv:2402.1011 10176 [cs].
Radim Tyleček and Radim Šára. Spatial Pattern Templates for Recognition of Objects with Regular Structure. In Joachim Weickert, Matthias Hein, and Bernt Schiele (eds.), Pattern Recognition, pp.
364-374, Berlin, Heidelberg. 2013. Springer. ISBN 978-3-642-40602-77 doi: 10/ggwb5g.
Herbert Ulirich, Jan Drchal, Martin Rýpar, Hana Vincourová, and Václav Moravec. CsFEVER and CTKFacts: Acquiring Czech data for fact verification. Language Resources and Evaluation, 57 (4):1571-1605, December 2023. ISSN 1574-020X, 1574-0218. doi: 10/g6k95h. URLhttp?
http
//arxiv. wwwxiv.org/aba//2220
11115 arXiv:2201.11115 [cs].
Asahi Ushio, Fernando Maachea, and Jose Gamacho-Collados Generative Language Models for Paragraph-Level Question Generation, January 2023. URL http:://ax.v.org// orgbbs/// 2210. 03992 arXiv:2210.03992 [es].
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong. Wei Ko, Daniel D' D'souza, Ghemileke Onilude, Neel Bhandari. Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprins arXiv:2402.07827, 2024.
Dave Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 29029-29047. Curran Associates, Inc., 2023. URL
bttps://prcceeings..ee..... wwwpeurr.s....pppppeepeeeeeeeppe httpcc/pppppppppppppeeeeeee hitpapppr.fileeeepppe/////////// 5c 00011622883310222066633331321111 pdf
Paper seeerr-ataattstte and Benchmarks
Paul Vicol, Makarand Tapaswi, Lluis Castrejon, and Sanja Fidler. MovieGraphs: Towards Under- standing Iluman-Centric Situations from Videos, April 2018.
7712.06761 arXiv:1712.0676 [cs].
Anja Virkkunen, Aku Roube, Nhan Phan, and Mikko Kurimo. Finnish Parliament ASR corpus Analysis, benchmarks and statistics, March 2022. URL http:://rxix wwwpsiiiivrrr/aaa. wwt./rsg////eeeeeee....eeeeeeeee /2203
14876 arXiv:2203.14877 [cs, eess].
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, February 2019. URL http:://aaaat
org/ abs / 1804. 0 07461 arXiv:1804.07460 [cs].
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE: A Multi-Task Benchmark for Robustness Evalua- tion of Language Models, January 2022a. a.X0v:2111.02840 [cs].
Changhan Wang, Anne Wu, and Juan Pino. CoVoST 2 and Massively Multilingual Speech- to-Text Translation, October 2020a. URL http://arxiv. http:///www.../////////00/00 arXiv:2007. [cs, eess).
73

---

Published as a conference paper at ICLR 2025
Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: A Large-Scaal Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation, July 2021. URL htt.::://aaxi..orgggoooooo
21011. 00390 arXiv:2101.0033 [cs, eess).
Dong Wang and Xuewei Zhang. THCHS-30: A Free Chinese Speech Corpus, December 2015. URL http://arxiv. wttp://g/......aaaa 01882 arXiv:1512.01882 [cs].
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex Wade, Kuansan Wang, Nancy Xin Ru Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. CORD-19: The COVID- Open Research Dataset, July 2020b. URL http: /at//aiiii//// hbtgg/bba/000a00 10 7 arXiv:2004. [cs].
Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting Code Clones with Graph Neural Networkand Flow- Augmeneee Abstract Syntax Tree, February 2020c. URL http::/arxxyi org/abs 2002. 08653 arXiv:2002.08633 [es].
Wenhao Wang and Yi Yang. Vidprom: A million-scale real prompt-galless dataset for text-to-video diffusion models. arXiv preprint arXiv:2403. 060800 2024a.
Wenhao Wang and Yi Yang. VidProM: A Million-scale Real Prompt- Dataset for Text-to- Video Diffusion Models, September 2024b. Uttphhttp.//arrii...r///////.... arXiv:2403.06098 [cs].
Xiaoxuan Wang. Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating Loleege-Level Scientific Problem-Solving Abilities of Large Language Models, February 2024. URL http: 2363..03...5553 arXiv:2307. 10635 [es].
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VATEX: A Large-Scale, Suglityality Multilingual Dataset for Video-and-Language Language Research, June 2020d.
URL httpt:ttttpii...../////a/// org/ abs/ 03493 1rXiv:1904.03493 [cs].
Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. FERV39K: Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos, March 2022b. URL [cs].
org 2tt./ 220.2222330233333 09463 arXiv:2203.09463
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Ielf-unstruct: Aligning Language Models with Gelf-Generated Instructions, May 2023a. URLpttt:://arx: orw. hbtp/ab/22220 2et.//2222....10100
arXiv:2212.10566 [cs].
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi- Helpfulness Dataset for SteerLM, November 2023b. URL http:
/larxiv.org. abs/ /2111. 09528 (.Xiv:2311.09522 [cs].
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English, February 2023. htttttt///xxiv.....
abs 1912. C0582 arXiv:1912.00582 [cs].
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models zero-shot learners. In International Conference on Learning Representations, 2021.
Wei Wei, Quoe Le, Andrew Dai, and Jia Li. AirDialogue: An Environment for Goal-Oriented Dialogue Research. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. Brussels, Belgium, October 2018, Association for Computational Linguistics. doi: 10/gf6gq2. URL tttt:s://accaathhte
1419
74

---

Published as a conference paper at ICLR 2025
Philippe Weinzaepfel and Grégory Rogez. Mimetics: Towards Understanding Human Actions Out of Context, February 2021. /ttts//s////awt..oo///////////22
arXiv:191..07249
[es].
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing Multiple Choice Science Questions, July 2017. URL httt:://aaaiii org/ htsp//bb/1077 06209 arXiv:1707.0om [cs, stat].
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing Datasets for Multi-hop Reading Comprehension Across Documents, June 2018. URL 1710.06483 arXiv:1710.06481 [cs].
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447-2469, 2021.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. Learning from Task Descrip- tions, November 2020. URL [cs].
08115, arXiv:201 1.08115
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-Complete Question Answering: Set of Prerequisite Toy Tasks, December 2015.
httpa//ww...g//b//////000 05698 arXiv:1502.0566:
[cs, stat].
Adina Williams, Tristan Thrush, and Douwe Kiela. ANLIzing the Adversarial Natural Lan- guage Inference Dataset, October 2020. URL httpp//arxx...cct//bb//22001 arXiv:2010. [es].
E. B. Wilson. Untitled review. The American Economic Review, 4(2))42-444, 1914. ISSN 00028282.
URL http:// jstor. httgg/taaaeell00110
Wongun Choi, Khuram Shahid, and Silvio Savarese. What are they doing? : Collective activ- ity classification using spatio-temporal relationship among people. In 2009 IEEE 12th In- lernational Conference on Computer Vision Workshops, ICCV Workshops, pp. 1282-1289,
Kyoto, Japan, September 2009. IEEE. ISBN 978-144244-4442-7. doi: 10/fv4tmg. URL
wee.eeeeeeeee.............oo.ee. leee org/ /dccnentt/5574611
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC- LLaMA: Towards Building Open-source Language Models for Medicine, August 2023. URL
/tt/:/xxww...g//////////.... 14454 arXiv:2304.14454 [cs].
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A Benchmark for Molecular Machine Learning, October 2018. URLhttp:// wwwp:/iwwwwg/////////aa/babsaa/ abth:/b/////ai 00564 [physics, stat].
arXiv: 1XXi:1703300064
Wangmeng Xiang, Chao Li, Ke Li, Biao Wang, Xian-Sheng Hua, and Lei Zhang. CDAD: A Common Daily Action Dataset with Collected Hard Negative Samples. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3920-3929, New Orleans, LA, USA, June 2022. IEEE, ISBN 978-1-66548-739- https://ieeepploze ieee org/ http://oommeetttttee /
doi: 10/gtsqzp. URL
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. TWEETQA: A Social Media Focused Question Answering Dataset, July 2019a. UtLhttt:///arxi http:///aaa.v..o//a///////////// www.wvoorrggbbbaeee
06292 arXiv:1907.0622 [es].
Yu Xiong, Qingqiu Huang. Lingfeng Guo. Hang Zhou, Bolei Zhou, and Dahua Lin. A Based Framework to Bridge Movies and Synopses, October 2019b. URL http::tttttttiiiie wwwpaxii.irr//.a.oooo..a./a.base 191 0.000.. 11009 1910v 11009 [cs].
75

---

Published as a conference paper at ICLR 2025
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxify- ing language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2390-2399 2021.
Fangyuan Xu, Kyle Lo, Luca Soldaini, Bailey Kuehl, Eunsol Choi, and David Wadden. KIWI: A Dataset of Knowledge-Intensivee Writing Instructions for Answering Research Questions, March 2024a. /tLptttp://axx...rrrgg
03866, 1XXiv:2403003866 [cs].
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5288-5296, June 2016. doi: 10/ggv9gj. URI/sttsss/leeexp..oor
ISSN: 1063-6919.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing, June 2024b. URL
220p///00/b2244044444444 08464
arXiv:2406.0844 08444444444444 [es].
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions, July 2022. htttp://aaxiv.ooto
10337 arX1v:2111.10337
[cs].
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. TVSum: Summarizing web videos using titles. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2179-5187, Boston, MA, USA. June 2015. IEEE ISBN 1-4673-696444
doi: 10/gfsj74.
URL http::t/t///eeepppppp 1 ore. leee. org/ www.d/do.////ooc//oooooooo..eeee 7799154/
Xinyu Yang, Weixin Liang, and James Zou. Navigating dataset documentations in ai: A large-scale analysis of dataset cards on hugging face, 2024. URL httpss//ariii.orrgggggg///////// abs/ 2401
13:22
Zehui Yang. Yifan Chen, Lei Luo, Runyan Yang, Lingxuan Ye, Gaofeng Cheng, Ji Xu, Yaohui Jin, Qingqing Zhang, Pengyuan Zhang, Lei Xie, and Yonghong Yan. Open Source MagicData- RAMC: A Rich Annotated Mandarin Conversational(AAAM) Speech Dataset, March 2022. URL [cs, eess].
http://arxiv. org/ /200 16844 arXiv:2203. 16844
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question An- 09600 T.X.v:1809.09660
swering, September 2018. URL httpttttpp/xi..../o/// wwwp:iiiorrg/baaaa [cs].
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable
Real-Word World Web Interaction with Grounded Language Agents, February 2023. URL http: arXiv:2207.01200 [cs].
Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. SelFee: Iterative Self-Revising LLM Empowered by Self-Feedback Generation, May 2023. URL
htttps://kiistai.
1o/SelFee/
Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos, June 2017. URL
httt:://arxiv.orge/
1507 05738 arXiv:1507.05733 [cs].
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming- Wei Chang, and Jina Suh. The Value of Se- mantic Parse Labeling for Knowledge Base Question Answering. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers). pp. 201-206, Berlin, Germany, August 2016. Association for Computa- tional Linguistics. doi: 10/gkz3hq. URL htRs::/ http::::///aaaaahloogo.o org/P16 2033
Yue Yin, Daijiro Mori, and Seiji Fujimoto. ReazonSpeech: A Free and Massive Cor- pus for Japanese ASR, 2023. URL https://researc.ccem reazon. ap/_static/
Feezonnapecc_mmp2200
76

---

Published as a conference paper at ICLR 2025
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models, October 2023. arXiv:2309. 12284 [cs].
org. abs/23/9/990 12288
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: Conversational Text-to-SQL Challenge Towards Cross Domsain Natural Language Interfaces to Databases, September 2019a. [cs].
URL httptt/ttaaiittii org/ abs/ 05378
arXiv:
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li. James Ma, Irene Li, Qingning Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A Large-Scale Human-Laaheled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL. Task, February 2019b. URL httt:://aaxxiii org. abs/ 1809. 08887 arXiv:1809 88880889888888 [cs].
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. SParC: Cross- Semantic Parsing in Context. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4511-4523, Florence, Italy, July 2019c. Association for Computational Linguistics. doi: 10/gj4fwd. URL https: htttp:://aclanttoooooo..oooo
Weihao Yu, Zihang Jiang. Yanfei Dong, and Jiashi Feng. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning, August 2020. URL wtt.::tpp:///. http:/t::/aa....rggr./ooo/ arXiv:2002.04326 [es].
Xiang Yue, Xingwei Qu Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, October 2023. URL http: http://awxxvvv....aaa/
wtgg/bs////30gg 05653 arXiv:2309.0542 3556639356533 [es].
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. MultiWOZ 2.2 A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines, July 2020. URL http:://axx: wwwprxivworg/abb//00000
arXiv:2007.12720
[cs].
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence?, May 2019. URL TttKk/ttp//aaaavv..c////////21.0 arXiv:1905.07830 [cs].
Aohan Zeng, Mingdao Liu, Lu, Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang, AgentTuning: Enabling Generalized Agent Abilities for LLMs, October 2023. URL http::://xxiv..orrtttt MrXiv:2310.1 12822233 [cs].
abs/23100 12823
Kuo-Hao Zeng, T'snn-Huug Chen, Juan Carlos Niebles, and Min Sun. Title Generation for User Generated Videos, September 2016. URL 1xXiv:1608.07068 [cs].
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition, February 2022a. http:tt:////...////////
2110. 03370 arXiv:211000337 [cs].
Dawen Zhang, Boming Xia, Yue Liu, Xiwei Xu, Thong Hoang, Zhenchang Xing, Mark Staples, Qinghua Lu, and Liming Zhu. Tag your fish in the broken net: A responsible web framework for protecting online privacy and copyright. October 2023a.
Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang. Chenghua Lin, Wenhao Huang, and Jie Fu. Chinese Open Instruction General- ist: A Preliminary Release, April 2023b. URL arXiv:2304.07987 [es].
07987
77

---

Published as a conference paper at ICLR 2025
Jianguo Zhang, Kazuma Hashimoto, Yao Wan, Zhiwei Liu, Ye Liu, Caiming Xiong, and Philip S. Yu. Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection, April 2022b. MxXiv:210600566 [cs].
abaaabass2i/ 2bs/2106.04564 04564
Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang. Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang. Hao Wang. Xiaojun Wu, Zhongshen Zeng, and Chongpei Chen. Fengshenbang 1.0: Being the Foundation of Chi- nese Cognitive Intelligence, March 2023c. http://// org/ http://ab.//.....////// 220.//229920220 arXiv:2209.0297 [cs].
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Clas- sification, April 2016. URL http::/aaxx....rg//aa//1/1// 01666 MMXiv:1509.0122 [es].
Xing Zhang, Zuxuan Wu, Zejia Weng, Huazhu Fu, Jingjing Chen, Yu-Gang Jiang, and Larry Davis. VideoLT: Large-scale Long-tailed Video Recognition, August 2021. URL http: //arxiv 02668 arXiv:2105.0266 [cs].
Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. AppaCane/nntrrucioo-tuuned Large Language Models for Medical Application, April 2024. URL http://arxiv. org abs/ 2210 14558 arXiv:2310.14555 [cs].
Yu Zhang, Daniel S. Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang. Shibo Wang, Zongwei Zhou, Bo Li, Min Ma, William Chan, Jiahui Yu, Yongqiang Wang, Liangliang Cao, Khe Chai Sim, Bhuvana Ramabhadran, Tara N. Sainath, Francoise Beaufays, Zhifeng Chen, Quoe V. Le, Chung-Cheng Chiu, Ruoming Pang, and Yonghui Wu. Bigssl: Exploring the frontier of large-scale cemi-supervised learning for automatic speech recognition. JEEE Journal of Selected Topics in Signal Processing, 16(6): October 2022c ISSN 1941-0484, doi: 10.1 1000/sstp.20222.1882. 3182537
URL http: doi org.ioorg//// 1109/ 2022
Yuan Zhang, Jason Baldridge, and Luheng He, PAWS: Paraphrase Adversaries from Word Scrambling, April 2019. URL http:: www.vvorrgaaaaeo
www.oorg///b/o.....o http:/bs/11999 http::/ar.....oo//aa////////////
01130 1rXiv:1904.011 [cs].
Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization. September 2019. URL
http::/arxiv org/
09374 arXiv:1712.00 1.KIv1111.09934 [cs].
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT Interaction Logs in the Wild. In Proceedings of the Twelfth International Conference on Learning Representations, October 2023. URL http:: /ttp////peeeeeieeee//e/ee /
net / forumiiddu
B 737721bM
Liang Zheng, Zhi Bie, Yifan Sun, Jingdong Wang, Chi Su, Shengjin Wang. and Qi Tian. MARS: A Video Benchmark for Large-Scale Person Re-ldentification. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016, pp. 868 884, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46466-4,
doi: 10/gtwv9w.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging I. L.LM-as-a-Judge with MT-Bench and Chatbot Arena, December 2023. URL http arXiv:2306.0568 [cs].
/arrxiv. www.iv.org/abb.... hbtp://bb////0/// 05685
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zhuohan Li. Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMSYS-Chat I A Scale-Scale Real- Real-World LLM Conversation Dataset, March 2024a. URL http: org/ / abs/ / 2309. 11998 arXiv:2309.1 1 11998 [cs].
Lu Zheng, Tongtong Zhou, Ronggi Jiang, and Yueping Peng. Survey of video object detection algorithms based on deep learning. In Proceedings of the 2021 4th International Conference
78

---

Published as a conference paper at ICLR 2025
Algorithms, Computing and Artificial Intelligence, ACAI '21, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450385053, doi: 10.1145/350834633506622
URL http:/ttp///...ogg//o.oooo 1t1.://.00.......555555.3..11100 3508666266
Zangwei Zheng, Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March
com/ 2024b. URL httpt:::tttpthiiie httpaa/w...oooopaaaaaaeaee http//acaatthhoeeeeSee01
Tiankai Zhi, Ying Shi, Wenqiang Du, Guanyu Li, and Dong Wang. M2ASR-MONGO:
Free Mongolian Speech Database and Accompanied Baselines. In 2021 24th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisa-
tion of Speech Databases and Assessment Techniques (O-COCOSDA), pp. 140-145, Singa- pore, Singapore, November 2021. IEEE. ISBN 978-1-66540-870-7, doi: 10/gtsqzg. URL htttps://eeexxplrree ieee org/ wwwwment. 9965/41666
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz. Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization, April 2021. URL http:://ax:t org/
abs 04.004444 05938 arXiv:2104.0590 [cs].
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning, November 2017. URL arXiv: TTt.:1709.00000 [cs].
abs/ 1109/ 00103
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards Automatic Learning of Procedures from Web Instructional Videos, March 2017. /tt////tpe////aaiiv.../r////b//a 2 09788v3 arXiv:1703.09788 [es].
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. DocPrompt- ing: Generating Code by Retrieving the Docs, February 2023a. URL Ttt::2207.0598 [cs].
abs/2207. 05987
Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements, June 2023b. URL [es].
01985 arXiv:2306.01985
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective Vulnera- bility Identification by Learning Comprehensive Program Semantics via Graph Neural Networks, September 2019. URL http:://axxiv.oor//as///0000 03496) arXiv:1909.0344 [es,
stat].
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization, March 2021. URL 06410 arXiv:2103.06410 [cs].
/200
Cunjuan Zhu, Qi Jia, Wei Chen, Yanming Guo, and Yu Liu. Deep learning for video-text retrieval: review, 2023. URL htttp:://a.xi..rrrr.t///
2a.bs/22322 12552
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipireni,, and Chandan K. Reddy. XLCCST: Benchmark Dataset for Cross-lingual Code Intelligence, June 2022. URL http://arxiv. wwt./rggg/////eeee//../eeee
08474 arXiv:2206.08474 [cs].
Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadrezaa Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R. Manmatha, and Mu Li. comprehensive study of deep video action recognition, 2020.
06567
Dimitri Zhukov, Jean- Caantstt Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos, April 2019, URL wwt.:oooggg..ooo. 08225 1XXiv:1903.0822 [cs].
http:://rre http:/aawwvv.oggaa///bbbaa//////
79


Reviews:
[{"review_id": "RYz5bcjCb7", "reviewer": "Reviewer_Yvyf", "summary": "The paper conducts a large-scale dataset audit across text, speech, and video modalities, covering near 4000 public datasets between 1990-2024 by investigating their sources, use restrictions, and their geographical & linguistic representation. It is found that 1) Many multimodal ML datasets are from web-crawled and social media platforms; 2) Inconsistencies between dataset licenses and their source's restrictions prevail; and 3) Inequality in geographical representation remains very high, and geographical & linguistic representation has not significantly improved for many years.", "questions": "• Given that the study includes datasets from 67 countries, when studying data licenses or terms of use, are difference in legal regulations of copyrights between countries/regions taken into consideration?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "• The paper is well-motivated: Most previous works has primarily focused on text datasets, or a single feature or dataset. The paper conducted a multimodal and multi-feature dataset, which addresses this gap.\n\n• The scale of this study is impressive: It covers nearly 4000 datasets, 3 modalities (text, speech, video) and a time span of over 30 years (1990~2024), providing a comprehensive view to dataset provenance study.\n\n• The study points out vulnerabilities of current dataset sources. For example, the paper discovers prevailing inconsistencies between dataset licenses and their source's restrictions, alarming the research community about potential legal breaches. The paper also reveals that many measures to address geographical & linguistic fairness have failed, surprisingly, which motivates the research community to retrospect these measures and potentially propose new (effective) ones.", "weaknesses": "• The paper mentions a rise in synthetic data, but does not give any in-depth analysis of synthetic data v.s. non-synthetic data. I think it is worthwhile to analyze the source distribution of synthetic data v.s. non-synthetic data, so that researchers can know the search space where they are more likely to acquire desired data when they need human-written or machine-generated data.\n\n• The paper is not technically savvy, so it is crucial for the paper to highlight the importance of its insights. Given the large-scale of this study (nearly 4000 datasets) and the engagement of domain experts for annotation tasks, the study should be costly. But the paper did not make it very clear that why studying data provenance (probably with such high cost) is valuable at the first place. Why not use the same annotation labors to create new dataset resources instead, but studying the provenance of existing dataset (intuitively, the former one can be more valuable)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J7uKIKqjqg", "reviewer": "Reviewer_Kmty", "summary": "In this paper, the authors present an audit of nearly 4000 public datasets between 1990-2024, spanning 608 languages, 798 sources, 659 organizations, and 67 countries. They further present their analysis of data sources, geographical and multilingual inclusion.", "questions": "See my comments in the weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to understand. The authors explore an important topic relevant to extremely fast-paced growth of AI models and their adoption. They explore a large-scale collection of datasets across three modalities: text, speech and video. They further present interesting analysis highlighting the community inclusion for dataset creation and the restriction on the usage of the datasets. The study brings out the inequality in geographical representation and the multilingual representation and furthermore the situation has not improved significantly over the last decade on most measures.", "weaknesses": "I find the lack of call-to-action from the authors. There are no concreate directions presented in the paper that could help improve the situation. A mere analysis of the landscape is probably not going to be of much help given the pace with which the field is evolving. They have surveyed so many papers which should have nudged them to form an opinion on the next course of action for the community. \n\nAlso, since this field is growing rapidly, the analysis presented here would become obsolete very quickly. What might help in the long-term is to build a tool that can automate the analysis presented here with such dedicated effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KwEE2km5yT", "reviewer": "Reviewer_AQ3r", "summary": "This work conducts a large-scale and comprehensive audit of nearly 4,000 text, speech, and video datasets between 1990-2024, covering a wide range of tasks, languages, sources, organizations, and countries. Specifically, the authors analyze data trends for the state of data permissions (licenses and terms), sourcing (the web, human annotation, and synthetic generation), and representation (of tasks, organizations, languages, and countries). The datasets included in the audit include those that are publicly available, widely used for general-purpose model development, and relevant to generative tasks. Overall, the authors analyzed 3.7k text, 95 speech, and 104 video datasets. \n\nThe analysis revealed an increasing use of data from web-crawled, social media, and synthetic sources, driven by scaling laws. This was particularly prevalent in speech and video datasets, with YouTube being the most prominent source. The contribution of synthetic text data has also rapidly become significant in recent years. Additionally, the authors found inconsistencies in data source terms and documented license restrictions, with the former being much more restrictive. Further, many data collections do not clarify commercially licensed versus restrictive datasets. Finally, the analysis revealed that while there has been an increase in the languages and countries represented in the datasets, the relative contribution remains Western-centric.", "questions": "* Can the source of synthetic datasets (e.g. the models they were generated from) be determined? Are there any challenges in determining such sources?\n* Why was the scope of the datasets limited to generative tasks?\n* Was there a variation in the tasks represented by the data across the years? It would be interesting to observe emerging trends in AI research over the years (e.g. code generation and other domain-specific tasks).\n* Was the correlation between data sources and language/geographical diversity analyzed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper provides novel insights from a large-scale and comprehensive audit of widely used datasets across text, speech, and video modalities.\n* The paper is well-written and organized.\n* Larger implications of the authors’ findings are discussed, highlighting challenges and suggestions for practitioners.", "weaknesses": "* The audit does not include image datasets, which are widely used in multimodal settings.\n* The analysis does not include the tasks that the datasets are designed for.\n* Some more insights and analysis can be provided for the cause of observed trends- e.g. the sharp rise in encyclopedia-based and internet video-based sources in text and speech datasets after 2018, the drop in the Gini coefficient for geographic representation in video datasets after 2019, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nkczuDO1aN", "reviewer": "Reviewer_BAwp", "summary": "The authors conduct a comprehensive audit across modalities—popular text, speech, and video datasets—from their detailed sourcing trends and use restrictions to their geographical and linguistic representation.", "questions": "1. I'm curious that did you perform analysis on the portion of synthetic data and their characteristics? I think this is one of the most concerned topics by far, though it's really hard to correctly collect all the data.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is an extremely exciting work for multi-modal researchers, as one of the main bottlenecks is how to get balanced and good quality data from the wild. This work deepens the communities understanding of multi-modal data conditions at present and in the history.\n2. The authors conducted rigorous data collection and analysis work, which deserves much credit.\n3. The insights provided are valuable and interesting, which may provide great inspirations for people curating multi-modal data from real world.", "weaknesses": "1. I feel the data collection specifications are limited. Though the authors has mentioned that the discussion for geographical and linguistic representation is limited, I think more discussion should be casted on the data collection methods, as this is the foundation of all discussions in this work. How to ensure the recall and precision of data collection? The data and its conclusion could easily get biased when missing or mistakenly putting a large dataset into the data pool.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)