The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Authors: Shayne Longpre
Stella Biderman
Alon Albalak
Hailey Schoelkopf
Daniel McDuff
Sayash Kapoor
Kevin Klyman
Kyle Lo
Gabriel Ilharco
Nay San
Maribeth Rauh
Aviya Skowron
Bertie Vidgen
Laura Weidinger
Arvind Narayanan
Victor Sanh
David Ifeoluwa Adelani
Percy Liang
Rishi Bommasani
Peter Henderson
Sasha Luccioni
Yacine Jernite
Luca Soldaini
Venue: Accepted by TMLR ()

Abstract: Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and resources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. We hope this curated collection of resources helps guide more responsible development. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed so that capabilities and impact are assessed in context.

Published in Transactions on Machine Learning Research (12/2024)
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Shayne Longpre," MIT
Stella Biderman," EleutherAI
Alon Albalak, UC Santa Barbara, SynthLabs Hailey Schoelkopf, EleutherAI
Daniel McDuff, University of Washington
Sayash Kapoor, Princeton University Kevin Klyman, Stanford University, Harvard University Kyle Lo, Allen Institute for Al
Gabriel Ilharco, University of Washington
Nay San, Stanford University Maribeth Rauh, Google DeepMind
Aviya Skowron, EleutherAl
Bertie Vidgen, ML Commons, Contextual Al Laura Weidinger, Google DeepMind
Arvind Narayanan, Princeton University
Victor Sanh, HuggingFace
David Adelani, University College London, Masakhane
Percy Liang, Stanford University
Rishi Bommasani, Stanfort University
Peter Henderson, Princeton University
Sasha Luccioni, HaggingFace Yacine Jernite," HuggingFace Luca Soldaini,* * Allen Institute for Al
Reviewed OpenReview: https: // openreview. net/forun? forum? id= tH1 dQH20eeZ
Abstract
Foundation model development attracts a rapidly expanding body of contributors, scien- tists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and re- sources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed for capabilities to be assessed in context.
" Equal contribution. Correspondence: slongpre@mit.ecu.

---

Published in Transactions on Machine Learning Research (12/2024)
1 Introduction
As the capabilities (Üstün et al., 2024; Team et al., 2023; Gomez, 2024; Anthropic, 2024a; Radford et al., 2023; Brooks et al., 2024) and market prospects (Vipra & Korinek, 2023; McElheran et al., 2024) of artificial intelligence have quickly expanded, so have the communities of developers, scientists, and contributors who build foundation models (Bommasani et al., 2021). The fields' growth has spurred widespread adoption of many tools and resources used to build, deploy, evaluate, and govern large foundation models (FMs). However, these nascent practices are often immature. Many outstanding resources are neglected, in part for lack of discoverability, or awareness of good practices. To address these gaps, we conduct a focused survey, not of scientific literature (which already exists for many FM development topics (Albalak et al., 2024; Zhao et al., 2023a; Chang et al., 2023)), ) but of resources for FM development such datasets, software packages, documentation, guides, frameworks, and practical tools. In particular, this resource curation is tailored to responsible development practices for newer, smaller, or mid-sized development teams. Large FM development organizations, such as Google, OpenAl, or Meta, with substantial user bases, should adhere to more rigorous and product-specífic best practices than outlined in this review. We release the Foundation Model Development Cheatsheet, the repository of annotated tools for text, speech, and vision models, and open it for public contributions.
For each phase of model development, our contributions are summarized as (i) a survey of relevant tools and resources, (ii) a synthesis of the literature's recommended practices and use of those tools, and (iii) a review of the limitations and omissions of existing resources. The cheatsheet serves and recommended practices, prepared by foundation model developers for foundation model developers. The intended audience is a range of foundation model developers, including academic researchers, startup companies, research labs, who are pretraining from scratch, or simply finetuning, big and small.
succinct guide of the survey
Our survey and recommendations hope to bring wide attention to tools across several phases of development. First, we suggest resources that support informed data selection, processing, and understanding (SS 3 and 4). Data prepared without sufficient due diligence can lead unintended consequences (e.g. risks to privacy, copyright, or generating sensitive content), marginalization (e.g. by inadvertently filtering out certain distributions), or unexpected model behaviors (e-g. train/ /test overlap or security vulnerabilities). Next we survey resources for precise and limiiatiionaaunl artifact documentation (Section 5). When new datasets are released, setting their governance standards early will avoid misuse later. Foundation model training can be financially and environmentally expensive. We aggregate resources for efficient model training (Section 6) and estimating a model's scaling behavior and environmental impact (Section 7). Advance awareness of these quantities can inform more efficient training practices. For once models are trained, we provide evaluation frameworks, taxonomies of risk, and benchmarks for a variety of evaluation criteria (Section 8). Best practices suggest models should be evaluated for their intended uses, well some unforeseen misuses harms. Developers should design naturalistic evaluations for these settings rather than relying available but poorly fitting tools (Biderman et al., 2024b; Liao & Xiao, 2023). And our evaluation frameworks suggest evaluation metrics should be contextualized, to avoid over-claiming or misunderstanding the limitations of the reported numbers. Lastly, our survey informs responsible model release and deployment practices (Section 9), so developers can make informed selections of licenses and release mechanisms, to address misuse risks.
Beyond the survey of resources, we review the existing ecosystem of tools and resources. For each segment of model development, we examine the limitations, omissions, and opportunities for improvement of existing tooling and common practices. Summarized in Table 1, we find:
Tools for data sourcing, model evaluation, and monitoring are critically under-serving responsible de- velopment needs and real-world needs. For instance, they often fail to have sufficient documentation, imitate real use cases, or accurately reflect licensing permissions.
Popular model safety, capabilities, and environmental impact evaluation benchmarks lack repro- ducibility and transparency.
2

---

Published in Transactions on Machine Learning Research (12/2024)
Resources for multilingual and multi-modal development, across every phase of development, continue to receive significantly less attention than English and text-centric equivalents.
Resources to evaluate systems, rather than just models, is needed that capabilities and impact are assessed in the context of real-world deployment.
Documeniatio
Esimaling Impast Effcient Use of Rescurces
Gevemance
Environmental Impact
Documentation
Sources
Proparation
Training
Evaluation
Anotyst
Prefraining Finetuning Effoiency
Capasiltes
Model Documentaton
Fineturing
Exploration
Risce Harms
License Selectin
Clearing
Reproducttiity
Fittring
Usage Monttoring
Deduplicatonn
Decce nation
Audkiing
Figure 1: The phases of model development by which this survey and review paper is organized.
2 Methodology & Guidelines
We develop the following methodology to guide the collection of these tools and resources, as well as our analysis. First, we've ve've divided the model development pipeline into several phases, illustrated in Figure 1. Despite the visual representation, we acknowledge these phases are frequently interrelated rather than sequential. We also include categories that have been identified by existing scholarship necessary responsible development, even though they are frequently omitted from development pipelines: such documentation, environmental impact estimation, risks and harms evaluation. Next, authors allocated to these phases based on their areas of expertise, or to modalities (text, vision, or speech) across a few phases of development. Each author surveys the literature in their segment of model development to find a mix of (a) relevant tools, in the form of repositories, APTs, or interfaces, (b) scientific literature that directly surveys or guides development decisions, and (c) frameworks or standards for development (such as documentation or risk taxonomies). Due the breadth of categories, the literature survey inevitably non-exhaustive, but reflects dedicated search for the most prominent tools in each area. Certain phases of development, such curate criteria for inclusion (outlined
model training, have thousands of available repositories, below) that prioritizes certain qualities: popularity, usefulness, and advancing responsible practices.
Criteria for Inclusion. These principles for inclusion, described below, are incomplete and subjective, but we still believe sufficiently rigorous to make for thorough and useful compilation of resources. The resources are selected based on a literature review for each phase of foundation model development. Inclusion is predicated on a series of considerations, including: the popularity of the tool on Hugging Face or GitHub, the perceived helpfulness as development tool, the extent and quality of the documentation, the insights brought to the development process, and, in some cases, the lack of awareness useful resource has received in the AT community. For example of this last consideration, in Section 3.2 we try to include more Finetuning Data Catalogs for lower resource languages, that often receive less attention than ones featured more prominently on Hugging Face's Dataset Hub. Rather than sharing primarily academic literature as in most surveys, we focus on tools, such as data catalogs, selectively, literature that summarizes, surveys or guides important development decisions. Further, we hope to make the coverage more comprehensive with an open call for community contributions.
tools, evaluation repositories, and,
Scope & Limitations. We've compiled resources, tools, and papers that guide model development, and which we believe will be especially helpful nascent (and often experienced) foundation model developers. However, this guide is far from exhaustive-aa and here's what to consider when using it:
3

---

Published in Transactions on Machine Learning Research (12/2024)
DEVILORMENT PISASE
FREqUINT STATUS Quo
REVIEw & RECOMMMENDATIONS
Predominantly English, and text-centric resources.
More multilingual, multi-modal, and flexible resources.
All Phases
concentration of standalone, flashy, one-off projects, for credit incentives.
More collahorative, large-scale interoperable infrastructure projects that builds on existing tooling.
Predominantly synthetic, unrealistic finetuning data.
Naturalistic observations and realistic training tasks.
Predominantly English, and text centric data sources.
More multiingual, multi modil data sources.
Data Sources ($3)
Intended uses, licenses, consent, & provenance are haphazardly documented.
Prioritizing datasets with slandardized, structured, and linked metadata.
Sparse information especially on data's sensitive content, such as CSAM and NCUI
Comprehenstve "data measurements" as part of large, unstruc- tured data releases.
Data & modeling focus on casy-to-scale formats, e.g. captioned Images.
More attention to collecting neglected data formats, e.g, naturally interspersed text and images.
Diverse data Infrastructure and procesing standards for individ- use-cases.
Interoperable data formats and processing for many use-cases.
Data Preparation (84)
Mostly one-off and closed source data exploration tools.
More open data exploration tools.
Loose ideas of qualityy data that applies to domains.
Concrete precise definitions quality that are unique to set of domains, tasks, or evaluations.
Data preparation methool compared acroes models different training data.
Standardized data-centric benchmarks to fairly compare methods.
More multilingual and low-resource language tokenization and processing tools.
Eingiih-centric lokenization and processing tools.
Data documentation is diversely formatted, often terse, performa- tive, without achieving reproducibiiity.
and analysts scripts Standardized data documentation requirements (eg from conferences).
Executable, and verifallyy reproducible
Data Documentation ($5)
Documentation after-thought.
Documentation is started carly, assembled over the cousse of col- processing.
Data stewardship and maintenance are often ignoved beyond ini- tial release.
Data governance is proactively organized, with post-launch maintenance and licensing plan.
Mostly educational resources technical developers. Inflexible and disparate tooling,
More "last mile" educational resources for non-technical develop- ers.
Model Training (56)
Standardized and centralized resources, especially cross-modality.
Opaque environmental impact estimate programs from cloud providers.
Query-level energy and environmental usage transparency from.
Fine grained transparency from hardware makers and data ters.
Opaque information hardware makers centers.
Environmental Impact (67)
Inability to compare energy standards across systems.
"Energy Star" standards for nen-techrical users fairly compare services.
Multimodal scaling laws research. Intuttive interfaces to estimate and model scaling laws training forecasts.
Scaling laws text centric and impractical.
Evaluating model outputs.
Evaluating systems, within their application contexts.
Evaluating natural observations, real-world settings, and with human-inieraction studies.
Model Evaluation ($8
Evaluating synthetic todiitt and safety benchmarks.
Reporting evaluation metrics only.
Releasing evaluation scripts for verifiable reproducibility.
No uninformed license choices.
License selection guided by the context of data, potential and unforeseen uses, and legal considerations.
Model Release (99)
Weights release with limited support.
Accompanying documentation, and easy-to-run code for training. evaluation, and inference.
Limited plans for usage montorrng, or over claimed benefits from watermarking monitoring.
A plan that considers gating, watermarking, and misuse reporting (though they not always beneficial).
Harm & hazard taxonomies are based on existing benchmarks.
Harm taxonomies are based on empirical observations, and studies with real users.
Table 1: A summary of the reviews & take-aways from each phase of foundation model development ecosystem.
Temporal scope. Foundation model development is a rapidly evolving science. This document is only sample, dated to March 2024.
This not general survey of scientific literature (which can include the- ory, abstract recommendations, and analysis), but on practicable instruments for AI development, evaluation, and safety-which developer can directly apply. These tools and resources specifi-
Tools & Resources:
4

---

Published in Transactions on Machine Learning Research (12/2024)
cally scoped to datasets, databases, frameworks, taxonomies, protocols, interactive websites, APIs, software, and code repositories (for data processing, training, evaluation, or other uses). We also selectively include literature with specific best practice recommendations, and literature surveys for further context on topic.
Applicable developers. We scope these resources to mid-to-small foundation model developers. Large organizations, with commercial services and/or wide user bases, have broader considerations for responsible development that what outlined in this work.
Development phases & modalities We've scoped our data modalities only to text, vision, and speech, and to the phases of development outlined in Figure 1. We support multilingual resources, but acknowledge there remain significant community-wide gaps in awareness and adoption here..
We remind users follow standard practice in assessing the security and viability of each tool, particularly for their circumstance. At times we have provided resources with conflicting advice, is helpful to be aware of divided community perspectives. Our notes throughout designed to contextualize these resources, to help guide the reader's judgement.
3
Data Sources
Data Sourcing Best Practices
Pretraining data provides the fundamental ingredient to foundation models- their capabilities and flaws. Finetuning data improves the model's performance in specific settings, in the case of instruction finetuning or alignment training, improves the model's general usability and helpfulness while aiming to reduce potential harms.
More data is not always better. It is essential to carefully source data, and manually inspect it to ensure it fits the goals of your project.
Dataset selection includes many relevant considerations, such as language and dialect coverage,
topics, tasks, diversity, quality, and representation.
Most datasets come with implicit modifications and augmentations, from their selection, filtering, and formatting. Pay attention to these pre-processing steps, as they will impact your model.
Finetuning data can improve the model's performance in some settings or impair others. Use catalogs to support an informed selection, and prefer well-documented to under-documented datasets.
Crowdsourced data catalogs, including HuggingFace, may contain important omissions and errors in their dataset documentation. Verify information, such as data licensing and critical characteristics, with the original data sources and academic papers, possible.
The most appropriate datasets may not exist for given set of tasks. Be aware of the limitations of choosing from what is available.
3.1 Pretraining Data Sources
Pretraining corpora consist of millions of pieces of content, from documents, images, videos, or speech record- ings, often scraped directly from the web. Model pretraining represents the fundamental step in instilling foundation models with their abilities to represent syntax, grammar, reasoning, and world knowledge (Devlin et al., 2018; Brown et al., 2020; Chowdhery et al., 2023). Consequently, it is important to carefully curate the data composition, including the mix of sources, characteristics, and preprocessing decisions (Longpre et al., 2023b). However, the vast scale of this content often means its shallowly documented and understood, despite community efforts unpack it (Dodge et al., 2021; Elazar et al., 2023).
We highlight few of the most popular pretraining corpora which have accumulated deeper documentation and analysis. In the text domain, web scrapes from common crawl (commoncrawl. org), or OSCAR (https: /ooscar-project... org/) (Suárez et a 2019; Laippala et al., 2022) are the base ingredient for most pretraining

---

Published in Transactions on Machine Learning Research (12/2024)
corpora. In particular, derivatives of common crawl, such as C4 (Raffel et al., 2020; Dodge et al., 2021) multilingual C4 (Kreutzer et al., 2022) provide 2019 indexes that have been heuristically filtered for well- formed text. Subsequent pretraining datasets incorporate one or multiple indexes from common crawl. This includes the Pile (Gao et al., 2020), RefinedWeb (Penedo et al., 2023), RedPajama (Together AI, 2023), and Dolma (Soldaini et al., 2024), which have iterated on basic document quality filtering and deduplication. These corpora often merge and deduplicate multiple years of common crawl scrapes, or supplement additional sources including biomedical text (PubiMed), legal text (Freelaw, USPTO patent documents), code (Github, Stack Exchange), public domain books (Project Gutenberg), among others.
For multilingual text, the Open Parallel COPUS) offers a massive collection of translated text document pairs (Tiedemann, 2012), ROOTS (Laurençon et al., 2022) collates and processes diverse multilingual re- sources, including OSCAR (Laippala et al., 2022) and the Bigscience Catalogue (McMillan- et al., 2022), CulturaX (Nguyen et al., 2023) covers 167 languages from OSCAR and mC4, and WURA (Oladipo et al., 2023) centralizes and manually audits documents from 16 African languages. Most recently, MADLAD-400 (Kudugunta et al., 2023) provides a 3 trillion token, 2023 processed split of Common Crawl, spanning 419 languages.
Large, specialized corpora of text have also recently emerged, to specialize model abilities, or mitigate risks of possible copyright infringement. As examples, the Pile of Law (Henderson et al., 2022) and MultilegallPile (Niklaus et al., 2023) centralize court opinions, contracts, and legislative records; the Stack and Stack v2 scrape permiisively-licesseed GitHub repositories (Kocetkov et al., 2022); peS2o (Lo et al., 2020) releases cleaned academic papers from Semantic Scholar; and the Proof Pile 2 (Azerbayev et al., 2023) and Open WebMath (Paster et al., 2023) aggregate vast mathematical text resources. Notably, recent text corpora also attempt to isolate permissively-lienssee
or copyright free data, such as the Open License Corpus (Min et al., 2023) though this is a challenging task, and does not guarantee that all non-commercially licensed or copyrighted content has been removed.
In the context of speech, webscrapes of English audiobook data from LibriVox (a website hosting free public domain audiobooks) are commonly used for foundation model architecture development and evaluation. Sourcing data from LibriVox, LibriSpeech (Panayotov et al., 2015) is 960 hour fully supervised dataset (i.e. all audio are paired with transcriptions) and Libri-Light (Kahn et al., 2020) is 60k hour dataset for benchmarking using no or limited supervision (10h, 1h, and/or 10min). Multilingual models are typically pre-trained on combination of speech corpora, e.g. for XLS-R (Babu et al., 2022) combination 436k hours from VoxPopuli (400k hours from 23 languages based on European Parliament recordings: Wang et al., 2021) ), Multilingual LibriSpeech (50k hours of audiobooks from 8 languages Pratap et al., 2020), Common Voice (28k hours of crowd-sourced read speech from 100+ languages Ardila et al., 2020), VoxLingua107 (6.6k hours from 107 languages scraped from YouTube Valk & Alumae, 2021), and various LARPA BABEL corpora (totaling 1k hours from 17 African and Asian languages).
In the context of vision, there are several large pretraining corpora, comprised of text and images found in large web scrapes. COYO aggregates 700M images with alt-text from the web. Multimodal C4, or MMC4, (Zhu et al., 2024) leverages the original C4 URLs to extract interleaved image and text pairs, totaling 570M images, and 43B tokens. Similarly, OBELICS also leverages the Common Crawl web collection, filters for multimodal web pages, and extracts images from the HTML (Laurençon et al., 3, totaling 353M images, with 115B tokens. Lastly, DataComp-1B and CommonPool-1138 (Gadre et al., 2024a) isolate high quality subsets of image-text pairs on CommonCrawl. For video specifically, WebVid (Bain et al., 2021) provides 10M videos and their text, from which image datasets can also be extracted. For vision or speech, WebDatasets provides a higg--performance data streaming tool.
3.2 Finetuning Data Catalogs
In this section, we survey sources that catalog finetuning datasets-somettine known more broadly post- training datasets. Finetuning data, differing from pretraining data in that it is curated for supervised learning, is more specialized to the intended inference-time task, and is typically much smaller in scale. Finetuning
hnctpe://uugiinggaccc. co/ wttp///tttt///aabbb///a///////// 2nttpa://github. http:/hhw...cceeeaaaaateeeeeeeaa
6

---

Published in Transactions on Machine Learning Research (12/2024)
datasets are used for variety reasons: to hone specific capabilities, orient the model to certain task format, improve its responses to general instructions, mitigate harmful or unhelpful response patterns, or generally align its responses to human preferences. Developers increasingly vary the types of data annotations and loss objectives, depending on the goal. Notably, after pretraining practitioners commonly use traditional supervised finetuning, DPO (Rafailov et al., 2023) or reinforcement learning objectives from human (or machine) feedback to generated responses (Ouyang et al., 2022; Bai et al., 2022).
Given the thousands of specialized data sources for finetuning, we recommend using data catalogs that provide well documented datasets, to make for an informed selection. HuggingFace Datasets (https: //huggingface cw//oc/////ccccccccccccccc.c.cco /dttastts//ideexx offers the largest and most popular Al community catalog across modalities and tasks (Lhoest et al., 2021a). Many datasets are accompanied by data cards, however Longpre et al. (2023b) show there are frequent omissions and errors, as is the nature of crowdsourced catalogs.
There exist an array of other specialized data catalogs with datasets that may not appear directly on Hug- gingFace. For instance, the NusaCrowd catalog for South East Asian languages (Cahyawijaya et al.), the Masader Arabic data catalogue (Alyafeai et al., 2022), the Al4Barat Indian data catalog (https: //huggingface. /tt.c//////// ai4bharat), as well as the Maskahane NLP (https://github com/masakhane-ion and Zenodo AfricaNLP catalogs fattts:///zennee http///ennoooooooooooooooo communi ttt.a///ttteee/a/a//aa/aaaaaaa offer specialized multilin- gual text and speech resources for the language communities. Recently, the Aya Collection (Singh et al., 2024) offers curated, multilingual text finetuning datasets across 65 languages. For more accurate and comprehensive data documentation, particularly for licensing and provenance, the Data Provenance Initiative publishes tools to search and filter popular HuggingFace finetuning datasets (text, speech, and vision) across a variety of criteria (Longpre et al., 2023b). Liu et al. (2024c) recommend best practices in developing and using synthetic data, which has become increasingly popular in the text domain.
In the context of speech, OpenSLR (http://opens/) org) is large collection of user-contributed datasets for various speech processing tasks. For the task of spoken language identification, VoxLingua107 (Valk & Alumae, 2021) comprises audio scraped from YouTube using various language-specific keywords and, analogously, for speaker iettffiicatio/veriiiittttt VoxCeleb (Nagrani et al., 2017) comprises audio of from 1,000 celebrities.
In the context of vision, there are a few well known data sources for finetuning. ImageNet (Deng et al., 2009) provides the historical framework for which image classification models competed on 1.3M samples with 1000 diverse classes. MS COCO (Lin et al., 2014) provides training data for image detection, segmentation, captioning and retrieval. There exist many options for modern instruction finetuning datasets in the vision domain. A few notable examples include the Multi-Modal, Multilingual Instruction Tuning (M3IT) dataset eLi al., 2023b), comprising 40 datasets, 2.4 million examples over 400 tasks in 80 languages; The Cauldron, comprising 50 vision-language datasets (Laurençon et al., 2024b); and LLaVA Visual Insruct 150k, a set of text-image datasets generated by prompting the GPT-4-0314 API (Liu et al., 2024a). For using web-scraped data, Child and Sexual Abuse Material (CSAM) is an acute concern. Tools such PhotoDNA can be used to help detect and filter for these images, though they may not be completely accurate or comprehensive."
3.3 Review
In this section we critically review the current state of resources for data sourcing, from our survey.
The community would benefit from more accurate and comprehensive licensing, provenance, and creator consent information for existing datasets. Many datasets tend to be under-documented (Bandy & Vincent, 2021; Sambasivan et al., 2021), or erroneously documented (Longpre et al., 2023b), with 65% of HuggingFace dataset licenses either omitted or incorrectly labelled. This is especially true of large data collections that have re-packaged and sometimes re-licensed hundreds of diverse datasets, each with different documentation standards (Longpre et al., 2023a; Sanh et al., 2021). The tools used to discover, select, and verify the dataset properties are unver-developed, especially with respect to concerns of creator consent, copyright infringement, and related terms of use. For creator consent, initial opt-in/opt-out tooling has yet to be widely adopted. While
7

---

Published in Transactions on Machine Learning Research (12/2024)
HuggingFace has integrated Spawning's opt-in database (https://api. www.wnin...iinn................a httpp//aa.waaaaaaaaaaiaaa.a wwt.ppaainggaaaaaa...aa. api), there remains limited creator and developer adoption. For copyright information new datasets and catalogs such as the Data Provenance Initiative (Longpre et al., 2023b), offer more detailed license tracing tools, but their coverage also remains limited. Lastly, synthetic data generation (usually using OpenAI APIs) has expanded significantly for finetuning datasets (Longpre et al., 2023b), but don't always document the limitations on the data use imposed by the APIs. This is further compounded by the fact that there substantial uncertainty about the extent which the terms of service of such platforms bind downstream users. The absence of infrastructure to trace and verify these types of data documentation lead to uninformed data sourcing and use.
The community would benefit from more accurate and comprehensive information sensitive content, such as CSAM and NCII. Prior work highlights the risks of proliferated CSAM and NCTI as a fundamental risk from generative Al models (Kapoor et al., 2024; Lakatos, 2023b; Thiel et al., 2023a). However, significant portions of the AI community have frequently trained on large-scale datasets that contain this sensitive content without widespread awareness- greater focus on generative, multimodal, and more memorization-prom large models, there a greater need for resources to help identify and filter for sensitive content.
as LAION-5B (Birhane et al., 2021; David, 2023). With a
Data & modeling have focused predominantly on easy-to-scale data formats, neglecting other formats. A prominent focus of multimodal modeling has been image-to-caption and caption-to-image tasks. The ease of sourcing and scaling these caption datasets to tens of billions has enabled these tasks to progress. However, this has neglected more complex and useful reasoning tasks, that may require text and images with different relationships, interleaved in different ways. In the context of speech, many automatic speech recognition datasets comprise only of read speech (e.g. as collection involved soliciting crowd-sourced participants to read various text stimuli), which is vastly different from informal, multt-speaker conversations which are the "primordial home of human language" (Dingemanse Liesenfeld, 2022).
There is a scarcity of realistic training tasks, and naturalistic observations. Many open academic datasets are developed for niche, even artificial purposes (e.g. academic visual question datasets that are often detached from real world cases). To collect realistic use cases, however, requires access volunteered user logs from real products and services. Some attempts have been made to do this, such as WildChat (Zhao et al., 2023b), however their participation may still skew to superficial distribution. Tools and even policy to scalably source real data, while preserving privacy, is critical to sourcing grounded and relevant training data.
8

---

Published in Transactions on Machine Learning Research (12/2024)
4
Data Preparation
Data Preparation Best Practices
Tools for searching and analysing can help developers better understand their data, and there- fore understand how their model will behave; important, but often overlooked, step of model development.
Data cleaning and filtering have immense impact on the model characteristics, though there is not one size fits all recommendation. The references provide filtering suggestions based on the application and communities the model is intended to serve.
When training model on data from multiple sources hiatee//dmmiinns/ the quantity of data seen from each domain (data mixing) can have significant impact on downstream performance. It is common practice to upweight domains of "high-quality" data; data that is known to be written by humans and has likely gone through an editing process such as Wikipedia and books. However, data mixing an active area of research and best practices are still being developed.
Removing duplicated data can reduce undesirable memorization and can improve training effi- ciency.
is important to carefully decontaminate training datasets by removing data from evaluation benchmarks, so their capabilities can be precisely understood.
4.1 Data Search, Analysis, and Exploration
A critical step to understanding a dataset to explore and analyze what it contains. In particular, exploring training datasets with search and analysis tools can help practitioners develop nuanced intuition for what exists in the data, and therefore can help to predict what behaviors the model will exhibit.
Tools for search, analysis, and exploration can take few forms. Some tools are aimed at understanding the high-level statistics of dataset such as the length of inputs, frequency of specific n-grams, the languages in the corpus, possible biases, or the existence of undesirable content. For example, tools such WIMBD (Elazar et al., 2023) and Infini-gram (Liu et al., 2024b) allow users to perform n-gram searches through commonly used pretraining datasets, and provide starting points for building search index over any arbitrary dataset. The ROOTS search tool (Piktus et al., 2023) additionally allows users to search with fuzzy n-grams over the ROOTS corpus, and similarly, the clip-retrieval tool (Beaumont, 2022) allows users to search for nearest neighbor images and text from a multimodal corpus (e.g. LAION-5B (Schuhmann al., 2022). Furthermore, the HuggingFace Data measurements tool gives users access statistics such as the most common n-grams, lengths of data points, and distribution over labels for number of pretraining and finetuning datasets. Yet more tools exist to explore the data manually and including the Data Provenance Explorer (Longpre et al., 2023b) for text datasets, Google's Know Your Data tool 7 for vision datasets and NVIDIA's Speech Data Explorer for speech datasets.
In most cases, it is highly recommended to explore datasets from the perspective of high-level statistics well as getting to know the data by looking at individual data points. For instance, text data can have a wide distribution of lengths, topics, tones, formats, licenses, and even diction, and understanding each of these dimensions will require a different tool. We recommend that developers use the many available tools search and analyze their datasets.
9nttps::/docs.nniidiaa

---

Published in Transactions on Machine Learning Research (12/2024)
4.2 Data Cleaning, Filtering, and Mixing
Once the contents of a dataset are understood, the next step is to clean and filter the dataset adjust the dataset's distribution towards desirable content. Filtering and cleaning do so by removing unwanted data from the dataset. They can improve training efficiency as well ensure that data has desirable properties, including: high information content, desired languages, low toxicity, and minimal personally identifiable information. Data mixing is another important component of data preparation, where the mixture proportions of data domains (e.g. scientific articles, GitHub, and books) have been shown to dramatically affect downstream performance (Gao et al., 2020; Xie et al., 2023a; Albalak et al., 2023).
A first step for filtering text data is by language, where there is a plethora of tools including langdetect ", cld3 10, OpenLID 11 (Burchell et al., 2023), GlotLID 12 (Kargaran et al., 2023), and FastText 13 (Grave et al., 2018). The majority of modern language identification methods have been built on top of the FastText model used in the CCNet pipeline (Wenzek et al., 2020). In addition filtering by language, datasets are commonly cleaned using heuristics (e.g. remove documents with fewer than 5 words, or remove lines that start with "siin-in")) which have been implemented through number of different tools including the Dolma toolkit 14 (Soldaini et al., 2023), Lilac 15 and DataTrove 16 (Peeedo et al., 2024), as well as DataComp 17 (Gadre et al., 2023) for image-text pairs. While heuristic filtering methods can remove significant quantities of data, they can be brittle. Model-based methods (e.g. remove data which is dissimilar to a known "high-quality" corpus or possibly toxic content) such as DSIR 18 (Xie et al., 2023b) and Detoxify 19 (Hanu & Unitary, 2020) can be used as additional filtering that allow for much more flexibilitty than heuristics. In addition to cleaning and filtering, mixing another component of dataset design which requires careful consideration. Many dataset mixing ratios have been determined by heuristics and human judgement. For example, the Pile (Gao et al., 2020) and Llama (Touvron et al., 2023) upweight domains that have likely gone through an editing process, such as books and Wikipedia articles. Tools for automated data mixing are, as of writing, very limited. However, academic research projects such as DoReMi 20 (Xie et al., 2023a) and Online Data Mixing 21 (Albalak et al., 2023) provide GitHub repositories that can be repurposed for new datasets.
Cleaning, filtering, and mixing are crucial components of designing an appropriate dataset, but due to the vast space of possible filters and downstream uses of ML models there is no one-size-fits-al recommendation. Practitioners looking to clean and filter their datasets should first use the search, analysis, and exploration tools to determine how to design appropriate filters, and iteratively improve the filtering. For more details and recommendations cleaning, filtering, and mixing, the recent survey by Albalak et al. (2024).
4.3
Data Deduplication
Data deduplication is an important preprocessing step where duplicated documents, or chunks within a document, are removed from the dataset. Removing duplicates can reduce the likelihood of memorizing undesirable pieces of information such as boilerplate text, copyrighted data, and personally identifiable information. Additionally, removing duplicated data improves training efficiency by reducing the total dataset size.
Deduplication generally relies on one of four methods: URL matching, hashing methods, string metrics, model representations, which can classify data as either exact or fuzzy matches. We suggest using the
phttpee//gttabb.cco//ogg//////
"htteer//gstaubbcc/////eeee///l
"httpe://ww.ddataccoppaaa/
Phttaar//itthho.ccmmmllaaaaaaa
10

---

Published in Transactions on Machine Learning Research (12/2024)
deduplication methods included in DataTrove 2 (Penedo et al., 2024), Google's deddupicate-text--taasee library 23 (Tee et al., 2022), or the Dolma toolkit 24 (Soldaini et al., 2023) for their ease of use. In practice, multiple steps of deduplication are performed. First, simple deduplication can be performed (e.g. URL-based filtering), followed by more complicated hashing- and model-based methods. Practitioners should always determine whether duplicated data will harm or help the model for their use case. While memorization is commonly cast as bad thing in machine learning, it can be positive such as when a model "memorizes" the answer factual question (Biderman et al., 2024a).
4.4 Data Decontamination
Data decontamination is the process of removing evaluation data from the training dataset. This important step in data preprocessing ensures the integrity of model evaluation, ensuring that metrics are reliable and not misleading.
Prior to training model on a dataset, it is important to decontaminate that dataset from the desired evaluation datasets. BigCode 25 and Carper AI 26 both implement contamination detection through the use of MSHH which can be used to detect contamination prior to training model. One concern with some modern models is that the model developers may not disclose their training data, so methods and tools have been developed to determine whether model was trained on a specific dataset. For example canary strings, unique sequences of characters, can be included in training datasets, and Jagielski (2023) explain how to interpret canary exposure, which can identify whether model was trained the specified dataset. One example of canary strings can be found in the BIG-bench dataset. 27 In addition to canary strings, Shi et al. (2023) propose Min-K% probability, method for finding possible contamination. 28
One important note for practitioners who are performing decontamination prior to training is to see data deduplication methods for inspiration. For example, data deduplication methods that use exact matching (e.g. Bloom filters from Dolma) are also good candidates for decontamination.
4.5 Data Auditing
Auditing datasets is an essential component of dataset design. You should always spend a substantial amount of time reading through your dataset, ideally at many stages the dataset design process. Many datasets have problems specifically because the authors did not do sufficient auditing before releasing them. The tools outlined in the data search, analysis, & exploration section ae typically sufficient to track the evolution of dataset it's being created. However, there also tools that can be used to audit previously created datasets.
The Data Provenance Initiative 29 (Longpre et al., 2023b) is a good resource that documents the source, license, creator, and other metadata for over 1,800 text finetuning datasets. The Have I Been Trained? 30 tool can assist in finding and detecting data within LAION datasets. See the blog post by Jernite (2023) for more details on auditing datasets.
4.6 Review
In this section we consider, evaluate, and critically review the current state of resources for data preparation.
Phttpe:/aaw.ddtaaprvveaana
Phttps:/haveiieeetrraiii
11

---

Published in Transactions on Machine Learning Research (12/2024)
The community stands to benefit greatly from increased efforts open-source data exploration tools. As we've discussed in this section, tools for data exploration and analysis are a crucial component of the iterative process of creating a dataset, however, the openly available tools for exploring data are limited. Specifically, the existing tools can search for n-grams, show random samples from the dataset, and return high-level statistics of the dataset. However, there are many additional features that may be useful. For example, after searching for an n-gram corpus, it may be helpful to see the containing documents to understand when this n-gram occurs and what the surrounding context is.
Open-source data exploration tools allow for retrospective analysis of datasets. Additionally, improving the functionality and ease-of-use for data exploration tools can help to retrospectively analyze existing datasets. For example, these tools can be used to find potential issues such as biases, illegal or copyrighted content, and personally identifiable information, as well as ensure that they contain the advertised content (e.g. alignment datasets should contain safe text).
Consolidating on a standardized format for data storage and processing will give developers more time to focus on developing infrastructure. Furthermore, many of the data preparation tools and methods presented here exist in separate one-off repositories. This has led limited open-sourced efforts on large-scale infrastructure. Conglomerating around limited number of data formats can reduce friction by allowing developers to make assumptions on the format of data, thus enabling developers to focus efforts on developing scalable tooling for data preparation. For example, the use of Apache Arrow in HuggingFace Datasets (Lhoest et al., 2021b) and Numpy's 's memmap in GPT-NeoX (Andonian et al., 2021) has reduced the effort required for developers to develop memory-efficiet data loading, allowing for developers to focus on building scalable tooling. Similarly, using a single data format, such as that from Dolma (Soldaini et al., 2024), can allow developers to focus on building better large-scale data infrastructure, and spend less time writing complicated code that considers multiple data formats.
Taking a fine-grained view on "high-quality" data. Referring to data as "high-quality" was originally used in the context of pretraining data, but has since become more widely adopted without a clear definition, leading to significant ambiguity in the community. In the context of cleaning and filtering web data, high- quality has referred to data that is known to have been written by humans, and has likely gone through an editing process, leading to the development of quality filters which aim to find data most similar to domains such as books or Wikipedia (Brown et al., 2020; Chowdhery et al., 2023). However, the exact definition of quality has since expanded to pretraining domains beyond web data (e.g. code, reasoning), and the phrase has been adopted in other training regimes such as preference fine-tuning. The field of foundation model development does not currently have a clear definition of what data leads to high-quality models, and the development of such definitions is high-impact direction of research and engineering. While research on data quality for pretraining may be out of reach for smaller institutions and individual researchers, research on data quality for specific downstream use cases (e.g. code, reasoning, math) is more feasible.
For this reason, we advocate for research on data quality to be contextualized within a specific domain or set of evaluations. This will not only allow for research to progress in parallel across many groups and institutions, but we believe will also lead to definitions of quality that are much more concrete, precise, and definitive. Furthermore, the development of clear definitions lowers the barrier for creating tools that can find additional "high-quality" data.
Developing data-centric benchmarks can catalyze progress. Data-centric research is currently being done across such a wide variety of settings, and with varying goals, that it has become nearly impossible to compare methods (Albalak et al., 2024). Some recent benchmarking works have tried to address this by providing a fixed model training setup, and requiring competitors to improve the data for training. Specifically, DataComp (Gadre et al., 2023) provides a data-centric benchmark focused on image-text pairs, DataPerf (Mazumder et al., 2023) provide benchmarks for 4 settings: vision, speech,, debugging, and language. Additionally for the language domain, FETA (Albalak et al., 2022) provides a benchmark for few-shot task transfer, and the Loose track from La (Warstadt et al., 2023) provides researchers benchmark for better data selection. However, due to the large number of training settings and domains of interest, there are wide
12

---

Published in Transactions on Machine Learning Research (12/2024)
variety of additional benchmarks and challenges that would be useful for the community (e.g. pretraining, instruction tuning, alignment, task-specific fine-tuning). Creating new data-centric benchmarks will not only allow for direct comparison between methods, improving our understanding of the data preparation methods, but also lowers the bar for entry into the field by creating an easy-to-use infrastructure, enabling increased progress.
Data preparation tools should consider not only English-centric data, but non-English and low-resource languages. While some data preparation tools may work out-of-the-box for low-resource languages (e.g. n-gram search), others may require more thought and effort (e.g. heuristic filtering). It is particularly important to include native speakers for low-resource languages throughout the data preparation process.
5 Data Documentation and Release
Documentation Best Practices
Data documentation is essential for reproducibility, avoiding misuse, and helping downstream users build constructively on prior work.
We recommend to start the documentation process early, as data is collected and processed.
For datasets with multiple stakeholders or derived from community efforts, it is important to be proactive in decision-making about access, licenses, and stewardship.
5.1 Data Documentation
When releasing new data resources with model, it important to thoroughly document the data (Bender & Friedman, 2018; Holland et al., 2020; Gebru et al., 2021; Bommasani et al., 2024). Documentation allows users to understand its intended uses, legal restrictions, attribution, relevant contents, privacy concerns, and other limitations. An example of how to describe and document data governance decisions can be found in the BLOOM project's report (Jernite et al., 2022). The StackV2 (Lozhkov et al., 2024) is another example of a carefully curated and well documented dataset. Data documentation can also be a way to empower model trainers and downstream users of AI systems who may be unaware of undesirable properties (David, 2023). While many data documentation standards have been proposed, their adoption has been uneven, when crowdsourced, with Hugging Face Datasets, they may contain errors and omissions (Lhoest et al., 2021a; Longpre et al., 2023b).
common for datasets to be widely used by practitioners
5.2 Data Governance
Releasing all datasets involved in the development of Foundation Model, including pretraining, fine-tuning, and evaluation data, can facilitate external scrutiny and support further research. However, releasing and hosting the data as it was used may not always be option, especially when includes data with external rights-holders; e.g., when data subjects' privacy, intellectual property, other rights need to be taken into account. Proper data governance practices can be required at the curation and release stages to account for these rights.
In some jurisdictions, projects may be required to start with a Data Management Plan that requires developers to ensure that the data collection has sufficient legal basis, follows principles of data minimization, and allows data subject to have sufficient visibility into and control over their representation in a dataset (CNIL resource sheet). Data curation steps to that end can include respecting opt-out preference signals (Spawning, HavelBeenTrainedd) or applying pseudonymization or PII redaction (BigCode Governance card).
Once a dataset released, it can be made available either broadly or with access control based on research needs (ROOTS, BigCode PII training dataset). Developers can also enable data subjects ask for removal from the hosted version of the dataset by providing a contact address (OSCAR, PAraCrawl), possibly
13

---

Published in Transactions on Machine Learning Research (12/2024)
complemented by membership test to check whether their data is included (Stack data portraits) or automated process (BigCode, AmlinTheStack).
5.3 Review
Better data documentation of existing and new datasets still needed. Most datasets still lack appropriate documentation. While data documentation tools exist they are underutilized at present. Longpre et al. (2024c) illustrate challenges documenting provenance, consent, and authenticity of datasets at scale, and driving adoption of rigorous data standards.
Datasheets and Data Cards are just start. While data documentation tools, such as datasheets and data cards are very useful, preferred to begin projects with Data Management Plan and ensure that data collection is designed thoroughly. Considerations should include: the legal basis for collecting data, ensuring that collection is limited to the necessary data, transparency, respecting opt-out preferences and redaction of Pll.
6
Model Training
Model Training Best Practices
The foundation model life-cycle consists of several stages of training, broadly separated into pre- tre-training and fine-tuning.
Decisions made by developers at any stage of training can have outsized effects on the field and the model's positive and negative impacts, especially decisions made by well-resourced developers during the pre-training stage.
Developers should be thoughtful about the effects of train-time decisions and be aware of the trade-offs and potential downstream effects prior to training.
Due to the large economic and environmental costs incurred during model training, making appropriate use of training best practices and efficiency techniques is important order not waste computational or energy resources needlessly.
Thousands of tools and resources have been developed for model training. While it is not feasible to comprehensively list them, we focus our efforts a subset of tools that are well documented, emphasize computational efficiency, or educational resources. For a more comprehensive list of training tools, we point the reader to surveys, such as the survey of foundation model training and serving systems (Zhou et al., 2024), the survey efficient federated learning methods (Woisetschläger et al., 2024), or this "comprehensive" survey of foundation models (Zhou et al., 2023), which can used to trace their training setup.
Foundation models are by their design frequently reused and applied for numerous diverse downstream uses. This takes the form of a multi-stage training process throughout models' lifestyles, starting with training strong base from scratch ("pretraining") and followed by further refinement adaptation to new cases (" ("fine-tuning") For this reason, all stages of training must be done with care: especially for pretrained models or models that will otherwise be deployed widely or used as base for further extensions, the decisions made by model trainers at train-time can have outsized impacts on the model's characteristics, both positive and negative, or on the field as a whole.
Here, we cover a selection of existing resources for model training. We include frequently-used codebases for model training that may be useful entry points for new developers to the field, but note that we cannot cover existing options. We additionally briefly discuss resources where practitioners can learn about improving the resource-efficiency of their training, as well educational resources for learning about model training.
6.1 Pretraining
Model pretraining requires by far the largest amount of resources computationally during model training.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Therefore, practitioners should consider using already-optimized codebases, especially in the pretraining phase, to ensure effective use of computational resources, capital, power, and effort. Existing open-source codebases targeted at foundation model pretraining can make pretraining significantly more accessible to new practitioners and help accumulate techniques for efficiency in model training.
Various codebases may be optimal depending on the scale of model or number of devices used for pre- training. Some codebases seek to be performant while also being lightweight (OpenLM (ELMMGGune (Gururangan et al., 2023), Nanotron (nan, 2024)) for effective training at medium scales and up, while others aim for maximal performance massive scales such as GPT-NeoX (Andonian et al., 2021), Regattro--DeepSped (Smith et al., 2022b), and Megatron-LM, (Shoeybi et al., 2020) and have been used in practice to train language models across thousands of GPUs. Depending good scalable codebases exist for training. For example, OpenCLIP (Ilharco et al., 2021) can be used for training CLIP text + vision models, Timm (Wightman, 2019) supports a variety of standard vision model architectures and training scripts, and UniLM supports the training of interleaved text-and-image models similar to Kosmos-2 (Peng et al., 2023). For modalíties other than text or vision, tooling exists (Lhotse for audio data processing and data loading (Zelasko et al., 2021), Stable Audio Tools (AI) for training audio generation models) but is less standardized. Lastly, Karamcheti et al. (2024); McKinzie et al. (2024) explore lots of design choices for multimodal models, including training, fine-tuning, image processing strategies, and data mixing.
the particular vision application or model architecture,
We emphasize the utility and importance of adopting existing codebases for pretraining, due to the difficulty of debugging and detecting errors in large-scale distributed systems as encountered in pretraining. Existing scalable codebases drastically reduce the chances of silent failures or correctness issues lowering the end model quality.
6.2 Fine-tuning
Fine-tuning, other types of adaptation performed on foundation models after pretraining, are an equally important and complex step in model development and have the ability to significantly steer the behaviors and characteristics of the end model. Fine-tuned models are also more frequently deployed than base models, making their safety and usability very important. Here we discuss a subset of finetuning resources that are well documented, flexible, and some of which cater to computational efficiency.
While fine-tuning is significantly less resource-intensive than pretraining, there are still many relevant considerations for practitioners. Use of widely adopted tools such as libraries designed for fine-tuning (Axolotl AlA-ccess----olllctivvvv trIX (Havrilla et al., 2023), Levanter (https //gi thub. com/stanford, crfm/levanter)) can ensure greater ecosystem compatibility of resulting models, or reduce the barrier to experimentation by abstracting away common pitfalls providing guidance effective hyperparameters. Similarly, the use of techniques such QLoRA (Dettmers et al., 2023) or other popular pfrfmeter-efficient fine-tuning approaches and libraries (peft (Mangrulkar et al., 2022). Otter, LLaMA-Adapter, LLaVA (Li et al., 2023a; Zhang et al., 2023a; Liu et al., 2023a)) can allow for fine-tuning for lower cost or on more accessible hardware.
6.3 Efficiency and Resource Allocation
Knowledge of training best practices and efficiency techniques can reduce costs to train desired model significantly. Beyond systems-level optimizations and approaches to increase efficiency, the most important technique is determining the most efficient allocation of resources, such as allocating compute between model size and dataset size for a given budget for the best results. The most common approach to solving this problem is to apply scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023b), a common tool for cheaply extrapolating findings across scales of cost in order to make decisions based on smaller-scale experiments for a final model training run.
Additionally, practitioners seeking to embrace an open approach to model development should consider how their decisions when training foundation model may have impacts long after that model's creation
15

---

Published in Transactions on Machine Learning Research (12/2024)
and release. For instance, model that is released openly but is too computationally demanding to be run consumer-grade hardware will be limited in its impact on the field, or model trained to mínimize training compute but not minimize inference cost may result in a greater environmental impact than spending more training compute in the first place for a theaper-to-infer model (Hoffmann et al., 2022; Touvron et al., 2023). Practitioners should thus be aware of potential second-order effects of their model releases and training choices.
6.4 Educational Resources
Training models at any scale can be quite daunting to newer practitioners. However, there are options available for learning about how to train and foundation models.
Resources which themselves collate and provide reading lists or recommendations for learning about large- scale ML training ATleutherAT Cookbook (Anthony et al., 2024), ML Engineering Open Book (Bekman)) can be especially useful place to begin. Additionally, minimal codebases created as educational examples such as NanoGPT (Karpathy, 2023), or other blog posts detailing fundamental concepts in training or running foundation models may be useful (Chen, 2022; Anthony et al., 2023). We recommend that practitioners review these resources and use them to guide further reading about model training and usage.
6.5 Recommendations
Regardless of the stage or cost of training being performed, we urge developers to carefully consider the impacts of their training choices, and how they can be improved, as we have described. We hope the linked resources will be a helpful jumping-off point.
We additionally encourage practitioners to use the codebases linked as foundation for their work, to avoid "reinventing the wheel" for every new project. Unifying and collectively improving existing codebases, tooling, and commonly used standards around model training, as with tooling around other parts of the foundation model development process, can allow for the entire community to benefit from others' effort via using codebases that have been well-tested for correctness and scalability. This can leverage the unique advantages of open model development via strength in numbers.
6.6 Review
More resources, especially non-English ones, for lowering the barrier to entry for less technical developers are needed. Current tooling requires at minimum a technically knowledgeable user or proficient programmer, and further may have lacking documentation or be high in complexity. Openly available tools for training, especially fine-tuning, models that do not presume familiarity with training methods or even programming would have the potential to allow many more users not formally educated in computer science to customize and build models suited to their use cases, such as for lower-resource languages.
More standardized and centralized resources, especially cross-modality, should be focused on in the future. Many current resources are centered around language model training. However, resources for other modalities are often more scattered or rely on custom architectures and implementations. Exploring topics such as multimodal fine-tuning best practices, and multimodal preference learning, is an important future area. For instance, there is lack of efficient tooling for data loading for multimodal training. While "ebDataset offers a resource for images, there are fewer options for videos, which can cause the GPU to stay idling, waiting for data. Future work should look to unify these techniques and modalities in the same tools and infrastructure to enable more and easier investigation or transfer of techniques that work on language models to the rest of the field. Even within the text modality, an interoperable ecosystem with shared, well-tested code building blocks and standards should be pursued, to provide strong base for further extension and the pooling of developer efforts.
Other options for massively collaborative development should be pursued. To allow for better pooling of resources, both over time and across the community, methods for more collaborative training are an
3/nttps:/giihhbb Phteppss/ggkhuo..ccceeatttteee/s http:/ccc..ebaaaeeeee/aaaatttttt
16

---

Published in Transactions on Machine Learning Research (12/2024)
important future frontier. Some of these methods are in their early stages already being provided by tools such as MergeKit (Goddard et al., 2024) and being explored by the research community (Matena & Raffel, 2022; Don-Yehiya et al., 2023; Stoica et al., 2024; Yadav et al., 2023). By enabling better re-use of existing artifacts or allowing a greater number of collaborators to contribute to building a model together, these methods could potentially make model training as parallelizable and reusable as work on data improvement, and the merging of efforts and compute expended could allow wider range of contributors steer model development.
7 Environmental Impact
Environmental Impact Best Practices
Training and deploying AI models impacts the environment in several ways, from the rare earth minerals used for manufacturing GPUs to the water used for cooling datacenters and the greenhouse gasses (GHG) emitted by generating the energy needed power training and inference.
Developers should report energy consumption and carbon emissions separately to enable an apples-to-apples comparisons of models trained using different energy sources.
It is important to estimate and report the environmental impact not just of the final training run, but also the many experiments, evaluation, and expected downstream uses.
It is recommended, especially for major model releases, to measure and report their environmental impact, such as carbon footprint, via mechanisms such as model cards (see Section 5).
7.1 Estimating Environmental Impact
The environmental impact of Al of increasing concern (Schwartz et al., 2020). Estimating the environmental impact of model development is a challenging task due to the number of relevant, hard to compute, and often unrecorded variables. For instance, to estimate the Life Cycle Assessment (LCA) of model development (Klöpffer, 1997), variables include: model size, architecture, duration of training, number of training runs, storing and transferring data, hardware manufacturing, the specific type and setup of the hardware, network configuration, the geographic location of the data center, the carbon intensity of the energy grid, the power usage effectiveness (PUE) of cooling, overhead, and the broader data center infrastructure, as well as a similar set of questions for various are also infeasible to precisely estimate, given for instance, that Nvidia does not disclose the carbon footprint of its GPUs (Luccioni et al., 2023). Another challenge includes the variety of relevant environmental outcome measures, from CO2 emissions, to energy footprint, or water use.
ppployment/inferenn
settings (Patterson al., 2021). Many of these variables
Existing tools to measure environmental impact rely series of assumptions and estimates based on the available information. Perhaps the most accurate tools are those built into cloud services, which are able to trace the hardware configurations and geographical locations of data centers but may not otherwise be publicly available to users. The Azure Emissions Impact Dashboard (https: //www www.mmoottt.ommmmmmmmmmoo.ww...t com/en- AWS carbon footprint tool (https //wws. . and Google Cloud Carbon Foot- are three such sys- tems, available only when using their cloud services directly. Independent systems, such as Carbontracker (Anthony et al., 2020), CodeCarbon (Schmidt et al., 2021), the Experimental Impact Tracker (Henderson et al., 2020) offer basic estimate modeling and reporting, based on limited input information. Similarly, Li et al. (2023c) provides an estimate tool for the water usage footprint of language model training and deployment. ML CO2 Impact (https: h/mpco////cc.....// /tt///ol..gttubbb... sl//mmact//) improves these repositories with a wrapper interface for easier use. However, these services are forced to trade-off between ease of use and accuracy, as significant input information required obtain precise results, which imposes burden on users and widespread adoption.
amazon.
print measurement system (https://cloud) popple/cc///aaaatooooooooooorooa
17

---

Published in Transactions on Machine Learning Research (12/2024)
7.2 Effective use of resources
Several decisions made prior to model training can have significant impacts on the upstream and downstream environmental impact of a given model. Empirical scaling laws can be used to find the best allocation of resources. Kaplan et al. (2020); Hoffmann et al. (2022) estimate the optimal model size and training duration, given a training compute budget. And Aghajanyan et al. (2023) investigates the equivalent efficient compute allocation for multi-modal settings. When working with text training data that is constrained, recent work explores how to allocate compute efficiently (Muennighoff et al., 2023b). For models frequently used downstream, it is important to consider the inference footprint and inference cost during model creation (Gadre et al., 2024b), to minimize the environmental impact of inference. For further resources and discussion, see 6.3.
7.3 Review
In this section we critically review the current state of resources for environmental impact analysis. First, we note dearth of transparency, from hardware manufacturers, data centers, and corporate model developers stymies environmental impact estimates, and particularly product comparisons for users. Lastly, scaling laws research needs to expand to cover newer multi-modal modeling efforts, well as provide intuitive tooling for open developers to adopt.
Transparency from consumer-level API providers, into query-level energy and environmental usage measures. Currently, some of the most widely adopted generative AI services, including the APIs and playgrounds from OpenAl, Google, Anthropic, Inflection, Midjourney, and others, do not expose any information into the environmental footprint of using their models (Bommasani et al., 2023). As these systems dominate consumer market usage (Korinek Vipra, 2023), this leaves a wide gap in our knowledge of net effects. The scientific community relies largely on assumptions and ballpark estimates, both for training and inference impact.
Transparency from hardware makers and data centers, into fine-grained energy and environmental usage measures. Upstream of AI developers, the data centers and hardware makers expose limited information about environmental and energy measures (Luccioni et al., 2023). These metrics would enable more accurate and real-time estimates of compute, for closed and open developers.
"Energy Star" standards for non-technical users to fairly compare Al services. Data center, hardware, and developer transparency are a precursor to accurate estimates and, more importantly, environmental footprint competition between corporate services. Currently, these services compete on model quality and price, but not on environmental impact-a property that many consumers are likely to care about if given the option. "Energy Star" standards, from other industries, allow competitors claim equal or better services, less energy expenditure (Brown et al., 2002). These sorts of apples-to-apples comparisons may be necessary to inhibit negative environmental consequences from AT.
Scaling laws research currently lacks empirical evidence for the new wave of multi-modal models, and intuitive user interfaces for new developers. Scaling laws research has focused predominantly on text (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023b; Gadre et al., 2024b), with limited work for multi-modal foundation models (Aghajanyan et al., 2023). As developers increasingly pursue image, video, and speech models (both in input and output), such as Sora (Brooks et al., 2024), Stable Video Diffusion (Blattmann et al., 2023), Claude (Anthropic, 2024b), and Whisper (Radford et al., 2023), the efficient scaling laws are currently under-investigated. Secondly, while scaling law research investigates fundamental questions, can be presented in unapproachable, complex ways. The ecosystem lacks tools for less technical developers to heed efficient compute estimates, such as a plug-and-play interface.
18

---

Published in Transactions on Machine Learning Research (12/2024)
8
Model Evaluation
Model Evaluation Best Practices
Model evaluation is an essential component of machine learning research. However many machine learning papers use evaluations that are not reproducible or comparable to other work.
One of the biggest causes of irreproduciiility is failure to report prompts and other essential components of evaluation protocols. This would not be problem if researchers released evaluation code and exact prompts, but many prominent labs (OpenAI, Anthropic, Meta) have not done so for model releases. When using evaluation results from a paper that does not release its evaluation code, reproduce the evaluations using public codebase.
Examples of high-quality documentation practices for model evaluations can be found in Brown et al. (2020) (for bespoke evaluations) and Black et al. (2022); Scao et al. (2022); Biderman al. (2023) (for evaluation using public codebase).
Expect a released model to be used in unexpected ways. Accordingly, try to evaluate the model on benchmarks that are most related its prescribed use case, but also its failure modes or potential misuses.
All evaluations come with limitations. Be careful to assess and communicate these limitations when reporting results, to avoid overconfidence in model capabilities.
8.1 Capabilities
Many modern foundation models are released with general conversational abilities, such that their use cases are poorly specified and open-ended. This poses significant challenges evaluation benchmarks that are unable to critically evaluate many tasks, applications, and risks fairly systematically. As result, it is important to carefully scope the original intentions for the model, and to tie evaluations to those intentions. Even then, the most relevant evaluation benchmarks may not align with real use, and so developers should qualify their results, and carefully supplement them with data from real user/human evaluation settings where feasible.
For language models, common capabilities benchmarks include those that evaluate models on narrow tasks such as software engineering (Jimenez et al., 2023), topic classification (Adelani et al., 2023), and explaining code (Muennighoff al., 2023a). More comprehensive evaluation suites, such as the Language Model Evaluation Harness (Gao et al., 2023) and HELM (Liang et al., 2023), are also common. Leaderboards like LMSys Chatbot Arena (Zheng et al., 2023) offer another type of capability evaluation based on human feedback.
There are far fewer capability evaluations for other modalities. Comprehensive evaluation suites exist for vision models as well (Lee et al., 2023b; Awadalla et al., 2023), but they are relatively less well developed. There are also number of common benchmarks for evaluating vision models on a large number of tasks (Gadre et al., 2023; Liu et al., 2023b; Fu et al., 2023). Evaluations for speech models' capabilities are still nascent, with the OpenASR Leaderboard," 32 which ranks models based on their Word Error Rate and Real- Time Factor, and the Edinburgh International Accents of English Corpus (Sanabria al., 2023) as leading examples.
The cheatsheet includes common benchmarks as of December 2023, but we caution that each comes with substantial limitations. For instance, many benchmarks based on multiple choice exams are not indicative of real user questions, and can be gamed with pseudo-data contamination. Additionally, while leaderboards are exceedingly popular, model responses are often scored by other models, which have implicit biases to model responses that are longer, and look similar to their own (Dubois et al., 2023).
#httpss//hnggiiggaacc.o http:///wg..gaaae..oo//////////a http://ppeeeerredeeeerarrdddd
leaderboard
19

---

Published in Transactions on Machine Learning Research (12/2024)
8.2 Harm & Hazard Taxonomies
Taxonomies provide a way of categorising, defining and understanding risks and hazards created through the use and deployment of AT systems. Some taxonomies focus primarily on the types of interactions with models that create risk of harm (often called "hazards") whereas others focus on the negative effects that they lead to (often called "harms"). Some taxonomies focus on existing issues, such as models that create hate speech or child abuse material, as well as more intangible or indirect forms of harm, such as the risk that models perpetuate biases and stereotypes, and misrepresent social groups. Other taxonomies are focused on the longer-term threats posed by more sophisticated models, such as illtra-ersoonalise disinformation, cybersecurity threats, and military use (Brundage et al., 2018). Some work has also focused on categorising catastrophic "existential" risks presented by Artificial General Intelligence, such as rogue Al agents and Chemical, Biological, Radiological, Nuclear and high-yield Explosive weapons (Carlsmith, 2022; Hendrycks et al., 2023; Bucknall & Dori-Hacohen, 2022). Further, a few taxonomies also assess the available evidence for the risks and hazards, discuss their impact, and offer mitigation strategies (Deng et al., 2023; Kapoor et al., 2024; Klyman, 2024).
Many taxonomies are released with an associated dataset, which can be used to either train models to minimise safety risks or to evaluate those risks. Several datasets have been released as benchmarks (e.g. Wang et al., 2024), which can be used to track progress across the community. We provide a non-exhaustive list of existing taxonomies with datasets that are available open-source. We also note forthcoming work planned by organisations like ML Commons, which aims to standardise assessment of Al safety risks by introducing new benchmark, comprising taxonomy and dataset.
1. TrustLLM is a benchmark that covers six dimensions in English, including truthfulness, safety, fairness, robustness, privacy, and machine ethics (Sun et al., 2024). The benchmark comprises over 30 datasets from existing research. In the paper, they test 16 open-source and proprietary models, and identify critical safety weaknesses.
2. SafetyBench is a benchmark that covers eight categories of safety, in both English and Chinese (Zhang et al., 2023b). Categories include Offensiveness; Unfairness and Bias; Physical Health; Mental Health; Illegal Activities; Ethics and Morality; and Privacy and Property. Unlike most safety evaluation datasets, SafetyBench comprises multiple choice questions which makes automated evaluation of models far easier. In the paper, they test 25 models and find that GPT-4 consistently performs best.
DecodingTrust is a benchmark that covers eight dimensions of AI safety and trustworthiness in English (Wang et al., 2024). It covers range of safety criteria such as Toxicity; Stereotypes; Adversarial Robustness; Out-of-Disttibution Robustness; Privacy; Machine Ethics; And Fairness. The benchmark has a leaderboard that is hosted on HuggingFace.
4. HarmBench is standardized evaluation framework for automated redteaming of LLMs in English (Mazeika et al., 2024). It covers 18 widely used red teaming methods, such as Persona, stochastic-few shot, PEZ and GBDA. The benchmark has been designed with both seven semantic categories (e.g. Cybercrime, Misinformation and Bioweapons) and four "functional categories" (e.g. Standard behaviours). In the paper, 33 LLMs are tested against HarmBench.
5. BigBench (Srivastava et al., 2023) and HELM (Liang et al., 2023) contain tests that are related to safety, such as tuxicity, bias and truthfulness in BigBench and toxicity, bias, disinformation, copyright infringement and truthfulness in HELM. Both make use of the widely-used RealToxicityPrompt dataset (Gehman et al., 2020a). Terms such as "toxicity" and "offensiveness" have been criticised in some papers for being overly broad and easy to misinterpret (Vidgen et al., 2019), and more recent work has tended to use more fine grained terms.
Individual datasets have also been released that can be used to assess specific safety risks of models, such as SimpleSafety Tests which tests for clear-cut safety problems (Vidgen et al., 2023) and XSTest which tests for model false refusal (Röttger al., 2024).
/tttpps:/nnIcomme wrg/working-ggraeeeeeeor groups/ai-safeety/stt safety/ai--aaftttt
20

---

Published in Transactions on Machine Learning Research (12/2024)
7. SafetyPrompts is website that hosts datasets for evaluating the safety of models 34 It does not aggregate or combine the datasets that it hosts, but has basic review process to check the quality and integrity of the datasets.
Almost all of the taxonomies described here are informed by practitioners' experiences of tackling safety issues in Al models; prior research; and exploratory red teaming. Many have drawn heavily on existing work in social media trust and safety, such as Banko et al. (2020); Dev et al. (2022). They are all top-down in the sense that they define categories of hazard (or harm) and then find, create or curate prompts that match those categories. An alternative way of addressing AI safety is to start from the bottom up and to task red teamers with creating their own categories. Grounded-theory style approaches in linguistics can be used to then standardise the categories and ascribe some structure to the taxonomy. Across all methods of creating taxonomies (and datasets), there is a substantial focus on text-only models and future work should pay more attention to multimodal and non-text based models. Similarly, there is a strong bias towards English language and the Western cultural context. This should also be addressed in future work.
8.3 Risks & Harms
Model developers have used various techniques to address risks and mitigate harms. Broadly, these attempts fall into three categories, roughly ordered by effectiveness.
In-context learning. In-context learning can be used to add instructions to a model's system prompt to steer the model's outputs. For example, OpenAl's DALL-E 3 model included instructions to avoid outputting copyrighted characters In some sense, this the easiest model-level intervention: developers do not need to retrain or fine tune the model, neither do they need to rely on external API calls, such as to third-party content moderation endpoints. However, this comes at a cost: system prompts are easy to jailbreak, and as result, interventions based on in-context learning might be more brittle compared to other guardrails.
Model alignment. One of the most prominent approaches for mitigating risks and harms is aligning models with preference data. This usually involves supervised fine tuning or training reward models that steer the outputs of language models. Popular techniques include reinforcement learning with human (RLHF) and AI (RLAIF) feedback (Bai et al., 2022; Ouyang et al., 2022). To help end users understand the performance of reward models, Lambert et al. (2024) develop a leaderboard for evaluating reward models for various desiderata, including safety. Still, model alignment techniques can be brittle against simple modifications to the model, such as fine tuning (Qi et al., 2023), even when the model is fine tuned using benign data (He et al., 2024).
Guardrails on model inputs and outputs. The two interventions above rely on changing the model behavior via in-context learning or fine tuning. However, guardrails model inputs and outputs that lie outside the model might be more robust intervention for preventing harmful content generation. For example, several model developers provide moderation endpoints that can be used to filter inappropriate user requests or model outputs. These can be general-purposs endpoints that can be modified for filtering and moderation (e.g,, Cohere's classification endpoint modified for toxicity detection or Anthropic's suggested prompt for content moderation as well as endpoints that are specifically built for content moderation (e.g., Perspective API (Lees et al., 2022) and OpenAl's moderation endpoint 38). Google's PaLM and Gemini models allow API users to set thresholds based on the safety likelihood and severity of model outputs?*).
The examples above are all from closed model developers. Recently, several open models have also been proposed for moderating model inputs and outputs. For example, Llama-Guard by Meta (Inan et al., 2023) can be used to filter harmful content. Nvidia's NEMO guardrails allow model providers to add programmatic guardrails to filter content (Rebedea et al., 2023).
Phetps::/saffetyyrooptt com/
3SSee: https://the-
3eSec: https://docs.. www.ee.checccooeeeeeoeeeeeo
aSSec: http://www.oooooeaooc////////sas
3eee http://ww...aalcccccccc//cceeles
htwp:////weeeeee///////////e/eee https://cloud.
21

---

Published in Transactions on Machine Learning Research (12/2024)
8.4 Review
A shift away from evaluating models, and towards evaluating systems. Existing evaluation practices and reporting often focus on probing individual models, in unconstrained settings. However, deployed systems often operate in the context of multiple interactive models, moderation endpoints, sophisticated decoding strategies, and rule-based constraints, that make up a "system". More pointedly, for dual-use systems, the context of use outside the system is what defines the presence of harm: Narayanan & Kapoor (2024) argue that "defenses against misuse must primarily be located outside models". Prior work has emphasized the importance of the system (Dobbe, 2022) and context (Raji & Dobbe, 2023) in diagnosing and resolving Al safety challenges. As a result, efforts to evaluate models alone are inherently limited, both in their findings, and informing effective changes. Safety problems in particular are better addressed by evaluations that consider the context (e.g. attack vector, deployed use setting), as well as the interactions between elements of foundation model system (of which only one is a the model itself). However, these types of evaluations can be difficult in the absence of transparency into the components of deployed systems (Bommasani et al., 2023). Recent work has even shown that independent researchers can face significant obstacles in fairly evaluating proprietary systems (Longpre et al., 2024a).
A shift away from evaluating on static toxicity/safety benchmarks, and towards evaluating on real-world, dynamically changing naturalistic observations, as well as human-interactio studies. Existing widely used model risk and safety evaluations, such as RealToxicityPrompl (Gehman al., 2020b), or bias benchmarks such as BBQ (Parrish et al., 2022), BOLD (Dhamala et al., 2021), and HolisticBias (Smith et al., 2022a), have been effective at testing models' superficial bias tendencies. However, these evaluations are static, not grounded in real-world contexts, and pay no heed to the human-interaction element-i.e -ie. the actual system users. These limitations suggest several dimensions of improvement for future safety benchmarks:
Naturalistic observations Collecting natural interactions with users enable researchers to identify realistic tasks and prompts, to simulate real harms. For instance, WildChat (Zhao et al., 2023b) collects voluntary user interactions with OpenAl systems through proxy interfaces-hhwever these datasets may be heavily skewed by the types of users that adopt it.
Domain expert designed tasks LegalBench (Guha et al., 2024) offers an alternative naturalistic design, whereby evaluations are hand-crafted by the practitioners (in this case, legal professionals) in the field of evaluation. These benchmarks distinguish themselves from existing evaluation suites in their relevance to natural, real-world usage.
Human-interaction studies Most evaluations are non-interactive-tter target the model, without real-world users. This form of analysis can identify model flaws, but falls short of investigating the actual affects, harms, and interaction patterns users. Lee et al. (2023a) proposes framework to evaluate interactive user experience, without which notions of harm and safety are under-developed. Le Ferrand et al. (2022) illustrates the importance of interactive evaluation, particularly on non- Western users. 40
These three evaluation dimensions enable more realistic studies of user interaction with foundation model systems. Collecting naturalistic or interactive data can also provide continuous and dynamic data sources, which are particularly beneficial when there is rapid model development (or the habit of training on prior evaluation sets). More generally, research into dynamic and evolving benchmarks, that test beyond the training set distribution, are an important research direction (Yu et al., 2023; Kiela et al., 2021).
A shift away from reporting evaluations, to releasing reproducible evaluation scripts. There are dozens of choices that affect the results of an evaluation CanationnAAnhiaaaa (Anthropic, 2023). Some choices include, but are not limited to:
Prompt format Results vary dramatically depending on if the input prompts are zero-shot, few-shot, chain-of-thoughh, and also if the prompts were manually refined directly on the test set. Multiple
in this study, Australian Aboriginal use an information retrieval app illuminated false assumptions and expectations the evaluation procedure.
22

---

Published in Transactions on Machine Learning Research (12/2024)
choice benchmarks have both versions with and without the answer choices given in the prompt. (See MMLU as a widely used dataset with inconsistencies in use and standardization."
Decoding strategies The decoding algorithm and its hyperparameter choices, such as the temperature and sampling probabilities, will affect model behavior. For systems, rather than models, responses can be the result of multiple iterations or models, or rely on external tools or sources. Ensembling techniques like self consistency (Wang et al., 2022) also improve performance significantly.
Evaluation metric The choice of evaluation metric can impact the apparent magnitude of differences between models, and even their ranking (Schaeffer et al., 2024).
Human review setup For human preference evaluations, several details affect fair evaluation: whether the response selection is sufficiently model-blind, the attentiveness and expertise the annotators to the given topics, and the chosen rubric. Human preferences can also be skewed by the same factors as model-based evaluations (Hosking et al., 2024; Xu et al., 2023; Wu & Aji, 2023)
Without transparency and reproducibility integrated into the evaluation procedure, the axes of design freedom can allow developers to game results, and prevent fair, apples-to-apples comparisons. For these reasons, only evaluation scripts that are directly executable by third-parties provide verifiable reproducibility. Executable evaluation scripts also allow auditors to unpack the choices made in evaluation. Auditors can quickly experiment with different prompt formats, decoding parameters, and evaluation metrics, to shed light on the scientific veracity of the claims. While evaluation documentation helpful, the required breadth of information inevitably leads to subtle omissions, and even if the information is comprehensive, does not provide verifiable reproducibility. For these reasons, we believe the evidence is clear that Al safety reporting is not sufficiently reliable trustworthy without verifiable execution scripts.
Ground harm and hazard taxonomies in empirical observations. Existing taxonomies of harm often created to cluster existing safety benchmarks, which mostly detached from real observations (Sun et al., 2024; Zhang et al., 2023b; Hendrycks et al., 2023). As the taxonomies of harm can guide practitioners priorities, this could lead to neglected or over-emphasized areas of safety research. It is essential that future harm taxonomies are strongly grounded in empirical or naturalistic observations, through research conducted with real users, rather than hypothetical situations envisioned by researchers.
Extending risks and harms studies to multimodal and highly sensitive attacks. A great deal of research into risks and harms is focused exclusively on text, in English, and on more conservative safety risks such as toxicity and bias (Gehman et al., 2020a; Hartvigsen et al., 2022). However, recent work has emphasized the particular risks of generative image, speech, and video models, being used to create deepfakes, NCII CSAM (Kapoor et al., 2024; Thiel et al., 2023b; Lakatos, 2023a). Other work has illustrated the much greater efficacy of jailbreaks and attacks in non-English languages, as compared to English (Yong et al., 2023). Multimodal jailbreaking is an emerging research area, with some nascent work (Qi et al., 2024; Shayegani et al., 2023; Niu et al., 2024). Research into risks and harms from autonomous weapons systems (AWS) is also highly under-explored, and an emerging risk (Simmons-Eddee et al., 2024; Longpre et al., 2022).
http://eee/eet..ttaarrrrreeee//e http://a.rr...aae/////////////// html
23

---

Published in Transactions on Machine Learning Research (12/2024)
9
Model Release & Monitoring
Model Release Monitoring Best Practices
Release models with accompanying, easy-to-run code for inference, and ideally training and evaluation.
Document models thoroughly to the extent possible. Model documentation is critical to avoiding misuse and harms, as well as enabling developers to effectively build on your work.
Open source is technical term and standard with a widely accepted definition that is maintained by the Open Source Initiative (OSI) (Initiative, 2024). Not all models that are downloadable or that have publicly available weights and datasets are open-source; open-source models are those that are released under license that adheres the OST standard.
The extent to which "responsible use licenses" are legally enforceable is unclear. While licenses that restrict end use of models may prevent commercial entities from engaging in out-of-scope uses, they are better viewed as tools for establishing norms rather than binding contracts.
Choosing the right license for an open-access model can be difficult. Apache 2.0 is the most common open-source license, while responsible Al licenses with use restrictions have seen growing adoption. Consider using one of the available tools for selecting the right open-source license for your model artifacts.
Frameworks for monitoring and shaping model usage have become more prevalent as policymakers have attemmted to constrain certain end uses of foundation models. Several approaches include adverse event reporting, watermarking, and restricting access to models in limited ways. Consider providing guidance to users on how to use your models responsibly and openly stating the norms you hope will shape model use.
9.1 Model Documentation
When models, code or applications are released, whether openly or not, it is important that they are documented thoroughly. Documentation should specify how to use the model, recommended and non- recommended use cases, potential harms, state or justify decisions made during training, and more. Docu- menting models is important not just for responsible development, but also to enable other developers effectively build on model. Models are not nearly useful as artifacts if not properly documented. Model cards (Mitchell et al., 2019) are widely adopted standard for documenting models. Several tools have been developed that support the creation of model 42
9.2 Reproducibility
Code to reproduce results are an important complement to other forms of documentation Kapoor et al. (2023). Releasing assets that reproduce results mean that scientific claims can be verified, and that systems can be interrogated, tested and audited Missing, incomplete, or poorly documented code hinders progress. There are tools that help make model training, inference and evaluation reproducible. Anaconda and Docker make necessary environments and dependencies easier to manage. Google Colab and Jupyter notebooks enable easily shareable code snippets and organized tutorials. The Language Model Evaluation Harness provides framework for prompting and testing generative language models large number of different evaluation tasks (Gao et al., 2023).
9.3 Licensing
Licensing is mechanism creating a legally enforceable agreement that governs of artifacts. Licenses with use restrictions can be used to limit the ability of certain categories of stakeholders to re-use or adapt the
thtpss//tuggingfcccoo///ogg hoode
24

---

Published in Transactions on Machine Learning Research (12/2024)
models (Contractor et al., 2022; Foundation, 2024). The MIT and Apache 2.0 licenses the most commonly used. Responsible AT Licenses, including BigScience's Open RAII,, have seen growing adoption. However these also face criticism around how they pose challenges for well-intentioned actors and because their enforceability remains an open question (Downing, 2023). While RAIL licenses that restrict end use of models may prevent commercial entities from engaging in out-of-scope uses, they are better viewed as tools for establishing norms rather than binding contracts.
9.4 Usage Monitoring
Some open foundation model developers attempt to monitor the usage of their models, whether by water- marking model outputs or gating access the model. The cheatsheet provides resources related usage monitoring, including examples of how to watermark content, guidance on appropriate use, guidance on reporting adverse events associated with model u and ways to limit some forms of access to models. Sev- eral of these approaches have significant drawbacks: for example, there are no known robust watermarking techniques for language models and there are limits to watermarking for image models (Kirchenbauer et al., 2023; Saberi et al., 2023). As with many of the sections above, usage monitoring remains an area of active research.
9.5 Recommendations
Models should be released with accompanying documentation and easy-to-run code for training, evaluation and inference. Document model thoroughly to the extent possible. These are critical to avoiding misuse and harms and enabling developers to effectively build on your work. A well documented environment, code, and versions of the appropriate datasets.
Be thoughtful about the type of license to for artifacts. Open source is technical term and standard with widely accepted definition (Initiative, 2024). If there is a risk of misuse then consider behavioral restrictions from standardized tool (McDuff et al., 2024).
Frameworks for monitoring and shaping model usage have become more prevalent as policymakers have attempted to constrain certain end uses of foundation models. Several approaches include adverse event reporting, watermarking, and restricting access to models in limited ways. Consider providing guidance to users on how to use your models responsibly and openly stating the norms you hope will shape model use.
9.6 Review
In this section we critically review the current state of resources for model release and monitoring, from our survey.
Reproducibility, especially through executable code, benefits all parties. The research community has benefited substantially from openness and transparency in how foundation model artifacts are documented and released. As have proprietary developers who benefit from the rapid prototyping and innovations on their released technical systems, propelled by the open community. Our review of resources suggests that often the most widely adopted tools are those that are not just well decuumnnted/descriieen but also those that are easy to run with executable code.
Usage monitoring remains challenging, and offers both advantages and disadvantages. Existing tools such as watermarking for AI outputs (Kirchenbauer et al., 2023; Saberi et al., 2023) or model fingerprinting (Xu et al., 2024) offer some degree of verification regarding the source of data or models. However, their robustness to adversarial removal, or to detection, remain dubious- this is particularly true for text data. This can result in over-confidence that these methods provide a panacea, and instead result in false positives. There are also open questions on privacy as the right for individuals to produce content without attribution. We hesitate to prescribe these nascent solutions broadly, without fully understanding the particular context under consideration for the data, the model, and their potential uses and abuses.
F.g https://wwet microsoft. uttp//oon
25

---

Published in Transactions on Machine Learning Research (12/2024)
Permissions and restrictions around model use should be more explicit about their data provenance, and the other upstream ingredients which may impact use intentions, consent, and permissions. The ethical, responsible or legal use of a model may depend on a series of upstream factors, beyond the final developers' license. This could include the license of the datasets used for training, the terms of service attached to those data sources (e.g. some data is synthetically generated), or the license of the base model (if finetuned). prrsdditiio-depeneeee
The norms, best practices, and legal relevance of each of these factors evolving, and beyond the scope of this cheatsheet. However, we suggest developers document these factors in their model releases, beyond their own license, such that downstream developers and users can make informed decisions.
10 Discussion
General- AI systems require both general-purpose and case-specific development tools. General- purpose Al systems are used for an increasingly broad, and often unforeseen, range of applications (Zhao et al., 2023b; Schillaci, 2024; Longpre et al., 2024b). These include consumer-oriented uses including creative composition, information seeking, brainstorming, and reasoning, as well as industry applications in software development, entertainment, law, medicine, and journalism (Bommasaní et al., 2021; Brigham et al., 2024). Across these uses, the common denominator is often the foundation model, while the system setting, expectations, and user base vary. Supporting these broad uses will likely require suite of tools informed by "naturalistic" uses in each setting, to understand the real risks and shortcomings of given application (Lin et al., 2024; Kapoor et al., 2024). The tools compiled in this work are aimed at the foundation model layer, but may miss other essential components of responsible development or evaluation at the system or application level. For these reasons, we suggest these tools are a starting point, but not in themselves sufficient for responsible development that requires a deeper study of the intended use. For example, medicine there are likely to be considerably different considerations regarding foundation model behavior compared to those in creative tasks. Although the tools here can help, it is unlikely that they will address all the nuanced aspects of different vertical applications.
Recommendations across Development Stages. We have synthesized recommendations for each devel- opment stage and here we note some commonly recurring themes. First, throughout data sourcing, model training, and evaluation there is a lack of transparency, documentation, and reproducibility. Starting docu- mentation early and maintaining it throughout a project makes this process easier. Reproducibility, while not substitute for documentation, provides starting point for downstream developers to understand, iterate, and improve prior practices. We especially recommend that closed-source source evaluations release their evaluation scripts, to disambiguate the many evaluation details and settings that can complicate results. For transparency, environmental impact metrics from data centers and consumer-level dashboards could provide more fine-grained information.
A common theme across development stages is also the dearth of tools for non-English, internationally representative and multi-modal systems. Especially for data sourcing. these tools are lacking. In addition, datasets are often shaped by their availability and expedient collection processes. For instance, many text-to- image datasets rely on collecting captions, but less effort has been invested in collecting interspersed text and images, with more complex and realistic relationships. This makes many datasets ill-fitting to the necessary distribution, quality, and diversity of more specialized tasks. Synthetic data has demonstrated promise as a way to fill some of these gaps.
Lastly, we recommend that evaluation procedures shift away from evaluating models toward evaluating entire systems, which may include the settings of deployment, ioput/outpor filters, other guardrails, and the user interface. Tools for evaluation of models "in-the-field field" are much less mature than those for static benchmark testing. We encourage practitioners to expand upon this type of testing. These systems are deployed to interact with people, and other software in complex environments. These interactions need be part of the evaluation cycle.
26

---

Published in Transactions on Machine Learning Research (12/2024)
Acknowledgements
Stella Biderman, Hailey Schoelkopt, and Aviya Skowron's work on this project was funded in part by a grant from the Omidyar Network.
27

---

Published in Transactions on Machine Learning Research (12/2024)
References
Nanotron.
2024.
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. arv preprint ar X4::02309.00445, 2023.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp. 265- 279. PMLR, 2023.
Stability AL. Stable audio tools. Github URL http://hhhh..oaaaaaaaaaaaaaaaaaa tools.
Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang. FETA: A benchmark for few-sample task transfer in open-domain dialogue. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Pp. 10936-10953, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. ww..8663/112222.mmnpppp org 2022. emnlp-main 751.
URL
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arviv preprint ar XxX:::2312.02206, 2023.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. Xiv preprint arXit:2442..18877 2024.
Zaid Alyafeai, Maraim Masoud, Mustafa Ghaleb, and Maged S Al-shaibani. Masader: Metadata sourcing for arabic text and speech data resources. In Proceedings of the Thirleenth Language Resources and Evaluation Conference, pp. 6340-6351, 2022.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large scale autoregressive language modeling in PyTorch, 8 2021. URL https: //wp:////wwwww.. https://www.iichub..c www.ggitt.....eeeeeeeeeeeeeeeeee hemm//eletheeraigttt. nexx.
Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arv preprint arXio:2007.0305. 2020.
Quentin Anthony, Stella Biderman, and Hailey Schoelkopf. Transformer math 101. GitHub Repo, 2023. URL blog. eleuther ai/.
Quentin Anthony, Hailey Schoelkopf, and Stella Biderman. The EleutherAl Model Training Cookbook. GitHub Repo, 2024. URL http://ghttb.oceeeeeeetttt/ta/ta
Anthropic. Challenges in evaluating ai systems, October 2023. URL http://www..tttropcccccoco/e//e/ mealuating-aissyaae
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. https: https://xww-coe hwwp:wwcdd.anthhpiiii... anthropic.
http://ooodeeearaaadaaaeeeeeee http:///wwwwcccaloooooooc/ooo/cc [ Claude_3.pdf.
oow////eaallaaaaaattttatottttter Card Claude_3.pdf 3 2024a. URL:
Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024b. URL http:::/awr-cc....rrriiii..
Card @l__Cllud___ _3..
28

---

Published in Transactions on Machine Learning Research (12/2024)
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-uultllngual speech corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceddings of the Twelfth Language Resources and Evaluation Conference, Pp. 4218 44222, Marseille, France, May 2020. European Language Resources Association, ISBN 979-10-95546-3444. URL https: http://///aaaaaaooooo.o.oorgggg
1re.-1.520.
Anas Awadalla, Irena G Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arx preprint arXtm:2308.0133 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The 3rd Workshop on Mathematical Reasoning and AI at NeurlPS'23, 2023.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. XLS- R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeach 2022, pp-
2022. doi:
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from AI Feedback, December 2022. URL //axii/ org/abs/22122 08073. arXiv:2212.08073 [cs].
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE//CV International Conference Computer Vision, pp. 1728-1738, 2021.
Jack Bandy and Nicholas Vincent. Addressing "documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXto:2105.05221, 2021.
Michele Banko, Brendon MacKeen, and Laurie Ray. A unified taxonomy of harmful content. In Seyi Akiwowo, Bertie Vidgen, Vinodkumar Prabhakaran, and Zeerak Waseem (eds.), Proceedings of the Fourth Workshop on Online Abuse and Harms, pp. 125-137, Online, November 2020. Association for Computational Linguistics.
doi: 10.18653/v1 /2020.alw--1116 10.18653/11/2000aaw-11..
URL
20t.t///oooooooo./222000000 a0000..alw-1. 11w-1.16.
Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https: //gt thtb.b.tt..tttg//g cww comm/rom100//c11 1ttp0////1ip-rrrrrraaaaa 2022.
Stas Bekman. Machine learning engineering open book. GitHub Repo. URL httttp:://ggthuu.ccooo//////e/ ml-engineering.
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6: 587-604, 2018. doi:
URL URI.https://aclaaahollg hawpetthhooogo.oo///eeeee. org/Q18-104: 1041.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430 PMLR, 2023.
29

---

Published in Transactions on Machine Learning Research (12/2024)
Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advancess in Neural Information Processing Systems, 36, 2024a.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo JaaoJonaaaan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arxiv preprint XrXo:22455.4478, 2024b.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornogra- phy, and malignant stereotypes. arXiv preprint ar XrXi::2110.01963, 2021.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint ar XrXf::2244.06755, 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint ar arXiv:2311.15127, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXii:2108.07255 2021.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index, 2023.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Betty Xiong, Sayash Kapoor, Nestor Maslej, Arvind Narayanan, and Percy Liang. Foundation model transparency reports. arXiv preprint arXtt:2402.16266, 2024.
Natalie Grace Brigham, Chongjiu Gao, Tadayoshi Kohno, Franziska Roesner, and Niloofar Mireshghallah. Breaking news: Case studies of generative ai's use in journalism. arXiv preprint arXiv:2406.13700 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https://openai. www////aaaee////eeeee.eeo wwweeegeieeeendedaeeeeeee simulators.
R Brown, C Webber, and JG Koomey. Status and future directions of the energy star program. Energy, 5(27): 505-520, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), aids.),Adao Advances in Neural Information Processing Systemss, volume 33, pp- 1877-1901. Curran Associates, Inc., 2020. URL https:// hetripp.cc/ppper..llee///ppeep http://///....eeee......ooo/o//.
.df.
Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hEigeartaigh, Simon Beard, Haydn Belfield, Sebastian Farquhar, Clare Lyle, Rebecca Crootof, Owain Evans, Michael Page, Joanna Bryson, Roman Yampolskiy, and Dario Amodei. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation, 2018.
Benjamin S. Bucknall and Shiri Dori-Hacohen. Current and near-term ai as a potential existential risk factor. In Proceedings of the 2022 AAAT/ACM Conference on AI, Ethics, and Society, AIES '22, pp. 119-129, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392471 doi: 10.1155/35149944544444 org/10. 0tte:/1..115//51140000 3534146
URL https://doli http:///ooooooo/...
30

---

Published in Transactions on Machine Learning Research (12/2024)
Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. An open dataset and model for language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 2023. doi: short.75. htt: / htpp/xxdddo/ doi org/10. wtrg/10.100 18653/11/2023 acl acl-short. .75.
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Indra Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, et al. Nusacrowd: Open source initiative for indonesian nlp resources.
Joseph Carlsmith. power-seeking ai an existential risk?, 2022.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models, ACM Transactions on Intelligent Systems and Technology, 2023.
Carol Chen. Transformer inference arithmetic.
arithmetic/, 2022.
2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Danish Contractor, Daniel McDuff, Julia Katherine Haines, Jenny Lee, Christopher Hines, Brent Hecht, Nicholas Vincent, and Hanlin Li. Behavioral use licensing for responsible ai. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 778-788, 2022.
to include Emilia David. Ai image training chd sexua abuse imagery. The Verge, December 2023.
URL https://www. theverge. com cam/20003/0220022204444//eeeree2
ieagemmmaann---eeeeeeaeaa-aaaaaa laiioniionnnnnn iacion-csam-gee csaac google-
stability-stanford 7:57
AM PST.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi:
Jiawen Deng, Jiale Cheng, Hao Sun, Zhexin Zhang, and Minlie Huang. Towards safer generative language models: survey on safety risks, evaluations, and improvements, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023.
Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jüin Kim, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. On measures of biases and harms in NLP. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Findings of the Association for Compulational Linguistics: AACL-IJCNLP 2022, pp. 246-267, Online only, November 2022. Association for Computational Linguistics. URL
202/2022. f findings aacl.22 24.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXto:1810.04805, 2018.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 862-872, 2021.
Mark Dingemanse and Andreas Liesenfeld. From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5614-5633, 2022.
Roel Dobbe. System safety and artificial intelligence. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1584-1584, 2022.
31

---

Published in Transactions on Machine Learning Research (12/2024)
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 1286-1305, 2021.
Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, 2023.
Kate Downing. Ai licensing can't balance "open" with "responsible". The Law Office of Kate Downing's Blog, 2023. URL https: http:////kaeedodinllll.. 1ttc://0222/00/////aaaaaaaaaaana waaat--bbaan---p---it-aa -with- responsible/.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arii preprint afXio:2305.14387, 2023.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data?, 2023.
The Free Software Foundation. What is free software?, 2024. URL https://web. archive ve. org/web/ 002303060104477mtttpp:: http://ttttt::/w...eet gnu. org/ httggghllsoppyy////eee Saaphhyy/eeeeeeeeee sw. en html. Last accessed on 2024-02-20.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. aro preprint arXit:2306.133%4 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024a.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over- training and on downstream tasks. arXiv preprint ar Xii::2403.0854) 2024b.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arxiv preprint arXiv:2101.000227 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https: org wwgg/record//1in /rtcordd//0026666.
Timnit Gebru, Jamie Morgenstern, Briana Secchione,Jennice Wortman Vaughan, Hanna Wallach, Hal Dauné Tii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):88-922 2021.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxieityPromppe Evaluating neural toxic degeneration in language models. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356-3369, Online, November 2020a. Association for Computational Linguistics. doi: hdw:: http:://aclanthooooy..org/ org./00000000000000 htt.///2....iinnnneoee findings-emmlp..5
URL
32

---

Published in Transactions on Machine Learning Research (12/2024)
Samuel Gehman, Suchin Gururangan Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxictyprompts Evaluating neural toxic degeneration in language models. In Findings of the Association for Compulational Linguistics: EMNLP 2020, pp. 3356-3369, 2020b.
Charles Goddard, Shamane Siriwarddhaaa, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models, 2024.
Aidan Gomez. Introducing command r+: A scalable IIm built for business. https: cohere com/ command-r-plae rewlrr--ppoosoooooooe microsoft-aateeeee azure/, 2024. URL: azure/-
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36, 2024.
Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. open_Ime Im: minimal but performative language modeling (Im) repository, 2023. URL https:::: wtt.htttts/ggitbbb..ttc www.ettt.....oooooooeeeeeeeeeete
1m/. GitHub repository.
Laura Hanu and team Unitary. Detoxify, November 2020. URL https: /ttp::://gtthneee httpa////ggiibb...oomitaaaaaaaae www./o/ttttaai////c//oo com.///maaarrra///tttiitt
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine--eneraaeee dataset for adversarial and implicit hate speech detection. arv preprint arXio:2203.09509, 2022.
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. trlX: A framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 8578-8595, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ /v/l 2023. 2023.emnlp-main.53. URL
http:/2022..mm1ppmppmmmmpm emnlp-main.530 530.
Luxi He, Mengzhou Xia, and Peter Henderson. What's in your" safe" data?: Identifying benign data that breaks safety. ar preprint XrXiv:2404.01099 2024.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. The Journal of Machine Learning Research, 21 (1 10089, 2020.
Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances Neural Information Processing Systems, 35.29217-29234, 2022.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. ar preprint ar arXir:2203.15555 2022.
Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. The dataset nutrition label. Data Protection and Primacy, 12(12):1, 2020.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback not gold standard, 2024.
33

---

Published in Transactions on Machine Learning Research (12/2024)
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL nttps://doll http://///oooo...../////// 5281/zenodo 5143773. If you use this software, please cite as below.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Lim-based input-output safeguard for human-ai conversations. arXiv preprint arX1t:3312.06674 2023.
The Open Source Initiative.
The open source definition, February 2024. URL.ttps::/opensorree ortg
Matthew Jagielski. A note on interpreting canary exposure. arXiv preprint arXie:2306.00113, 2023.
Yacine Jernite. Training data transparency in ai: Tools, trends, and policy recommendations. In Hugging Face Blog, 2023.
Yacine Jernite, Huu Nguyen, Stella Biderman, Anna Rogers, Maraim Masoud, Valentin Danchev, Samson Tan, Alexandra Sasha Luccioni, Nishant Subramani, Isaac Johnson, et al. Data governance in the age of large-scale data-driven language technology. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2206-2222, 2022.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint erXiv:2310.06770, 2023.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu,
Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, IEEE, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Sayash Kapoor, Emily F. Cantrell, Kenny Peng, Thanh Hien Pham, Christopher A. Bail, Odd Erik Gundersen, Jake M. Hofman, Jessica R. Hullman, Michael A. Lones, Momin M. Malik, Priyanka Nanayakkara, Russel A. Poldrack, Inioluwa Deborah Raji, Michael Roberts, Matthew J. Salganik, Marta Serra-Garcia, Brandon M Stewart, Gilles Vandewiele, and Arvind Narayanan. Reforms: Reporting standards for machine learning based science. ArXiv, abs/2305 2220.3000322 2023. URL https://:///at http:///aaaaivvooaaa/aee///e/ 230..07838.
Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, al. On the societal impact of open foundation models. Xiv preprint ar arXit:2403.00911. 2024.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-condittoned language models. arxiv preprint arXtt:2402.07865, 2024.
Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. Glotlid: Language identification for low-resource languages, 2023.
Andrej Karpathy. nanogpt. GitHub Repo, 2023. URL UKL.rtpp..//gttta..../////pt//pp
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. Xrviv preprint arXiv:2104.14337 2021.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. watermark for large language models. Xio preprint arXit:2301.100226 2023.
34

---

Published in Transactions on Machine Learning Research (12/2024)
Walter Klöpffer. Life cycle assessment: From the beginning to the current state. Envirommental Science and Pollution Research, 4:223-228, 1997.
Kevin Klyman. Acceptable use policies for foundation models: Considerations for policymakers and devel- opers. Stanford Center for Research on Foundation Models, apr 2024. URL httttttt/////crm.eeee stw.fffffm..aa.rd...orooooo edu/ 2024/04/08/ http///pss.stee. html.
Denis Kocetkov, Raymond Li, LI Jia, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, et al. The stack: 3 tb of permissively licensed source code. Transactions on Machine Learning Research, 2022.
Anton Korinek and Jai Vipra. Market concentration implications of foundation models: The invisible hand of chatgpt. 2023.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzti-Orsiikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50-72, 2022.
Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400: A multilingual and document-level large audited dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Veronika Laippala, Anna Salmela, Samuel Rönnqvist, Alham Fikri Aji, Li-Hsin Chang, Asma Dhifallah, Larissa Goulart, Henna Kortelainen, Mare Pàmies, Deise Prina Dutra, et al. Towards better structured and less noisy web data: Oscar with register annotations. In Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022), pp. 215-221, 2022. 2022.
S Lakatos A revealing picture: Ai-generated 'undressing' sudreesinnginagee move from niche pornography discussion forums to scaled and monetized online business. Technical report, Technical report, Graphika, Dec 2023. URL https:/ /ptpp/pbubbcccssss.....sss graphika ...., 2023a.
Santiago Lakatos. A Revealing Picture: Al-Generated 'Undressing' Images Move from Niche Pornography Discussion Forums to Scaled and Monetized Online Business. Technical report, December 2023b. URL https://publiccaasette
pdf.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint ar Xit::2403. 13787, 2024.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Fro- hberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The bigscience roots corpus: A 1.6tb composite multilingual dataset. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 31809-31826 Curran Associates, Inc., 2022. URL Uttpsttp:///rroceeiinns neur neurips. wwcppappprffiiiecceeeeeeeeeeeeee / BeadddBeehmmaeee.._ pdf.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024a.
35

---

Published in Transactions on Machine Learning Research (12/2024)
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? arXiv preprint ar arXi::2405.02244 022466 2024b.
Eric Le Ferrand, Steven Bird, and Laurent Besacier. Learning from failure: Data capture in an australian aboriginal community, In Proceedings the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4988-4998, 2022.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burchy and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424-8445, Dublin, Ireland, May 2022. Association org/
for Computational Linguistics. doi: 10.18655/w1/2022.2cooom.... URL https:h/aclaandtaaeeee 2022. acl- -ong-lon...0 577.
Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard- Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. Evaluating human- human-language model interaction. Transactions on Machine Learning Research, 2023a.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. arviv preprint ar XrXx:22311.04287 2023b.
Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- pp. pp.3199-3207 2022.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, 2021a.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMilllan-Major, Phillip Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. URL https: / /t///aaaanttttt aclanthologyyy...om . org/ emnlp-demo..... 211
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. ar preprint arXt::2305.03726, 2023a.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXio preprint ar arXio:2306.04388 2023b.
Pengfei Li, Jianyi Yang, Mohammad A Islam, and Shaolei Ren. Making ai less" thirsty": Uncovering and addressing the secret water footprint of ai models. arviv preprint ar Xr:2:22304.0327,, 2023c.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2023.
36

---

Published in Transactions on Machine Learning Research (12/2024)
Q Vera Liao and Ziang Xiao. Rethinking model evaluation narrowing the socio-technical gap. ar preprint arXto:2306.03100 2023.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking Iims with challenging tasks from real users in the wild. arXiv preprint arXio:2406.04770, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Swittzrrandd, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint ar arXi::2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.1737) 2024b.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data for language models. arXiv preprint axXi::2444.07503 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXit:2307.06281 2023b.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. S2ore: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational L Linguistics, pp. 4969-4983, 2020.
Shayne Longpre, Marcus Storm, and Rishi Shah. Lethal autonomous weapons systems & artificial intelligence: Trends, challenges, and policies. MIT Science Policy Review, 3(1):47-56, 2022.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint ar arXte:2301.13688 2023a.
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXiv preprint ar arX11.2310.16677, 2023b.
Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili- Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xin Yong, Suhas Kotha, et al. A safe harbor for ai evaluation and red teaming. Xrv preprint ar (ttr:203300889 2024a.
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. Consent in crisis: the rapid decline of the ai data commons. ar Xiu preprint ar Xiv:24007 11493,, 2024b.
Shayne Longpre, Robert Mahari, Naana Obeng-Marnu, William Brannon, Tobin South, Katy Gero, Sandy Pentland, and Jad Kabbara. Data authenticity, consent, & provenance for ai are all broken: what will it take to fix them? XrXiv preprint ar arXtt:2404.12691 2024c.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. Xiv preprint arXin:2402.11177, 2024.
37

---

Published in Transactions on Machine Learning Research (12/2024)
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, 176b parameter language model. Journal of Machine Learning Research, 24(253), 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficiet fine-tuning methods. https://github www.hugginggace//ppaaeeeco 2022.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging, 2022.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: standardized evaluation framework for automated red teaming and robust refusal, 2024.
Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William A Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Will Cukierski, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjiao Chen, Mehul Smriti Raje, Max Bartolo, Sabri Eyuboglu, Amirata Ghorbani, Emmett Daniel Goodman, Addison Howard, Oana Inel, Tariq Kane, Christine Kirkpatrick, D. Sculley, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Y. Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi. Dataperf: Benchmarks for data-centric Al development. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: pettps://opnntee http:///ppeeeeeeeeeeeeeeee net// forum? di-aFiFFF-FF
Daniel McDuff, Tim Korjakow, Scott Cambo, Jesse Josua Benjamin, Jenny Lee, Yacine Jernite, Carlos Muñoz Ferrandis, Aaron Gokaslan, Alek Tarkowski, Joseph Lindley, et al. On the standardization of behavioral use clauses and their adoption for responsible licensing of ai. arxiv preprint ar arXfo:2402.05977, 2024.
Kristina McElheran, J Frank Li, Erik Brynjolfsson, Zachary Kroff, Emin Dinlersoz, Lucia Foster, and Nikolas Zolas. Ai adoption in america: Who, what, and where. Journal of Economics & Management Strategy, 2024.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal llm pre-training, Xrv preprint ar xrXio:2403.09611,22 2024.
Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilić, et al. Documenting geographically and contextually diverse data sources: The bigscience catalogue of language data and resources. arXiv preprint 0rXio:2201.10066 2022.
Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in a nonparametric datastore. In NeurlPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. Xro preprint -tx:::20080012224 2023a.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. Xr preprint arXio:2305.16264 2023b.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: A Large-Scale Speaker Identification Dataset. In Proc. Interspeech 2017, pp. 2017. doi: 0.214477/neergpecc22277-999
38

---

Published in Transactions on Machine Learning Research (12/2024)
Arvind Narayanan and Sayash Kapoor. Ai safety is not model property. p/ai-safety-iis a-s--ntta-moddl--pppeeee March 2024. Accessed: YYY-MMM-DD.
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilinguul dataset for large language models in 167 languages. arviv preprint arXiv:2309.094000 2023.
Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E Ho. Multilegalpile: A 689gb multilingual legal corpus. arXiv preprint arXio:2306.02065 2023.
Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin. Jailbreaking attack against multimodal large language model. Xiv preprint ar arXtt:2402.02309, 2024.
Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, and Jimmy Lin. Better quality pre-training data and t5 models for African languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 158-168, Singapore, December 2023, Association for Computational mainn1. URL https::/acllatthote
Linguistics. doi: 11.
org/2023. amnlp-maing
OpenAccess-A-Coolective Axolot. GitHub axolotl.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint Xrxx:22203.021 02155, 2022. UtL https//////axx
:/apppii////xv..../// /arxiv.vv.. http//axvv....aaaaa.aaae wtg//br/////////ooo
200..221.......2222 02155.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086-2105 2022.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmyy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arxiv preprint arXto:2310.06786, 2023.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint #rXi::2104110300, 2021.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.011 2023.
Guilherme Penedo, Alessandro Cappelli, Thomas Wolf, and Mario Sasko. Datatrove: large scale data processing, 2024. URL https: //gi: hwtp////ggggggggc////attttotoo.t dattttroe..
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023.
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexandra Sasha Luccioni, Yacine Jernite, and Anna Rogers. The roots search tool: Data transparency for lims. arv preprint arXto:2302.14035, 2023.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS: A Large- Scale Multilingual Dataset for Speech Research. In Proc. Interspeech 2020, pp. 2757-2761, 2020. doi: 10.21437/Tnteere http://eeesseeccee0000000000
39

---

Published in Transactions on Machine Learning Research (12/2024)
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- tuning aligned language models compromises safety, even when users do not intend to! ar preprint arXite2310003699, 2023.
Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21527-21536, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, PP. 28492-28518. PMLR, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint ar arXie:2305.18290, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 (1) (1:5):5555555 5551, 2020.
Inioluwa Deborah Raji and Roel Dobbe. arXiv:2401.10899, 2023.
Concrete problems in ai safety, revisited. ar preprint
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe Ilm applications with programmable rails. arxiv preprint arXt0:2310.10001, 2023.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024.
Mehrdad Saberi, Vinu Sankar Sadasivan, Keivan Rezaei, Aounon Kumar, Atoosa Chegini, Wenxiao Wang, and Soheil Feizi. Robustness of ai-image detectors: Fundamental limits and practical attacks. arXiv preprint arXie:2310.00076 2023.
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "Everyone wants to do the model work, not the data work": Data cascades in high-stakes AI. In CHI, CHI 21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380966. doi: 10.1145/341166..4455181 URL https://ddll org/ 10. 1145/3411764 3445518.
Ramon Sanabria, Nikolay Bogoychev, Nina Markl, Andrea Carmantini, Ondrej Klejch, and Peter Bell. The edinburgh international accents of english corpus: Towards the democratization of english asr. In ICASSP 2023-2023 JEF.E. International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. ICLR 2022, 2021. URL https: http://wwwwwww..aa/////////
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. ar Xiv preprint Xix::2211 1005100, 2022.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024.
Zachary Schillaci. Lim adoption trends and associated risks. In Large Language Models in Cybersecurity: Threats, Exposure and Miligation, pp. 121-128. Springer Nature Switzerland Cham, 2024.
40

---

Published in Transactions on Machine Learning Research (12/2024)
Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. Codecarbon: estimate and track carbon emissions from machine learning computing. Cited on, pp. 20, 2021.
Scruhmmanhhhhaannn Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt,, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Roy Schwartz,Jesse Dodge, Noah Smith, and Oren Etzioni. Green ai. Communiications of the ACM, 63(): 54-63, 2020.
Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. In The Twelfth International Conference on Learning Representations, 2023.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. ar Xiv preprint arXtt::310.11678, 2023.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2020.
Riley Simmons-Edler, Ryan Badman, Shayne Longpre, and Kanaka Rajan. Ai-powered autonomous weapons risk geopolitical instability and threaten ai research. arXiv preprint ar Xix::24050008555 2024.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.066119 2024.
Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. "i'm sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9180-9211, 2022a.
ShadenSSmthh Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train eergatron-tuinng nlg 530b, a large-scale generative language model, 2022b.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. Allen Institute for AI, Tech. Rep, 2023.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. arx preprint 2rX 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa
41

---

Published in Transactions on Machine Learning Research (12/2024)
Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burde Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Mosegui González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Star, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-Lóóez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, Jasas, B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Jekateii Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Peiuua Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad,
Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi,
Leter
42

---

Published in Transactions on Machine Learning Research (12/2024)
Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game:
Quantifying and extrapolating the capabilities of language models, 2023.
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training, 2024.
Pedro Javier Ortiz Suárez, Benoit Sagot, and Laurent Romary. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache, 2019.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jilling Tang, Jindong Wang, Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. Trustllm: Trustworthiness in large language models, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint ar arXtt:2332.11880 2023.
David Thiel, Melissa Stroebel, and Rebecca Portnoff. Generative ML and CSAM: Implications and Mitigations. 2023a. doi: 10.25740/jy206y33733. URL https://pput/ sttaa/// edu/jv206y33793.
stanford
David Thiel, Melissa Stroebel, and Rebecca Portnoff. Generative ml and csam: Implications and mitigations, 2023b.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pp. 2214-2218. Citeseer, 2012.
Together AI. Redpajama-data-v2: An open dataset with 30 trillion tokens for training large language models. Blog post on Together AI, Oct 2023. URL https: //ww:://wwwwwwww www.ttgeteer.....m http.ttteeeer.abbloggg/////
aama-data-22
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar Xiv::22302.1397/ 2023.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. Xiv preprint ar Xrxo:22002.08877, 2024.
Jörgen Valk and Tanel Alumãe. Vo07ingua107: dataset for spoken language recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 652-658. IEEÉ, 2021.
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. Challenges and frontiers in abusive content detection. In Sarah T. Roberts, Joel Tetreault, Vinodkumar Prabhakaran, and
43

---

Published in Transactions on Machine Learning Research (12/2024)
Zeerak Waseem (eds.), Proceedings of the Third Workshop on Abusive Language Online, pp. 80-93, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1 /W19-350. 11116/6/1//19993000 URLhttps:
//aclanthology. org/W19-3509.
Bertie Vidgen, Hannah Rose Kirk, Rebecca Qian, Nino Scherrer, Anand Kannappan, Scott A Hale, and Paul Röttger. Simplesafetytests: test suite for identifying critical safety risks large language models. arXiv preprint 0rX:::2311.088 008370, 2023.
Jai Vipra and Anton Korinek. Market concentration implications of foundation models: The invisible hand of chatgpt. The Brookings Institution, 2023. URL https: http://///wwwbooooissssssse///// -aareette-eeeeee-------r-r
concentration-impt o -a---e-oooouuao---dooaa--a---iii
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehensive assessment of trustworthiness gpt models, 2024.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arviv preprint ar arXie21011.0039 2021.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, arv preprint arXtt:2203.11171, 2022.
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), Proceedings of the BabyLM Challenge at the 27th Conference on Computaaionnll Natural Language Learning, Pp. 1-34, Singapore, December 2023, Association for Computational Linguistics. doi: (1066///1//233.cool-bbayy// 2023. conll wallllbbbllmmm.mm 1.
URL https: /aclanthoooyy org/
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twneffth Language Resources and Evaluation Conference, pp. 4003- 4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL
www.:olloyyyr.g/2222222/ org/2020..0...2..oo 1rec- 1, 494.
Ross Wightman. Pytorch image models. https:/ /ttp:///gtttt.tt///
2019.
Herbert Woisetschläger, Alexander Isenko, Shiqiang Wang, Ruben Mayer, and Hans-Aro Jacobsen. A survey on efficient federated learning methods for foundation model training. arXiv preprint arXto:2401.044)
2024.
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. arXiv preprint ar arXiv:2307.03025, 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. arviv preprint arXiv:2305.0042, 2023a.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, 2023b.
44

---

Published in Transactions on Machine Learning Research (12/2024)
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lingwistics (Volume 1: Long Papers), pp. 3225-3245, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 /00/////o//2..llloooog.g.. URL https://aclaanthllogs wrg/2023.../.. 2023 acl-long. 181.
Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models. arXiv preprint ar arXio:2401.12255, 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models, 2023.
Zheng Xin Yong, Cristina Menghini, and Stephen Bach. Low-resource languages jailbreak gpt-4. In Socially Responsible Language Modelling Research, 2023.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models. In The Twelfth International Conference on Learning Representations, 2023.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention, 2023a.
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions, 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arviv preprint XrXo:2203318822,, 2023a.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. (inthe) wildchat: 570k chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv22306.05685 2023.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. ar aiXiv preprint ar axXt::3302.09419 2023.
Jiahang Zhou, Yanyu Chen, Zicong Hong, Wuhui Chen, Yue Yu, Tao Zhang, Hui Wang, Chuanfu Zhang, and Zibin Zheng. Training and serving system of foundation models: A comprehensive survey. ar preprint arXto:2401.026433 2024.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
Piotr Zelasko, Daniel Povey, Jan "Yenda" Trmal, and Sanjeev Khudanpur. Lhotse: speech data representation library for the modern deep learning ecosystem, 2021.
45

---

Published in Transactions on Machine Learning Research (12/2024)
A
Contributions
To create this cheatsheet, variety of contributors were asked propose resources, papers, and tools relevant to open foundation model development. Those resources were grouped into sections, which were each curated by subset of the contributors, We list the main curators of each section, listed alphabetically below. However, it is important to note that many contributors advised across sections, and helped with preparing the interactive cheatsheet tool. Nay San led the speech modality, and Gabriel Ilharco led the vision modality.
Pretraining Data Sources David Adelani, Stella Biderman, Gabriel Ilharco, Kyle Lo, Shayne Longpre, Luca Soldaini, Nay San
Finetuning Data Catalogs David Adelani, Stella Biderman, Gabriel Ilharco, Shayne Longpre, Nay
San
Data Search, Analysis, & Exploration Stella Biderman, Gabriel Ilharco, Shayne Longpre, Nay San
Data Cleaning, Filtering, & Mixing Alon Albalak, Kyle Lo, Luca Soldaini
Data Deduplication Alon Albalak, Kyle Lo, Shayne Longpre, Luca Soldaini
Data Decontamination Alon Albalak, Stella Biderman, Shayne Longpre
Data Auditing Stella Biderman, Aviya Skowron
Data Documentation Stella Biderman, Aviya Skowron
Data Governance Stella Biderman, Yacine Jernite, Sayash Kapoor
Pretraining Repositories
Stella Biderman, Gabriel Ilharco, Nay San, Hailey Schoelkopf
Finetuning Repositories
Gabriel Ilharco, Nay San, Hailey Schoelkopf
Efficiency& & Resource Allocation Hailey Schoelkopf
Educational Resources Hailey Schoelkopf
Estimating Environmental Impact Peter Henderson, Sayash Kapoor, Sasha Luccioni
Effective use of Resources
Sayash Kapoor, Sasha Luccioni
General Capabilities
Rishi Bommasani, Shayne Longpre, Kevin Klyman
Risks & Harms Maribeth Rauh, Laura Weidinger
Risks & Harm Taxonomies
Bertie Vidgen
Model Documentation
Sayash Kapoor, Shayne Longpre
Reproducibility Stella Biderman, Shayne Longpre
License Selection Stella Biderman, Yacine Jernite, Kevin Klyman, Aviya Skowron, Daniel McDuff
Usage Monitoring Kevin Klyman
Website Justin Riddiough, Shayne Longpre, Luca Soldaini
Advising Stella Biderman, Peter Henderson, Yacine Jernite, Sasha Luccioni, Percy Liang, Arvind Narayanan, Victor Sanh
46


Reviews:
[{"review_id": "25kjZxvw6Q", "reviewer": "Reviewer_EeXL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper formulates the pipeline of model development and survey resources of each stage that requires. \n2. This paper has provided a comprehensive survey to cover many topics that are involved in foundation model building. \n\n\n**Weaknesses**\n\nThe motivation of this paper is good but lacks some in-depth insights about each part in building foundation models. Some conclusions are concrete. For example, some practices in data preparation are common, while how to well deploy these operations could be more important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The rise of foundation models has attracted enormous attention in many scenarios and applications. To facilitate the development of foundation models in practical applications, this paper introduces a cheatsheet about foundation models. It surveys many resources including software, documentations, frameworks, guides, practical tools and other topics. Based on this list, it could help the community to understand the development of foundation models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It will be better to define the scope of tools and resources. Both \"tool\" and \"resource\" are quite common words that will make some misunderstanding. Some research topics like AI Agent also introduce tool learning while is a different direction. To make a better understanding, I suggest authors first define the scope of tools and resources used in this paper first.\n2. Besides, it would be better to add some sections to describe the applications of current models.", "broader_impact_concerns": "This paper does not require broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VEkm3I4Drq", "reviewer": "Reviewer_VaTQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The reviewer wishes to commend the authors of the cheatsheet for embarking on such an important and timely task, which clearly requires the effort of large number of individuals with a wide range of expertise and interests. The sections about model evaluation, environmental impact and model release are particularly unique and a contribution of the review.\n\nThe main limitation of the work is that as intended and to be successful it requires an ongoing effort and community involvement/contribution, both of which do not lend itself for a one-time publication in an academic journal. That being said, this is naturally the case for other review papers, but less critical in fields that are more mature and established as foundation models. One way to address this will be to provide an online resource mirroring the content in a website (as perhaps indicated in Section 10, but not explicitly described in the main body of the paper), in a manner that can be expanded (hopefully by a larger community) in the future.\n\nAnother important problem has to do with the methodology used to conduct the review. Though the methodology used to collect and include resources is described, there does not seem to be a well defined methodology to systematically evaluate and report on the resources being considered. Note that the reviewer is not suggesting a quantitative evaluation of such resources, but a systematic or at least well defined criteria and reporting of such evaluation based on for instance popularity, features, documentation, maintenance, use in the literature, etc. Some of these criteria are mentioned in Section 2, however, when describing the resources in each section, such criteria are not explicitly or systematically described in a manner that makes it easier for the reader to understand the advantages and disadvantages of each resource being described.\n\nNote that in general, the reviewer is not necessarily disagreeing with the list of the completeness thereof, however, in its current for it is difficult to grasp which audience is pretending to serve (despite the brief, almost to vague, mention to it in the second bullet point at the end of page 4).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce Foundation Model Development Cheatsheet, which consists of a growing collection of 250 tools and resources spanning different modalities (text, vision and speech) to support data selection, processing, model training, model evaluation, while helping with the understanding of environmental impact, limitations, risks, responsible release, licensing and deployment best practices. Moreover, during the development of the cheatsheet, they found that existing tools underserve ethical and real-world datasets, evaluation lacks reproducibility and transparency, English dominates the benchmarks, and evaluation is center around models rather than systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Table 1 needs references pointing back to the relevant subsections in the text.\n\nConsider having a table with all web resources rather than the almost excessive use of footnotes.\n\nThe content is dense. Consider having a table for each section containing tools/packages/websites with references and descriptions as is typically done in review papers. For instance reusing the cheatsheet samples in the end that are present without being referenced in the text.\n\nThere seems to be a need for a last section to support the status quo, review and recommendations for all phases as well as an overall discussion of the complete review (besides what is already presented in the introduction).", "broader_impact_concerns": "There are no concerns about broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YrU4x51EHa", "reviewer": "Reviewer_SxsS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- The survey is well-conducted, examining over 250 available tools.\n- The findings are organized and summarized effectively, as seen in Table 1 and the best practices listed at the beginning of each section.\n- It comprehensively addresses both technical and non-technical issues related to foundation models, serving as a valuable reference for future researchers.\n- Especially, the cheatsheets in the Appendix seem useful.\n\n\n## Weakness\n\n- Does not offer many new insights. For example, the bias towards English and textual data, and the necessity of realistic evaluation over benchmark scores are well-known issues.\n- Not much depth. For instance, when discussing data, it would be informative if the survey provided qualitative differences between datasets, such as the pros and cons of them.\n- Primarily discusses supervised instructions when addressing fine-tuning data, without delving deeply into human preference data.\n\nMinor comments:\n- Check spacing on page 43. Consider using \\clearpage for the Appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper examines the responsibility issues associated with foundation models. It explores the development process, offering recommendations for each phase: data sources, data preparation, model training, model evaluation, and model release.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Good survey paper as is. Maybe add a discussion about human preference data in Section 3.2? https://huggingface.co/datasets/Anthropic/hh-rlhf is one example.", "broader_impact_concerns": "The paper discusses the risks of foundation models in their survey.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Survey certification**: This is a rather in-depth survey.
Decision: Accept (poster)