A Survey on Data Selection for Language Models
Authors: Alon Albalak
Yanai Elazar
Sang Michael Xie
Shayne Longpre
Nathan Lambert
Xinyi Wang
Niklas Muennighoff
Bairu Hou
Liangming Pan
Haewon Jeong
Colin Raffel
Shiyu Chang
Tatsunori Hashimoto
William Yang Wang
Venue: Accepted by TMLR ()

Abstract: A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required.

Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies.

To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.

Published in Transactions on Machine Learning Research (07/2024)
A Survey on Data Selection for Language Models
Alon Albalak,* UC Santa Barbora, SynthLabs
Yanai Elazar, Allen Institute for AI, University of Washington
Sang Michael Xie, Stanford Universiby Shayne Longpre, Massachusetts Inststute of Technology Nathan Lambert, Allen Institute for AT Xinyi Wang, UC Sonta Barbara
Niklas Muennighoff, Contextual AI Bairu Hou, UC Sonta Barbara Liangming Pan, UC Santa Barbora
Haewon Jeong, UC Santa Barbara
Colin Raffel, University of Toronto, Vector Institute Shiyu Chang, UC Sonta Barbara Tatsunori Hashimoto, Stanford University
William Yang Wang, UC Santa Barbara
Reviewed OpenReview: https: openreview. net/ pormm? id= WFXHHWNNTSH
Abstract
A major factor in the recent. success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount. of training required.
Data selection methods aim determine which candidate data points include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection rescarch. Consequently, knowledge effective data sclection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies.
To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related rescarch arcas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing entry point for new and established researchers. Additionally, throughout this review draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.
Work completed while UC Santa Barbara. Correspondence alon_all aln.auaaakduccbb.....
1

---

Published in Transactions on Machine Learning Research (07 (07/2024) /204)
Table of Contents
1 Introduction
4
2 A Taxonomy for Data Selection
5
2.1
Background and Motivation of Data Selection
5
2.2 A Unified Conceptual Framework for Data Selection
6
2.3
Dimensions of Variance the Data Selection Taxonomy
2.3.1
Distribution Matching vs. Diversification
2.3.2
Altering the Dataset Data Point
8
2.3.3
Output Space: Binary Natural Number Selection
8
2.3.4
Training Stage
8
3
Data Selection for Pretraining
9
3.1
Language Filtering
9
3.2 Heuristic Approaches
12
3.3 Data Quality
15
3.4
Domain-Specific Selection
3.5 Data Deduplication
3.6 Filtering Toxic and Explicit Content
21 ................................
3.7 Specialized Selection for Multilingual Models
23 ................................
3.8 Data Mixing
24
3.9 Current Best. Practices and Landscape For Data Selection
26
3.9.1 Best Practices
26
3.9.2 Current Landscape of Datasets For Data Selection Research
4 Data Selection for Instruction-Tuning and Multitask Training
28
5 Data Selection for Preference Fine-tuning: Alignment
31
6 Data Selection for In-Context Learning
33
7 Data Selection for Task-specific Fine-tuning
35
8 Data Selection for Other Domains
38
9 Related Topics
41
10 Discussion
43
10.1 Test set deecontamination.
43
2

---

Published in Transactions on Machine Learning Research (07 (07/2024)
10.2 Tradeoffs between memorization and generalization
44
10.3 There's no free lunch for filtering
44
10.4 Tools for data sclection..
45
10.5 Considerations when applying data selection your setting
45
Future Directions: Challenges and Opportunities
47
11.1
Accelerating research data selection
................................
47
11.2 Better understanding the properties of the target distribution
48
11.3 Shifting compute time from model training to data processing
48
12 Conclusion
49
3

---

Published in Transactions on Machine Learning Research (07/2024)
1 Introduction
Data selection is a long-standing challenge of machinc learning where, given collection of raw data, the goal is to design a dataset that is optimal under some objective function (John & Draper, 1975).
One often-used sense of optimality in data selection is with regard to a model's performance. In this work, we adopt the commonly held view that, at their core, machine learning models are a method for modeling statistical potterns in data and, from the probabilistic viewpoint, the optimal dataset is that which most closely matches the distribution under which the model will be evaluated (Murphy, 2012). While the probabilistic viewpoint presents a common view of how to select data that improves model performance, this is not the only goal of data selection methods. Data selection methods can reduce costs (by reducing dataset size (Ortiz Suárez et al., 2019; Schreiber et al., 2020; Brown et al., 2020a; Lee et al., 2022a; Sorscher et al., 2022)), ensure the integrity of evaluation metrics (by removing data that is suspected to be from the evaluation data (Rac et al., 2021; Marone& & Van Durme, 2024; Oren et al., 2024)), and reducing undesirable behaviors (such as bias and toxicity (Dodge et al., 2021; Welbl et al., 2021; Luccioni & Viviano, 2021; Longpre et al., 2023c)).
Data selection has recently become especially important in the context of large language models (Zhao et al., 2023b; Minaee et al., 2024). Language models can undergo multiple stages of training (pretraining (Peters et al., 2018; Radford & Narasimhan, 2018; Devlin et al., 2019; Raffel et al., 2020; Touvron et al., 2023a), instruction-tuning (Mishra et al., 2021; Sanh et al., 2022; Longpre et al., 2023a; Muennighoff et al., 2024), alignment (Ziegler et al., 2019; Bai et al., 2022b; Ouyang et al., 2022; Rafailov et al., 2023), etc.), and data selection plays an important role in each stage. However, as the objectives of training differ across each stage, the goals of data selection also vary accordingly. For example, language models are commonly pretrained on large corpora of text from varicty of sources. Among these sources is Common Crawl, a collection of around 250 billion webpages scraped from the internet. These webpages amount to around 11 petabytes of data, collected from internet scraping efforts since 2008, with an additional 3-5 billion new web pages being crawled monthly.² Due to the massive size of pretraining corpora, a common goal of data selection during pretraining is to remove significant quantities of data through a series of filters (Conneau & Lample, 2019; Raffel et al., 2020; Wenzek et al., 2020; Gao et al., 2020; Rae et al., 2021; Tee et al., 2022a) that aim to only retain data that is deemed "high-quality". In contrast with pretraining, one form of data selcetion for fine-tuning a model on larget task is to select additional auxiliary samples that will be most beneficial additional learning signals for the target task (Albalak et al., 2023b; Ivison et al., 2023a).
In this work, we unify the wide array of data selection methods under a conceptual framework that allows us to compare and contrast the variety of methods under our probabilistic viewpoint ($2.2) with a focus on model pretraining. Through this survey, we demonstrate how all data selection methods define a utility function that determines the utility of data, as well as a selection mechanism that determines how to use a data point based on its utility. Our conceptual framework enables us to classify the wide variety of data selection methods and create a faxonomy of the existing literature. Overall, this survey aims to achieve two goals: 1) we provide a collected resource of data selection methods that describes the current best practices and considerations when selecting data for training a language model, and 2) we provide a unifying view of data selection methods that allows us to define and describe potentially fruitful future directions of research. While this survey aims to be comprehensive, it would be prohibitively long to include the exact details of every method, so we select a representative sample to discuss in depth and provide citations to the many methods that we cannot cover in depth.
We organize the survey as follows. First, we present taxonomy and unified conceptual framework for data selection (62). Next, we present the main focus of this work: surveying methods of data selection for language model pretraining ($3). Then, we follow up with data selection methods for other language model training regimes including multitask training and instruction-tuning (64), alignment ($5), in-context learning ($6), and task-specific fine-tuning (S7). We then extend the survey to notable methods of data selection used in domains other than language ($8), as well as a brief discussion of related topics (59). Then, we discuss some of the implications surrounding data selection ($10). Although the main focus of the survey is on language model pretraining, also cover methods from other training regimes and domains that can point out promising directions of research in the final section ($11).
1 Accordding to httpe///ommonccawlorgg. accessed on 11/02/23 (MM/DD/YY).
4

---

Published in Transactions on Machine Learning Research (07 /024)
Raw Data
Data Selection
Dataset
Language Model
Data Pipeline
ollo
alle
Learning Stage
Pretraining
Instruction-tuning
Alignment
In-Context Learning
Task-specific Fine-tuning
Reduce Blas, Bius/Toxiiitt
Rediuce Blas/ Tosicity
Selection Objective
Derformence
Data ffficiency
Model Derformance
Ifficinccy
Selection Emellecyy
Evalustion interiity
Selection Emfcionccy
Evalustion integrity
Figure 1: An overview of the data pipeline for language models. The process starts with raw data that is cleaned, filtered, and mixed to create a final dataset by the data selection process, then used to train (or evaluate) a model. The details and objectives data selection methods vary depending on the learning stage, and we identify five common objectives: improving model performance, improving data efficiency, selecting data efficiently, ensuring evaluation integrity, and reducing model bias and toxicity. For example, when selecting data for pretraining ($3) may care about data and selection efficiency more than about reducing toxicity in the data. In addition to pretraining, data selection can be used for instruction-tuning (64), alignment ($5), in-context learning (66), and task-specific fine-tuning ($7), where each stage has differing priorities.
2 A Taxonomy for Data Selection
In this section we define our taxonomy of data selection methods. We start by defining concepts for data points and datasets ($2.1), then we describe the components of data selection method our unified framework ($2.2), followed by common dimensions variance in the taxonomy (52.3).
2.1 Background and Motivation of Data Selection
Prior to discussing data selection methods, we first define and describe the "units" " of a dataset, from smallest largest. First, the smallest unit of data for language models is the token, which can be composed of bytes, characters, subwords, larger components, depending the tokenizer." 2
Definition 2.1 (Data point). data point, I(+), is an ordered collection of tokens that constitutes single sample of data used to train or evaluate language model.
For example, in language modeling, ( can be a sequence of tokens from internet document, book, scientific article. Language models often have limited input sequence length when training, so long documents are often split into multiple data points. In practice, this means that z00 may constitute a portion of a document (rather than a complete one).
Definition 2.2 ( (Data point characteristics). The characteristics of data point refer to a number of measures used to describe a data point, 2ff) and determines where ((()) is located within the space of all possible data points.
Common measures used to describe data points can be either simple statistics (e.g., number of characters in z or the percent of alphabetic characters in 200) or distributed representation (e.g., the embedded
How effectively encode is still active research area (Rust et al., 2022), date splitting into takens the leading approach.

---

Published in Transactions on Machine Learning Research (07/2024)
representation using BERT (Devlin et al., 2019)). The data point characteristics are a crucial component of data selection as they are frequently used to determine whether a data point should be cleaned or possibly removed entirely.
Definition 2.3 (Dataset). A dataset, D, is the collection of data points (,(1),..,,,())) (where N = [D] is the number of examples in D) that will be used either train or evaluate model.
Definition 2.4 (Dataset distribution). The distribution of a dataset, D, refers to the distribution of data points within the space of all possible data points. The distribution of 'D has a significant impact on the final capabilities and properties of a model that is trained on it.
The dataset distribution is also a crucial component of data selection as models can struggle with out- of di-iiiriitutio generalization and the dataset distribution specifies what data is in-distribution vs. out-of- distribution (Shi et al., 2023). Specifically, dense region of the data distribution generally suggests that a model will perform well on unseen data from that region (e.g., similar distribution of tokens). With this in mind, increasing the density of data around desirable regions while reducing density around undesirable regions generally leads to a model that performs well in the desired settings.
2.2 A Unified Conceptual Framework for Data Selection
High-level goal of data selection. Data selection is the process of creating a dataset from a collection of candidate data points, which will be usod to train or evaluate a machine learning model. Prior to data selection, generally speaking, data is collected, (optionally) annotated, and stored. Then, once the target use case for model is determined, the goal of data selection is to filter and select the data points that marimize desired objective.
Formal definition of data selection.
Definition 2.5 (Data selection). A data selection function, d, takes as input a dataset Draw and objective function fobj and filters, cleans, and selects data points from Draw to create final dataset D= (DDraw) such that, for model M trained dataset D, & aims to maximize/ maxinize/miiiiize miiiiiiee the objective function Sobj (M).
In practice, multiple selection methods are often composed to improve the coverage and specificity of the final dataset. this case, we denote the component functions as oj and the composition as ((D) = -1 on (D). For example, o Φ1 a 22 o 63 can include a simple filtering component, d3, that first removes undesirable data points, followed by cleaning component, Φ2. that removes undesirable content from within individual data points, followed by a mixing component, d1 that determines the number of times the remaining data points should be used in the final dataset. Through these mechanisms, data selection functions can adjust both the individual data point characteristics and the dataset distribution towards more desirable regions of the data space in the aim of improving desirable propertics of the model for the target purpose.
Components of data selection method. We identify the common components of selection functions by: the utility function and selection mechanism.
The utility function v(x(()) D - R defines a mapping from a data point to real number representing the calculated utility. For example, a utility function may be a binary indicator defined as: 1 if the total number of tokens in _(() is greater than 10, or 0 otherwise. Another utility function might assign z(1) a utility equal to the likelihood of 2(() being a Wikipedia article.
The selection mechanism uses the output from the utility function determine whether a data point will be included in the resulting subset (and in some cases, how many times the data point should be repeated). It can be simple indicator (e.g., only include z(()) in D if the number of characters is greater than 10), probability functions (c.g., if o is the likelihood of (+) being a Wikipedia article, include a(() in D according to the probability defined by v)). Additionally, a selection sensitivity is needed for selection mechanisms that require a threshold (e.g., only include z0(() if (1)(()) > 0.9).
Data selection methods are used solve a wide range of objectives, and by adjusting the utility mechanism, selection mechanism, and filter sensitivity can achieve the desired outcomes. We save discussion of the specific instantiations of each component to the respective sections where they are introduced.
6

---

Published in Transactions on Machine Learning Research (07 /024)
+
+
+
X
X
Distribution Matching
Distribution Diversification
Figure 2: A conceptual demonstration of two common goals for data selection methods: Distri- bution Matching and Distribution Diversification. On the left we see distribution matching where the goal is to select data points (green crosses) that are similar to data points sampled from a target data distribution (blue circles) and reject data that is too far out of distribution (red exes). On the right we see dataset diversification which aims to /rejet data points (green crosses/ /red exes) in such way that maintains coverage over the full distribution while reducing the total number of data points.
2.3 Dimensions of Variance in the Data Selection Taxonomy
Data selection methods can be utilized for various goals, where the goal of the method will, in part, determine the exact utility function and selection mechanisms used. To help form a taxonomy and better understanding of the relationship between methods, we define some specific dimensions of commonality and variance across methods (in no particular order).
2.3.1 Distribution Matching vs. Diversification
The main goal of distribution matching (Figure 2, left) mothods to select data with propertics similar to the desired larget distribution, upon which a model will be evaluated or deployed. For instance, the desired distribution could be defined as data of known high quality, a specific language, or a target domain (e.g., finance, medicine, or law). The exact specifications of the desired target distribution well-defined (e.g., in detecting a language) to being quite ambiguous (e.g.. "high-quality" data). Some distribution matching methods will try to match data representation distributions where the similarity to the target distribution will usually be the utility (e.g., similarity to Wikipedia data). Other distribution matching methods use the statistics of data sampled from the target dataset as the utility (c.g., total number of characters per example).
vary from being
Distribution diversification methods (Figure 2, right) aim to prioritize heterogeneity in sample, removing redundancies. They operate within representation space that allows for similarity to be measured across data points. The utility of data point is defined by its relation (similarity) to the other data points in the representation space. Distribution diversification methods often remove data points that are similar in some representation space (e.g., characlers or vectors). By removing data points that are very similar, diversification methods can remove redundancies in the data, leading to improved training efficiency by reducing the dataset size. Additionally, bocause distribution diversification methods force the datasot distribution assign probability a broader range of datapoints, they can lead to decreased memorization, decreased bias, and improved robustness.
7

---

Published in Transactions on Machine Learning Research (07/2024)
2.3.2 Altering the Dataset vs. Data Point
Methods that alter the dataset aim to increase or decrease the frequency of individual data points within the dataset in order to increase (decrease) the resultant distribution density around desirable (undesirable) regions. These methods take as imput a dataset and assign each individual data point a non-negative integer value representing the number of times that data point should be included in the resultant dataset. Formally, dataset distribution altering function Ø3 : DD- + NC maps each dała point x;(1) E D to non-negative integer based on the utility function, v(z(()). Each datapoint is then filtered (0)(x()) = 0), maintained (x)(1() = 1), or oversampled (0)(x > 0) according to the utility function.
Methods that alter the data point aim to adjust the content within a data point to better match a desirable distribution of tokens so that the individual data points within a dataset more closely resemble the desired characteristics. In practice, this can take the shape of removing individual lines, or chunks of text, from a document. For example, when training model for natural language, it can be beneficial to remove HTML tags as they are likely to be out of distribution if the desired target domain is natural language.
2.3.3 Output Space: Binary vs. Natural Number Selection
Methods that alter the dataset distribution lead to another dimension of variance: whether the selection function & assigns binary values (i.e., include remove) or whether larger integer values are permitted. Methods that only assign binary values are generally referred to as filtering, and the goal is usually to adjust the dataset distribution by removing undesirable data points from the dataset. On the other hand, methods that can assign any natural number are commonly referred to as data mixing, and their goal is often to adjust the dataset distribution by prioritizing certain subsets of the dataset that have high utility values, while decreasing the distribution density around subsets with lower utility. It very common for data mixing methods to define a utility function that assigns the same utility to entire subsets of data (e.g.. all web text gets the same utility, and all books get a different utility). Filtering and mixing can be used part of pipeline, where data is first filtered and then mixed.
2.3.4 Training Stage
Language model training often requires multiple stages where each stage serves a different purpose. Tn this work we identify and discuss five distinct language model training stages during which data selection can be used: pretraining, instruction-tuningg alignment, in-context learning, and task-specific fine-tuning.
Each training stage has different goals, and so the data selection methods for each stage will use different mechanisms to achieve those goals. For example, the larget distribution is fairly well-defined when training a language model for a specific domain or task, but much less defined when pretraining a general purpose language model, thus data selection methods can utilize this information to target better data. Another consideration is the number of candidate data points each training stage. For example, pretraining likely has significantly larger number of candidate data points than rnsttuctiontuuinn motivating the need for efficient data selection methods in pretraining, while instruction-tuning can afford more expensive methods.
Data selection has been studied, to some extent, in all stages of training. However, information on data selection for pretraining is most limited, and therefore we dedicate the main portion of this work to comparing, contrasting, and better understanding the methods of data selection in pretraining, with more concise sections discussing data selection for other training stages.
8

---

Published in Transactions on Machine Learning Research (07/2024)
Pretraining Data Selection Methods
Distribution Matching vs. Diversification
Output Space
Adjust Dataset
Selection Method
vs. Data Point
Language Filtering ($3.1)
M
{0, 1
D D + P D D + P
Heuristic Approaches ($3.2)
M
No
Data Quality ($3.3)
{0,1]
Domain-specific ($3.4)
M
{0,
Deduplication ($3.5)
D
{0,1]
Toxic and Explicit Content ($3.6)
M
{0,1]
D ++P
Multilingual Filtering ($3.7)
M D
No
D D+P
Data Mixing ($3.8)
M M+D D
No
D
Table I: Pretraining data selection methods can be described along three axes of variation: whether they aim to do distribution matching or diversification, what the output space is, and whether they make adjustments the dataset distribution or data point charactoristics.
3 Data Selection for Pretraining
The goal of pretraining is usually to train a "general- purpose" model, which requires training on massive quan- titics of text (often trillions of tokens for recent large language models). Selecting the best data from such large quantities can be very expensive, so common first step in the process is to remove data with various filters, and it is common to apply multiple filters that are pipelined to- gether to achieve the desired dataset. The order in which we present pretraining data selection methods is, roughly, based on the order that they used within real data selection pipelines. Of course, not all pipelines require every method presented here, and depending on the case the exact ordering may differ slightly. We provide a high level overview of the filtering pipline in Figure 4, and references to relevant subsections as well as axes of variation Table 1. The pretaining data selection step illustration depicted in Figure
Learning Stage
Selection Objective @
Pretraining
Redie Rox/Tedatty
Instruction tuning
Allgrment
Indoerraze
Filllwyy
In-Content Learning
Elfceeey
steghtr
Figure 3: Data selection: pretraining. The first training stage of modern language models. Typically, the most important selection objectives are model performance, evaluation integrity, and selection efficiency.
3.1 Language Filtering
When curating data for language model pretraining, a crucial first step to consider the languages the model will operate in and filter data that doesn't belong those languages. This applies not only natural languages, but coding languages multilingual language models, in addition to filtering out undesired languages, it is also important to track metrics the quantity of data coming from each language. Similarly, for models with code capabilities, it is important to track the quantity of data from each coding language.
well, however, the methods to determine each differ in practice. For
Common utility functions. When filtering for language, it is crucial that the utility functions be fast to compute the utility calculation is often performed all available raw text data (potentially huge quantity). Thus, many methods that aim to filter for specific natural languages utilize fast-to-compute ut utility functions from classifiers based on character n-grams (Conneau & Lample, 2019; Wenzek et al., 2020; Raffel et al., 2020; Xue et al., 2021; Laurençon et al., 2022) including LANGDETECT3, CLD34, and FASTTEXT (Joulin et al., 2016; Grave et al., 2018). Some methods aim to remove all non- -on-English data, while others may include over 100 languages and require different utility function strategies.
9

---

Published in Transactions on Machine Learning Research (07 (07/2024)
en
3j
Language Fitering
Heurietics
Data
Domain Specific
Deduplication
Toxic Explioll Content
Data Mbxng
Pretraining Corpus
Quality
Figure 4: An overview of the data filtering pipcline for pretraining. Each filtering component is described in Section 3, and depicts common filters used for preprocessing lext data. Note that different works employ different filters, at different stages, and do not necessarily adhere to the order couveyed here. This figure was adopted and modified from Soldaini et al. (2024).
Filtering for English-only datasets. When developing the English-only C4, Raffel et al. (2020) use a naive Bayes classifier with character n-grams (LANGDETECT) to filter out any pages not classified as English with a probability of 0.99. They find the use of very high threshold sensitivity to be appropriate as the filtered dataset still contains a significant quantity of data (-750GB of data). More recently, when developing Dolma, Soldaini et al. (2024) utilize FASTTEXT's (Grave et al., 2018) language identification model and filter out any documents classified as English with a probability less than 0.5, finding that this removed 61.7% of web pages. However, when filtering books (from Project Gutenberg") they first split each book by paragraph and compute the language score for each paragraph, only removing books that have an average probability of English (assigned by FASTTEXT) of below 0.5. Penedo et al. (2023) also use FASTTEXT when filtering Ref inedWebe but differently to Dolma, they use higher threshold (0.65) and therefore filter out higher quantities of web pages.
The lower thresholds used by more recent methods are likely due to a combination of factors. First, ever- growing model sizes have necessitated larger pretraining datasets, and using a lower threshold will allow more data to pass through the filter. Additionally, it is difficult to measure the differences in accuracy across different language identification models (e.g., LANGDETECT vs. CLD3. vs. FASTTEXT) due to the differences in supported languages and strengths on text of varying lengths, but some experiments show that FASTTEXT has improved accuracy over other language identification models," 6 as well reduced latency. In general, the threshold used for keeping or filtering out data point can be in part determined by the desired datasct size, where a lower threshold will retain higher quantitics of data, but possibly containing non- non-English text.
Filtering for multilingual datasets. Filtering methods for multilingual corpora often rely on the FASTTEXT (Joulin et al., 2016) language classifier from CCNET (Wenzek et al., 2020) which was trained to classify 176 languages using wikipedia data (Grave et al., 2018; Conncau et al., 2020; Laurençon et al., 2022). The FASTTEXT classifier is desirable as it can process 1,000 documents per second on a single CPU core (Wenzek et al., 2020). One example of this method comes from Laurengon et al. (2022) who use a FASTTEXT model to obtain a prediction for the document language along with a prediction confidence score when creating the ROOTS corpus. If the confidence score is below threshold, the document is removed, where the confidence score is determined by native speakers on a banguage-b--languaae basis, but are not disclosed in their paper. The need to determine threshold for each language individually adds significant. amount of additional effort to develop language filters for multilingual corpora. Additionally, these filters can fail on documents in which the language changes several times as the classifier cannot confidently predict a single language. Furthermore, while this classifier was trained on 176 languages, there are 1000s of languages that cannot be covered by this method (van Esch et al., 2022). This has prompted the creation of even broader language detectors covering thousands of languages, however, they still suller from poor precision for extremely low-resource languages (Caswell et al., 2020; Kudugunta al., 2023).
hhttps://wwx http://wwwweebbegg.ggg org/
onttp:://nodeelprriitt wttpe/diittttccaannaeeeeeee com/language-iie wwwaanngugg-iieeniiieee
10

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Another common approach for multilingual filtering is by country domains selecting URLs that are known to contain data in certain languages (Luukkonen et al., 2023). Especially for very low-resource languages, such as Uyghur, this can be a more reliable method than language classifiers (Zhang et al., 2023a).
For further discussion on data selection for multilingual models, see Section 3.7.
Filtering for code languages. The utility functions that have been used to detect code languages are relatively simple compared to those used for natural languages. For example, Chowdhery al. (2023) filter for data from 24 programming languages simply by searching for documents that match set of approved filename extension (e.g., ".py" for python). This strategy works well when filtering over data from known code data that contains file names, such as snapshots of Github repositories (Chen et al., 2021; Li et al., 2022a). However, no filtering methods that we know of find code within natural language documents, which is entirely possible within some domains such as in Stack Exchange' and may contribute meaningful quantities of interleaved natural language and code.
Challenges for language filtering. There are tradc-offs occurring in the filter sensitivity for language detection, where lower threshold allows greater quantities of data to pass through the filter the risk of including lower quality data. On the one hand, using strict filter for English data may reduce the quantitios of non-English data, but on the other hand, using lower threshold can mitigate some of the inherent biases of the language detector against dialects spoken by minority groups (e.g., African-American Vernacular English (Blodgett et al., 2016)). Determining the exact threshold to use for language classifiers is highly dependent on the use case. While FASTTEXT is the current standard for language detection due to its combination of speed and accuracy, it is not perfect. More accurate systems built in on recurrent- (Toftrup al., 2021) and transformer-based architectures have been developed, but run with higher computational costs.
Language Filtering Summary & Idcas
Language filtering an important first step of data selection where documents are selected to include only the desired languages.
Classifier-based methods are the norm when creating both english-only and multilingual datasets.
For lower-resoureed languages, URL-based methods are also useful for classifying the language reliable than classifiers.
used in multilingual datasets and can sometimes can be
Filtering for code languages very simple with most works simply using the file extension.
V One interesting direction for future research is developing methods that find code data within documents of natural language. the best of our knowledge, no such methods currently exist, but code-language identifier could be used in domains with interleaved code and natural language to find meaningful quantities new code data.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Heuristic Category
Common Utility Functions
Example Selection Mechanisms
# of characters in (word/line/ patain/aaaaappp////a/// /docummet)
Remove documents with fewer than words (Raffel et al., 2020)
Item Count
# of gwords/lines/sentenn
in document
Remove lines that repeat the same word more than times consecutively (Laurengon et al., 2022)
Repetition
# of times a [character/ antrractter/ggrraan /nwgra////////eeeeeee/e / sentence/ p is repeated
Count
Whether (word/n-granil is in the document
Remove lines starting with "sign-in" (Penedo et al., 2023)
Existence
Whether a terminal punctuation is at the end of a line
Remove documents with symbol-to- word ratio greater than 0.1 (for "#" and "...") (Rae et al., 2021)
% of alphabetic characters in a document
Ratio
% of humerals/uppercees characters in sine/doccment)
Remove code files that have mean line length greater than 100 characters (Chen et al., 2021)
The mean length (and standard deviation)
Statistics
all lines document.
Table 2: Commonly used heuristic utility functions and demonstrative selection mechanisms.
3.2 Heuristic Approaches
When pretraining language model, raw datasets are generally composed of massive quantities of text from wide variety of sources that filtered through set of simple heuristics. Major models are often trained web scrapes such CommonCrawi and GitHub, though transparency into their precise compositions are on the decline (Bommasani et al., 2023). It is widely known however that raw text data from the internet can contain significant quantities of boiler-plate text, error messages, and offensive text (Raffel et al., 2020; Touvron et al., 2023a; Elazar et al., 2024). For example, Gao et al. (2020) find that, when creating the Pile, the most common 13-grams were character repetitions such as a string dashes (" ") with 11 million instances. Removing such undesirable text very important, but must be done efficiently due the size of the corpora involved. In this case, common approach to filtering data involves simple and efficient-to-compuute heuristics.
The goal of heuristic approaches is to constrain the training distribution along some dimension (e.g., sentence length, repetitiveness). with the assumption that the evaluation distribution will exhibit similar characteristics. The number of hcuristics that have been used in past works extensive, but generally fall into one of the following categories heuristics: item count, repetition count, existence, ratio, or statistics. In this section we discuss representative sample of them, provide Table as an overview of heuristic methods, and include comprchensive listing in Appendix A.
Common utility functions. For heuristic- baasistic-based text filtering, the utility functions should be very fast to compute and generally rely on one or more qualitics of the raw text included in each data point of the dataset. However, the exact text qualities that lead to best performance depend on the desired use case for the model, and are yet unknown. Thus, a wide variety distinct utility functions have been proposed, each making assumptions about the expected testing distribution, and filtering out data that does not fit those assumptions. For example, Raffel et al. (2020) discard any web page with fewer than five sentences, Rae et al. (2021) discard documents with fewer than 50 or greater than 100,000 words, and Xue et al. (2021) discard documents with fewer than three lines of text with 200 or more characters. Another common heuristic is to remove documents with specific black-listed words or phrases. Ralfel et al. (2020) remove documents with any word on the "List of Dirty, Naughty, Obscene, or Otherwise Bad "ords However, Penedo et al. (2023) suggest that the NSFW word block lists as used by Raffel et al. (2020) can often lead to false posítives, resulting in the over-filtering of legal and medical content.
Sataeeene-nndOthhevvee Radeii----------e Words
12

---

Published in Transactions on Machine Learning Research (07/2024)
Another heuristic often associated with the quality of document is repetitiveness. Rae et al. (2021) find that excessive repetition within document is linked to uninformative content and suggest. that filtering out entire documents that have high proportions of repeated lines, paragraphs, and n-grams leads to improved performance. For example, if a repeated line accounts for greater than 30% of a document they remove the entire document. See Table A1 in Rae el. al. (2021) for their list of 13 different repetitions and threshold sensitivities.
Domain-specific heuristic approaches. Another dimension to consider when designing filter is the domain of interest. For example, when collecting code data for the ROOTS dataset, Laurençon et al. (2022) filter source files between 100 and 200,000 characters, with between 15-65% alphabetic characters (a-zA-Z), and line lengths belween 20-1000 characters. If these filters were applied on natural language data, significant quantities of desirable data would be removed (e.g., many books have greater than 65% alphabetic characters). Additionally, Laurençon et al. (2022) filter out files that have line lengths that are highly uniform (line length standard deviation 3 characters or less). While filtering out data with uniformly lengthed lines may improve data quality for code, it would likely filter out significant quantities of poems and other rhythmic text due to the similarity in line length, and would likely remove data that has been collected from pdf files, which could be desirable when training a model for a non-code domain.
Adjusting data points with hcuristics. While the proviously described approaches remove entire data points from the training dataset, removing content from within data point can be useful method. Rather than removing entire data point, heuristic met hods that remove individual lines within larger document aim to improve the coverage of the training data distribution, while limiting the density around undesirable qualitics. For example, when working with content from web crawled data, removing HTML can be an important part of the filtering process that helps to reshape the data distribution towards natural language (assuming that the desired outcome). Laurençon al. (2022) and Aghajanyan et al. (2022) create lists of HTML tags where all content within sub-trees of the tags will be removed. Additionally, Raffel et al. (2020) remove lines that do not end in a terminal punctuation mark and lines with fewer than four words, and Penedo et al. (2023) remove lines that are mainly composed of uppercase or numerical characters. Furthermore, a common component of line-by-line filtering is the removal of the entire data point if enough of the individual lines within it are filtered out. For example, Penedo et. al. (2023) remove the full document if the individually removed lines account for greater than 5% of the total document length.
Deterministic vs. stochastic selection mechanisms. Many heuristic utility functions can be used as part of deterministic or stochastic selection mechanism. In a deterministic selection mechanism, any data points which are above/below the utility threshold will be removed. The vast majority of existing heuristic filters use a deterministic selection mechanism. However, this may unnecessarily remove data that resembles the evaluation distribution (Brown et al., 2020a). In particular, the target dataset distribution may be long-tailod in the characteristic of interest, and deterministically removing all data above a threshold can harm the modelling of that characteristic.
Heuristic filters commonly based on item counts, repetitions, ratios, and statistics can all utilize stochastic selection mechanisms. For example, Rac et al. (2021) remove documents with symbol-to- word ratio greater than 0.1, but a stochastic selection mechanism would allow for some data with low utility to not be filtered out. While a stochastic selection mechanism may not always be appropriate, there are certainly some heuristics where it may benefit the dataset distribution, but the exact settings where a stochastic selection mechanism is an open research question. Data quality filters ($3.3) often use stochastic selection mechanisms, and applying stochasticity in heuristic filters is one area of data selection that has not been explored.
Drawbacks of heuristic selection approaches. One disadvantage to heuristic selection approaches is that they do not judge the quality or content of a document, rather, they rely entirely on surface level statistics and counts, lacking the "finesse" of other data selection methods. This can lcad to desirable data being removed, but this often accepted as a trade-off as long as the quantity of data after filtering is still within the desirable range.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Another disadvantage of heuristic filters is the challenge of validating their effectiveness. After heuristic has been developed and used on raw dataset, to determine whether the heuristic is improvement requires either: 1) manually validating that the filtered documents are undesirable and the kept documents are desirable, or 2) training and evaluating a model. Both of these methods are time consuming and expensive, which make experimentation on heuristic filtering approaches very slow.
Considerations when designing a heuristic filter. When designing set of hcuristic filters, is very important to understand the data source. By first looking the data and validating the performance of heuristic filter small sample of data, the designer of such filter better understand what type of data the method will remove, and what types of data will remain. That way, by composing multiple heuristics in succession, the resultant dataset will reflect the desired data qualities. There are a wide variety of existing tools to better understand the data within a dataset (data auditing), as well as for data selection. See $10.4 for some useful tools.
The current known best practices for heuristics have not changed significantly in the past few years. Soldaini et al. (2024) recently use combination of the filters from MassiveText (Rae al., 2021) and C4 (Raffel et al., 2020), while Penedo et al. (2023) also follow the rules defined for MassiveText (Rac al,, 2021). However, it is unclear whether the lack of progress in common heuristics is due to lack of exploration in the space (due to limited resources and studies), or if the known heuristics truly are optimal. To the best of our knowlodge, there have not been studics spocifically comparing the many possible hcuristics in a systematic study.
Hcuristic Approaches Summary Ideas
Heuristics are very efficient method for removing large quantities of data, but lack finesse.
Most common heuristic utility functions fall into categories: item count, repetition count, existence, ratio, and statistics.
Henristics should be designed specifically for the domain of interest (e.g., heuristics for code may not work for natural language)-
Heuristics also be used to improve data point characteristies (e.g., removing HTML from web document, or specific lines within a document).
Because heuristics lack finesse, it may be worth exploring the use of stochastic selection
mechanisms to allow higher quantities of data through the filter (with the hope that filter farther down the pipeline will remove data that is actually undesirable).
Validating quality of heuristic filter is very slow process, requiring either manual inspection, training and evaluating a model. Developing methods that can directly measure the data itself could significantly improve iteration time.
14

---

Published in Transactions on Machine Learning Research (07/ (224))
3.3
Data Quality
Training on the highest quality data lead stronger performance (Du et al., 2022). However, "high-quality" is not well-defined term with respect to language model pretraining data. Additionally, there is no one-size- fits-all characterization as, depending on the use case, "high-quality" data can vary drastically (Longpre et al., 2023e). Additionally, implementations of quality filters (e.g. for detecting loxicity) can further marginalize communities whose text is not commonly considered "high quality" (Xu et al., 2021; Longpre et al., 2023c). In this work, we narrow the use of the phrase "high-quality" to a common use of the phrase referring to data that known be written by humans and has likely gone through an editing process (Grave al., 2018; Gao et al., 2020; Brown et al., 2020a; Chowdhery et al., 2023). Some data domains that fall under the "high-quality" category are Wikipedia, books, patents, and peer-reviewed journal articles.
Common utility functions. Similarly to heuristic methods, the utility functions used for data quality filtering should be fast and cheap to compute, however, in contrast with heuristic filtering approaches, quality filtering requires fuzzy matching which generally has higher computational requirements. Thus, methods for data quality often use relatively cheap distributional representation methods, such as n-grams, to allow for more ambiguous filtering criteria.
One commonly used method when filtering for quality is classifter-based quality filtering, where the goal is to identify data points that are likely from the same (or similar) distribution as a known "high-quality" corpus of data points (reference corpus) (Longpre et al., 2023c). For example, Brown el al. (2020a) filters a version of Common Crawl based on similarity to "high-quality" reference corpora (including WebText (Radford et al., 2019a), Books1, Books2, and English wikipedia). They train a classifier using the "high-quality" reference corpora as the positive class and unfiltered Commnn Crawl documents as the negative class. This classifier then defines the utility metric for selecting data that follows the desired distribution. The classifier used to assign quality scores to each document should be fairly lightweight due to the scale of pretraining datasets. For example, Du et al. (2022) use a feature hash based lincar classifier, while Brown et al. (2020a) and Gao et al. (2020) use FASTTEXT model (Joulin et al., 2016) with an n-gram size 2. Similarly, Xie et al. (2023b) computes the utility as an importance weight between two hashed n-gram generative models. While modern language models may be able to better model the reference corpus distribution, they would also significantly increase the amount of compute required over hashing and n-gram based methods.
A competing method of computing utility is perplexity-based quality filtering, where the goal is to train a language model on the reference corpora and evaluate on the data to be filtered, assigning a utility score to each candidate document. For example, Wenzek et al. (2020) train a 5-gram KNESER-NEY (Heafield, 2011) model on Wikipedia, and compute the perplexity of each paragraph their Common Crawl dump. Paragraphs with low perplexity are assumed to be close to the reference domain while data with paragraphs with higher perplexily are assumed to be of low quality.
Recently, Wettig et al. (2024) have explored very different direction, and create utility function from the outputs of language model system (GPT-4) by asking it to rate multiple documents along various dimensions of perceived quality. While their work shows that the model-based ratings can be a good signal for quality, it can be costly to make enough API calls to such systems. Additionally, biases from the system may be carried over into the quality signal.
Selection mechanisms. One potential negative impact of classifier-based quality filtering is that the reference corpora are unlikely to contain examples of all data that are considered high quality. Thus, it is desirable to allow some stochasticity in the selection mechanism. For example, Brown et al. (2020a) use their classifier to score Common Crawl documents and keep cach document only if they satisfy
> 11 -oouumnnt_soo score,
where a = 0.9 was determined by matching the distribution of scores from the WebText corpus (Radford et al., 2019b). By selecting documents that don't match the "high quality" distribution with some stochasticity,
15

---

Published in Transactions on Machine Learning Research (07/2024)
they ensure that documents in the final corpus mostly high-scoring, but still include some documents that are out of distribution with respect to the reference corpora.
When the utility metric is an importance weight (Xie et al., 2023b), the selection mechanism is to sample examples according to the importance weights across the dataset. Sampling (without replacement) according to the importance weights can be done efficiently with the Gumbel top-k trick, which perturbs the unnormalized log-importance weights with Gumbel noise before selecting the top k.
Potential challenges for quality filtering. An important consideration when performing quality filtering is that certain reference corpora may be biased towards or away from certain demographics, dialects, and socialects (Rac et al., 2021). Additionally, Gururangan et al. (2022) find that a quality filter trained on Wikipedia, books, and newswire (a. replication of the quality filter from Brown et al. (2020a)), evaluated on high school newspaper articles, prefers articles from high schools in wealthier, higher-oducated, and urban areas. They demonstrate that even though quality filtering is often assumed to be fairly neutral, carries with it an implied judgment of preferred values. Thus, quality filtering should be done with care ensure that biases are avoided and the distribution of data sufficiently covers the desired dialects and demographics. Filtering text for quality, while avoiding the introduction of biases is an important direction for future research.
It is still an open question whether using quality filters is a beneficial practice. Some recent works that have entirely foregone using a quality filter include MassiveText (Rac et al., 2021), Ref inedWeb (Penedo et al., 2023), and Dolma (Soldaini et al., 2024). Specifically, Penedo al. (2023) elect not to perform any quality filtering and spend more compute on a stringent deduplication instead. It is not fully understood when quality filtering is beneficial and when it is not required, requiring further research.
Data Quality Summary & Ideas
Data quality methods aim to select data points that similar to data which is of known "high-quality".
Classifier- and perplexity- quality filtering are the existing approaches, with classifier-basee filtering being the popular method.
For both classifie-- and perplexity-based methods, it is important to be careful when training the classifier because this can introduce biases (e.g., against lower-income, less-educated populations).
Exactly which situations quality filtering beneficial is still undecided, as recent works have trained performant models without using quality filters at all. This is an area that requires further study.
16

---

Published in Transactions on Machine Learning Research (07/ (204))
3.4 Domain-Specific Selection
While some language models pretrained for general domain use, they also be trained with a specific domain in mind, where selection can assist in finding data similar to the desired domain's 's distribution (e.g., medicine or law). Domain-specific filtering methods mostly assume access to some in-domain data and additional data from auxiliary sources. The goal of filtering is then to find the auxiliary data which most resembles the distribution of in-domain data.
Common utility functions. In order to produce representations for data that can separate in-domain vs. out-of-domain data, utility functions for domain-specific selection often make use of models trained on one or both data distributions.
Many demain-specifie selection methods stem from MOORE-LEWIS selection (Moore & Lewis, 2010) which follows from the following conditions. Let I be an in-domain dataset, N be general purpose dataset, and N₂ be a subset of N that is in-domain that we wish to discover. They note that the probability of a data point ((0) drawn randomly from N being in Ny is
P(N .N) ,N) (PP(NIN [N)
derived from a variant of Bayes rule. If the probability ((Nr .N) could be directly measured, it would form a very strong utility function. The exact distributions are intractable, but estimates for P(x(9) I) and P()()
can be calculated by training language models on I and a sample of N. Additionally, the probability P(N)(N)
can simply be disregarded because it does not depend on x.(1) so for all x(00 11)
(PP(NN))
P(x N) O P(
In order to work directly with language model outputs, the ratio (T)
PNx(1|N is converted into the log domain
as (o - log( (l)) Finally, the cross-entropy loss from models trained on I and N is used estimate each of these values.
More recent variants of MOORE-LEWIS selection have improved on the original idea. Axelrod (2017) propose cynical data selection, theoretical framework for selecting data that maximizes the information gained by the model from each data point. Feng el al. (2022) empirically validate that the perplexity of encoder models improves when using cynical data selection compared to a "high-quality" subset (Wikipedia and Books). Xie et al. (2023b) reframe the ratio of probabilities as an importance weighting from an in-distribution and general purpose model.
Notably, while the original MOORE-LEWIS selection method suggests using language model to estimate the utility function, modern language models have grown to sizes that would make the naive use of this method extremely costly. Thus, more efficient language models based on n-grams are typically used. For example, Xie et al. (2023b) utilize a bag of hashed n-gram model that can be computed very efficiently and find that data selection with hashed n-gram features strongly correlated with performance on target domains.
Engstrom et al. (2024) propose depeedde alternative MOORE LEWIS- methods, which only depends on the data. They propose use datamodels (Ilyas et al., 2022), a utility function that can approximately map subset of training data to model performance after training that subset. Generally, producing datamodel requires training series of models with different subsets of data. At very high filtering ratios (less than 25% of original dataset size), they demonstrate improved performance over DSIR (Xie et al., 2023b), but require significantly more compute.
In general, methods derived from MOORE- LLEIS selection have been the gold-standard, but it may be possible adopt other methods of data attribution and valuation, similar datamodels, for the purpose of selecting in-domain data. However, these methods typically have higher computational requirements, a challenge that needs to be addressed. Perplexity- methods (similar to perplexity- based quality filtering methods ($3.3)) that only require an in-domain model are also a plausible research direction.
Uhttps://githut httpgithub..ooopp..eee lanida/dair
17

---

Published in Transactions on Machine Learning Research (07/2024)
Selection mechanisms and thresholds. Domain-spoeific filtering methods utilize either deterministic or stochastic selection mechanisms. Most MooRF-LRwiS--aweee methods tend to use deterministie selection with threshold. As previously mentioned, domain-specifie filtering methods often assume access to in-domain data in addition to the auxiliary data. In order to select an appropriate threshold sensitivity, a held-out set of in-domain data can be used (Sethy al., 2006). is also possible to use stochastic selection mechanisms. For example, DSIR (Xie al., 2023b) resamples the data according to the importance weights. A stochastic mechanism can allow for a wider diversity of data, especially when there are minority subpopulations of the target that may be ignored when deterministically selecting high-scoring data under some metric.
Comparison to data quality filters. Domain-specifie filtering methods quite similar to data quality filterigg and can be viewed as a generalization of data quality filtering. The core difference is that in data quality filtering, the in-domain data (reference corpus) is used as a proxy for the desired distribution but in reality this not guaranteed, thus an element of stochasticity is crucial for data quality filtering, On the other hand, given enough in-domain data, domain-specific filters can directly measure whether auxiliary data matches the desired distribution.
Challenges for domain-specific selection. Prior methods of domain-specific selection utilize just model of the in-domain data, but these methods can skew the resultant distribution towards the modes of the existing in-domain data (Sethy et al., 2006). For example, it is possible that the in-domain data contains some very common phrases (e-g., "okay") which are not necessarily representative of the domain. To avoid this issue, possible to perform deduplication on the in-domain data. However, a delicate balance must reached, as deduplication can skew the in-domain data away from valid modes. Domain-specife selection methods that utilize general-domain model in addition to the in-domain model (e.g., MOORE-LEWIS selection) handle this issue quite well and thus it is highly recommended to such methods.
Domein-SSeccifc Selection Summary & Ideas
Domain-specific selection aims to find data most similar to a specific domain (e.g., medicine or law).
MOORE-LEvEE style methods (requiring one in-domain model and one general purpose model) have stood the test time and are still the most popular methods for domain-specific filtering. v Datamodels and other methods based data attribution and valuation be explored as competitive methods, but generally have higher computational requirements.
Perplexity-based methods may be another direction to explore. They require only an in-domain model, which could improve efficiency. It's possible that they would not perform well as MOORE- LEWIS based methods, but because they use less compute they could be used in multiple rounds of selection, in bootstrapping.
18

---

Published in Transactions on Machine Learning Research (07/ (07/2024)
3.5 Data Deduplication
Datasets originating from internet dumps often abound with duplicate and near duplicate documents (Elazar et al., 2024; Magmusson et al., 2023). A dataset with duplicated data, or near duplicates, increases the distribution density around those areas. In cases where these high-density points are of high value, keeping near duplicates can be beneficial. However, for pretraining where the exact evaluation distribution is unknown, it is generally preferred to remove duplicates so that the training distribution provides greater coverage with less rodundancy. Carlini ct al. (2023) and Kandpal al. (2022) demonstrate a diroct relationship between the number of times data point is duplicated and the degree to which a model will memorize the data point, a relation which increases with model scale. Additionally; filtering out duplicates can reduce the training time of machine learning models (Sorscher et al., 2022) and has been shown to sometimes improve accuracy on downstream tasks (Lee et al., 2022a; Tirumala et al., 2024).
Common utility functions. Deduplication can come in multiple forms, where most approaches are based on URLs, hashing, string metrics, and model representations. Some methods find exactly matching strings and documents while others use approximate matching methods calculated according to some similarity measure.
Exact match utility functions. Determining eract matches over whole documents is fairly straightforward, and can be performed cheaply. URL deduplication is a common first step when using a snapshot of crawled web pages. It is possible for individual web pages to appear multiple times in the snapshot, so removing data that shares the same URL is a very cheap and efficient method for initial deduplication (Agarwal et al., 2009; Soldaini al., 2024).
Some exact-matching utility functions utilize hash functions. These hash-based methods are guaranteed to find all exact matches but, due to the possibility of collisions, may also find false positives (accidentally removing non-matching documents), although Elazar et al. (2024) demonstrate that a simple hash function on large scale datasets can experience no collisions. CCNET (Wenzek et al., 2020) performs deduplication by first normalizing all text, then computing a hash code for each document, and finally comparing the first 64-bits of SHA-1 digits of the hashed documents. When developing OSCAR, Ortiz Suárez et al. (2019) use a simple non-collision resistant hashing algorithm to determine whether whole documents are exact matches. Another common method for deduplication is the Bloom filter which utilizes spaco-efficient bit array to compare hashed documents (Bloom, 1970; Soldaini et al., 2024).
Determining ezactly matching tezt segments, as opposed to whole documents, is more expensive, but equally as important to remove. Lee ct al. (2022a) propose the EXACTSUBSTR algorithm that finds examples which share sufficiently long substrings. In their case, their utility function determines that a pair of examples are duplicates if they share a 50-token substring. To perform the deduplication efficiently, they utilize suffix arrays (Manber & Myers, 1993) which allows them to find duplicates in linear time. This method raises the question: how much text surrounding the duplicate should be removed? Lee et al. (2022a) remove only the exact substring that is duplicated, while Penedo et al. (2023) experiment with dropping the entire document, or masking the loss of the duplicated span, finding that all methods lead to similar downstream performance. Similar to the use of Bloom filters for matching entire documents, they can also be used to detect matching segments of any size.
Approximate match utility functions. Removing approximately matching docunents is generally more expensive to compute than exact matches, and is often performed as a secondary step after exact match deduplication (Laurençon et al., 2022).
Similarly exact matching methods, hash-based methods are very commonly used for approximate matching methods. For example, MINHASH (Broder, 1997) is an approximate hashing algorithm that excels at finding templated documents (e.g., licenses with only specific entities differing or placeholder SEO text repeated across websites). Modern deduplication methods use other variants of hashing such as MINHASHLSH (Brown et al., 2020a) and SIMHASH (Charikar, 2002) that utilize locality sensitive hashing." 12 The general method
12 See the blogpost by Mou (2023) for implementation details on MINHASHI.SH.
19

---

Published in Transactions on Machine Learning Research (07/2024)
behind hashing methods for deduplication is to first group small segments of each documents (often called shingling), then to encode each segment using large number of hashing functions into features. Next, aggregate the features (e.g., MINHASHLSH keeps only the smallest features), Finally, documents can be compared with some method of similarity computation between features (e.g., MINHASHLSH uses a Jaccard similarity and removing documents below some threshold. When creating MassiveText, Rae et al. (2021) use MINHASH with 13-grams and 450 hash functions and set the jaccard similarity threshold at 0.8, randomly removing either of the documents that are determined to be duplicates. In contrast, Gao et al. (2020) use only 10 hash functions, but Rae et al. (2021) find that the more aggressive deduplication leads to better model performance.
In addition to hashing techniques, string metric-based methods can be used for futher filtering, although they are generally more computationally expensive. For example, Lee et al. (2022a) compute an edit similarity metric based on token sequences using their token edit distance. Furthermore, when filtering code files, Chowdhery et al. (2023) find duplicated files based on a Levenshtein distance.
Finally, model-based approximate matching techniques utilize learned representation of data points to deduplicate similar data. These methods can be very costly, but allow for more flexibility by considering the semantics of text. For example, SEMDEDUP (Abbas et al., 2023) uses a pretrained 125M OPT model and calculates a representation for each data point in the C4 dataset by using the last layer embedding of the final token in the document. Then, they cluster data points to create groups of rough similarity, followed by computing the pairwise cosine similarity of data points within each cluster. Document De-duplication and Diversification (D4) (Tirumala et al., 2024) go even further, using a pipeline of first deduplicating documents with MINHASH, then applying SEMDEDUP (Abbas et al., 2023) and clustering the remaining data using K-means, and finally applying the SSL prototypes method of Sorscher et al. (2022) which removes the "most prototypical" examples in each cluster. The authors demonstrate that deduplication with multiple methods leads to improved performance as each method is designed to remove only some aspect of similarity. One concern with these methods is that they have only been tested on CommonCrawl data, which is known to be quile noisy, and have not been demonstrated to be beneficial on cleaner datasets. Prior work has shown that the embedding performance of pretrained decoder language models such as OPT 125M is suboptimal (Muennighoff, 2022). Thus, promising future direction for this line of work is using better embedding models that are specifically trained for clustering (Muennighoff et al., 2022a; 2024; Xiao et al., 2023).
Challenges for data deduplication. One concern with hashing-based method for approximate deduplica- tion is that because hashing functions do not consider the order of each individual shingle (the chunks of text. used to create the hashed representation), they act similarly to bag-of-word models where long documents are more likely to have similar representations. Laurençon et al. (2022) found that this led to long documents having a higher percentage of false positive duplicates. Thus, they did not remove documents in a cluster of near-duplicates if a document is longer than 6000 characters. Instead, they use substring deduplication based on the suffix array (as in Lee et al. (2022a)) for documents with more than 6000 characters.
Methods can search for duplicated documents, or just chunks of text which are duplicated within and across documents. Once duplicated chunks have been found, the question arises: what to do with these chunks? The EXACtSUBSTR method Lee et al. (2022a) removes only the exact substring that is duplicated and Penedo et al. (2023) also elect to remove only the duplicated span and keep the rest of the document. This can lead to incoherent documents, but Penedo et al. (2023) empirically find it to work better than removing the entire document. Another option is to loss-mask the duplicated tokens (prevent the loss from being computed on duplicated text), but this adds code complexity as the start and end of each duplicated span will need to be tracked and considered when training the model.
20

---

Published in Transactions on Machine Learning Research (07/2024)
Data Deduplication Summary & Ideas
Deduplication is a very important step and often used multiple times within the data selection pipeline: first with simple URI-based filter, then with more complicated hashing- and model-based methods.
Deduplication generally relies one four methods: URLs, hashing, string metrics, or model representations.
Bloom filters are often used as state-oftthe-ar exact deduplication (even though they can remove mon-duplicated data) they are very space efficient.
For more thorough deduplication, approximate matching methods can be used, where MinHashLSH currently the most commonly used method.
In settings where data efficiency the primary goal, model- deduplication has demonstrated good results. However, these methods are relatively new and need further study to demonstrate their benefits other cases (e.g., when performance is the primary goal).
3.6 Filtering Toxic and Explicit Content
Practitioners often aim to curate dataset that is "unbiased and fair". As such, the aim is for models trained such data to follow similarly unbiased behavior. Certain popular datasets have also been found to have illegal or extremely inappropriate content, which motivates strict data filters (David, 2023). Towards this purpose, most datasets employ a process to filter out documents that do not conform with such behaviors. Bias and fairness are one of the least well-defined filtering requirements and may be in conflict with different idcologies (Gururangan et al., 2022) and implications (Longpre et al., 2023c). A crucial aspect of this topic is the lack of transparency (Bommasani et al., 2023), requirements, and the downstream effects of applying bespoke bias filtering to pretraining data. Many works solely provide minimal information about the heuristics used for filtering out "biased" data without reporting any ablation on the impact on downstream tasks. A notable exception is the work of Longpre et al. (2023c), which controls and ablates for some of these filters to comprehensively study their effects. They find that using toxicity filters (specifically, Jigsaw's 's Perspective AP! leads to models that produce less toxic texts, however, the model's ability to detect toxic content also decreases. As such, one must be careful when using such filters depending on the end-application.
In what follows summarize the filters related to bias and fairness used in different papers based the filter types. As mentioned above, such filters are often based on heuristics and the effect of such filtering is not well studied.
URL-based methods. Simple filters that aim to remove NSFW content can filter solely based the source URL, piece of metadata that is typically stored alongside the text of datasets. Datasets such RefinedWeb (Penedo et al., 2023) filter out documents coming from URLs that match words from curated list, as well as from blacklisted domains.
Lexicon-based methods. One of the most common methods simply filters documents containing words phrases that match a lexicon. It is often unclear what appropriate threshold is for this strategy (e.g., whether single match, or multiple matches, are required to justify the exclusion of an entire document),, but many datasets use such methods (Raffel et al., 2020; Xue et al., 2021; Achiam et al., 2023). Laurengon el al. (2022) go further and adapt such lexicon based on the detected language, to tailor such toxicity filters across multiple languages.
Classifier-based methods. Another common approach is to score documents using a classifier that aims to detect "toxic" documents. Such classifiers are often trained using a simple classifier (e.g., a lincar model over bag-of-words embeddings) so can be applied scale on a large dataset. Others use an external,
21

---

Published in Transactions on Machine Learning Research (07/2024)
blackbox tools for classifying toxicity, e.g., Jigsaw, Google's SafeSearch" 14 (Rae et al., 2021; Longpre et al., 2023c).
Perplexity-basee methods. A less common method is to score documents using a model trained only on the undesirable content. Jansen et al. (2022) train a model directly on adult and harmful content and then use the model's perplexity as utility function where documents with a high perplexity are assumed to not contain any adult or harmful content.
Personally Identifiable Information obfuscation. Finally, Personally Identifiable Information (PII) is another source of concern, which may expose the personal information of individuals. Frequent targeted PIIs are phone numbers, email addresses, IP addresses, and secret keys. A common approach for handling PIT is to detect them using regex (Subramani et al., 2023; Elazar et al., 2024; Soldaini et al., 2024; Groeneveld et al., 2024) or classifies (Allal et al., 2023). Once PII terms are detected, a common practice is to obfuscate them with special tokens.
Toxic and Explicit Content Filtering Summary & Ideas
There a wide variety of toxic and explicit content that are frequently considered undesirable to have in training evaluation data.
Common methods for filtering out toxic and explicit content include blacklisting certain web domains, blacklisting certain words, using classifier detect harmful content, using perplexity to detect harmful content, and removing PIl with regular expressions classifiers.
v There have been limited studies on the effects of these filters, warranting further research on the impacts such filtering to understand how they affect the distribution of data and the downstream models behaviors.
v A useful direction for research how to mitigate any negative impacts of this filtering, or conversely, how can toxic and explicit behaviors reduced other stages of model development (e.g., alignment), or possibly even at evaluation time through decoding methods.
22

---

Published in Transactions on Machine Learning Research (07/2024)
3.7 Specialized Selection for Multilingual Models
Besides filtering for the desired languages detailed in Section 3.1, there are additional considerations when dealing with multilingual pretraining data.
Language-specific filtering hyperparameters. Many selection methods, such as deduplication, can be reused across languages (Laurençon et al., 2022). However, is critical to adapt the hyperparameters for the respective language and often manually tune them (Laurengon et al., 2022; Scao, 2023; Scao et al., 2022; Workshop et al., 2022). For example, while heuristics based technical characters ([0-90+/(2)) valid, for length-based filter (discussed in $3.2) one needs to consider that languages like Chinese have much higher information density per character leading to the same content requiring fewer characters than English. Thus, in a filtering pipeline Chinese would require a smaller cut-off value for minimum text length than English (Scao, 2023). Additionally, when using language identification models (e-g-, FASITEXT EXT) to detect the language of a document, is important that the threshold for removal determined by native speakers, and specific to each language (Laurençon et al., 2022). Not only is the language ID used for determining the language of document, but it can also be used at the sentence level, where Kudugunta et al. (2023) consider removing sentences if the sentence level language ID and document level ID do not match.
still
Script-based filtering. Languages with non-Latin scripts (e.g., Mandarin, Arabic, and Hindi) commonly suffer from incorrect encoding which may require additional filtering noise correction (Kudugunta et al., 2023). Further, some scripts such as Dongba symbols are not yet supported by Unicode, making the selection of such data especially difficult. Using script-based filters can also serve as language identification for languages that have a one-to-one mapping with a script, such as Japanese Hiragana ($3.1).
Manual selection. Due to difficulties in language identification and data availability, noisy language clsssfications tend correlate with language rarity (Kudugunta et al., 2023). Thus, is often necessary manually inspect multilingual datasets to ensure language identification is correct and data in low-resource languages is useful (Kudugunta et al., 2023). Kreutzer el al. (2022) demonstrate that simply ignoring this issue has led to at least 15 corpora with no usable text, while a significant portion of the available multilingual datasets contain less than 50% acceptable quality. While largo-scale collaborations on projects such as ROOTS (Laurençon et al., 2022) have been able to outsource this to speakers of each language, this filtering step can also be effective when done by people who do not speak the respective language. Using translation services to gauge the quality and looking for irregularities is possible even for non-speakers (Kudugunta et al., 2023).
Specialized Selection for Multilingual Models Summary & Ideas
Many of the selection methods from other sections this work can simply be reused when filtering for multilingual data (e.g., deduplication, heuristics on technical characters, data mixing), but some require special parameters for each language (e.g., length-bases heuristies).
Particularly for low-resource languages, native speakers must be incorporated into the filtering process to ensure that the data maintains an acceptable quality. Aya (Singh et al., 2024) is successful example how this type of inclusion and collaboration can look.
23

---

Published in Transactions on Machine Learning Research (07/2024)
3.8 Data Mixing
Large pretraining corpora often comprise a mixture of data from k domains, where domains are typically defined by provenance. The weighting across domains are specified by the domain weights a E A which comprise a k-dimensional probability vector. For example, the Pile dataset (Gao et al., 2020) is composed of 24% web data, 9% Wikipedia, 4% GitHub, etc. 15 The choice of domain weights can result in significant differences in downstream accuracy (Xie et al., 2023a; Albalak et al., 2023a; Xia et al., 2024a; Fan et al., 2023). Given a fixed training budget, data mixing mothods optimize the domain weights improve training efficiency and model performance.
Utility function and selection mechanism. Data mixing methods work with a constrained utility function and selection mechanism. We define domain as a set of data points D S D. Given a set of domains Di,..., in all data mixing methods, the utility value v(x) for a data point the normalized domain weight Drl In particular, the utility value v(x) is constrained to be equal to e(y) whenever I, are elements of the same domain. Consequently, any two domains with nonemply intersection would have the same utility, and thus data mixing methods typically work with domains that are mutually disjoint. The selection mechanism simply samples data points according to the utility function (which is a valid distribution over data points) in a hierarchical manner. In particular, to sample a data point, we first sample a domain index I ~ Categorical(a) and then sample a data point uniformly within that domain Dr.
Dj is equal to
Common baselines. Using heuristic or mamually determined weights is a common baseline. Defining the domain weights according to the natural domain sizes weights all the individual data points equally (Raffel et al., 2020). Beyond this, popular datasets and models (e.g., the Pile (Gao et al., 2020), LLAMA (Touvron et al., 2023a)) often use heuristics to adjust the data mixture, upweighting domains with intuitively higher quality text that have likely gone through editing process, such as books and Wikipodia articles. However, these -entelllydeterrmined domain weights are likely not optimal (Xie et al., 2023a; Albalak et al., 2023a).
Empirically determined weights provide another common method, where the domain weights are tuned according to the performance of a model on downstream tasks. This approach tends to be expensive since it may require pretraining many models within a weroth-order optimization algorithm or heuristic search strategy. For example, the GLaM dataset (Du et al., 2022) sets mixture weights based on (1) the downstream performance of smaller models and (2) the sizes of cach domain, preventing small sources such as Wikipedia from being over-sampled. While the size of these smaller models is unknown, these models must be large enough to have a meaningful downstream performance (roughly 1B parameters). Similarly, the dataset used to train GOPHER (Rae et al., 2021) used domain weights tuned by sweeping over 7 combinations of domain weights and choosing the one that leads to the best downstream performance.
Overall, the optimal domain weights may not be intuitive and likely depend on all parts of the training pipeline, including the tokenizer, dataset, model architecture, optimizer, and other hyperparameters. For example, recent work has shown that repeating data can provide similar benefits to adding new data, up to a limit (Muennighoff et al., 2023b). Muennighoff et al. (2023b) also find that including large proportion of code (e.g., 50%) does not harm natural language performance, while having the additional benefit of improving performance on reasoning-base based tasks.
Principled approaches. Recent works have derived their data mixing methods based on well-studied fields and concepts including information theory, importance sampling, distributionally robust optimization, and multi-armed bandits. We divide these methods into two categories: methods that determine the weights soparate from model training (offline), and those which determine the weights during model training (onlino).
Offline data mixing methods optimize a static set of domain weights, which define a new reweighted dataset for training a language model. DOREMI (Xie et al., 2023a) optimizes domain weights without using downstream tasks by aiming to achieve a low loss on all domains, formalizing this problem as group distributionally robust optimization (GROUP DRO (Oren et al., 2019; Sagawa et al., 2020)). To avoid the pessimism of DRO, DOREMI optimizes the excess loss of each domain with respect to a pretrained reference
15The mixture weights depend on the tokenizer.
24

---

Published in Transactions on Machine Learning Research (07/2024)
model. DOREMI first optimizes the domain weights with small proxy model, then uses the optimized domain weights (the time-averaged domain weights over the optimization trajectory) to train larger model. DoREMr uses a multiplicative update by an exponential function of the excess loss, which is derived from online mirror descent and is equivalent to the exponentiated gradient/ /Hedge algorithm update. Similarly, DoGE (Fan et al., 2023) optimizes the domain weights with a two-stage process that generalizes across model scales, with the same update rule. However, instead of optimizing the excess loss, DoGE optimizes the gradient alignment between each domain and all other domains, with the goal of putting higher weight domains with better generalization or transfer to other domains.
Online data mixing methods change the domain weights for sampling examples during training, allowing for the algorithm to enforce an ordering on the training data. SKILL-IT (Chen et al., 2023c) uses similar multiplicative weights update to the offline methods, but additionally scales the losses by a matrix that captures pairwise interactions between domains. This matrix is learned from data by training many models on pairs of domains. SKILL-IT considers both domains defined by task or "skill" as well as by provenance. SHEAREDLLAMA (Xia et al., 2024a) uses an online variant of DoReMi (Xie el. al., 2023a) that replaces the pretrained reference model with estimates of the reference model loss, computed using domain-level scaling laws fitted on the Pythia (Biderman et al., 2023b) suite of pretrained models. ODM (Albalak et al., 2023a) proposes an efficient online data mixing method based on EXP3 (Auer et al., 2002), an extension of the Hedge algorithm in the adversarial bandit setting. Motivated by the goal of maximizing information gain during model training, ODM formulates the reward for each domain as the loss peeppexxtyy of each domain (directly proportional to information gain). In the machine translation setting, Schioppa et al. (2023) aim to train model using a mix of standard language modeling data and parallel machine translation data. They propose online data mixing method that optimizes a relative loss improvement score, and updates the domain weights by simply keeping a moving average of the scores.
Challenges of data mixing. There is a natural trade-off in data mixing; when increasing the proportion of data from one domain the distribution density increases for that domain, but the relative proportion (density) of all other domains decreases, which can lead to poorer performance on less represented domains.
One challenge that has not been addressed in any of these works is how their weights transfer to new scttings. For example, Albalak el al. (2023a) find that the weights determined by Xie et al. (2023a) do not transfer well when using different tokenizer. It is unclear how to transfer the weights learned under one tokenizer to new tokenizer without learning the weights from scratch. Furthermore, none of the principled approaches discuss the effects of extrapolating the calculated weights to training on larger quantities of tokens. To exemplify the problem, consider that the weights are calculated by training a model on 50B tokens, and the weights are then used to train a model on IT tokens. Any domains that saw epoch at 50B tokens will now see 20 epochs at 1T, leading to possible overfitting on smaller domains.
25

---

Published in Transactions on Machine Learning Research (07 (2742)24)
Data Mixing Summary & Ideas
When a pretraining corpus is composed of data from multiple domains, data mixing an important step of data selection that assigns weights to each domain, determining the final composition of the dataset.
Weights can be determined manually, by heuristics, empirically determined, or from principled approaches. Each of these methods have been proven under specific evaluations, but none has proven as a general-purpose best method.
Data mixing is challenging, as there is a natural trade-off when upweighting one domain, the remaining domains must. be downweighted. Research on overcoming this trade-off may be difficult, but would be very impactful. One possible direction are methods that consider the interactions (similarities) between domains, which could identify domains that can be downweighted without reducing performance, and those specific domains whose weights cannot be reduced without detriment to performance.
Fach of the approaches discussed has different benefits, and combining approaches (e.g., 1/3 heuristic, 1/3 empirically-drive, 1/3 principled) could lead better general purpose use, but how to best combine them leads to additional complexity.
Although principled approaches are prominent and have lead to state-of the-art mixing, the weights they determine also be brittle when used under different settings (e.g., tokenizer, longer training), which are issues that need to be addressed and explored in future research.
3.9 Current Best Practices and Landscape For Data Selection
The previous sections have described, in detail, wide variety of works, methods, and dalasets that are relevant to the data selection process. Data selection is an open and active area of research and while the exact details of best practices may change over time, there are some practices and ideas that are of high importance and will likely endure the test time. this section we take step back to look at the high-level of the current landscape of research on data selection, and best practices as they currently stand.
3.9.1 Best Practices
Best practices are actively being developed, and new models are consistently being released with new claims of being state-of-the-art. While there are no blanket statements of what is "best" in all use-cases for language models (because best can only be in reference to a specific evaluation protocol or benchmark, not an actual capability), we describe some trends that have either stood the test of time or have been adopted rapidly and widely. The first trend notice that filtering is generally performed starting with the methods that are most efficient and remove broad quantities data, with the later filters being those which are more specialized, but often more costly.
Language filtering. When filtering large quantities of data, an important first step is to ensure that the data is from the desired language, where the most commonly used method is the fast'Text classifier which has been trained to recognize 157 languages and runs very efficiently.
Heuristic filtering. The next step is usually to remove data using heuristics that are very fast to compute. Examples include removing documents with fewer than 50 or greater than 10,000 words or with fewer than 3 sentences. Examples of commonly used heuristics are those from C4 (RafTel al., 2020) and MassiveText (Rae et al., 2021). These heuristics can be accessed through the Dolma toolkit (Soldaini et al., 2024) or Trove (Penedo et al., 2024).
Data quality. Filtering for "high-quality" data (data which was written by humans and has likely gone through an editing process) is a useful step when processing large quantities of web data, but less likely to be useful for data collected from specific domains (e.g., scientific articles or logal documents). The most common method for quality filtering is to train a classifier that can differentiate between random web data and data from a known high-quality domain. There commonly used open-source models, but the methods have
26

---

Published in Transactions on Machine Learning Research (07/2024)
been described within previous works (Brown et al., 2020b). A common approach when using quality filters is to use stochastic filter, which will allow some data with a low quality score to pass through. This can help to reduce some of the biases known to exist in quality filters.
Domain-specific filtering. Methods for domain-specific selection are quite similar to the methods for data quality, but only required when training a model for a specific target domain. The goal of these methods will be to find data within large general purpose corpus which most similar a target domain. The current best methods, such DSIR (Xie et al., 2023b), train two models: one in-domain and one general domain model. To determine whether data point is similar to the target domain, the difference of probabilities under cach model is calculated.
Deduplication. Deduplication has proven to be a beneficial step in every pretraining data selection pipeline. There are multiple methods of deduplication that vary from very simple and broad deduplication methods which only consider motadata to very fine-grained deduplication on individual sentences. When using web-text, a simple and cheap first step is to deduplicate data based on URLs. Next, hash-based methods such as MinHash (Broder, 1997) (approximate matching) and Bloom filters (exact matching) have been used with great success. The previously mentioned methods are fairly cheap, but only consider word or token matching, and do not consider semantic matching. Model-based methods such as SemDeDup (Abbas et al., 2023) have demonstrated success for deduplicating semantically similar data, but are much more computationally expensive and should generally be applied later in the filtering pipeline after cheaper filtering methods.
Filtering toxic and explicit content. Removing toxic and explicit content from web text is generally performed through multiple steps to handle different. types of content. NSFW content can be removed, in part, by filtering out URLs that match NSFW word lists and from blacklisted domains. Beyond just scarching for bad words in the URL, it is also common to filter out documents that contain the same list of NSFW words. Removing personally identifiable information (PII) is another very important step and is commonly found and removed using regex filters, a good example of which is Dolma (Soldaini et al., 2024).
Selection for multilingual models. Generally speaking, most methods from the other steps will work the same for multilingual models, with the exception that some hyperparameters may need to be adjusted to work for cach language. An important bost practice for creating multilingual models is to incorporate native speakers into the decision making process to ensure that data maintains an acceptable quality.
Data mixing. Data mixing is only used when the training dataset is composed of data from multiple clearly defined domains or sources (e.g., books or web text). As of writing, there are no clear best practices for automated methods for data mixing as this is still a very young area of research. Methods of determining domain weights automatically tend to use loss-per-domain as part of signal as to the importance/ /diffcccct/fflf of each domain. Starting points for research are DoReMi (Xie al., 2023a) and ODM (Albalak et al., 2023a).
3.9.2 Current Landscape of Datasets For Data Selection Research
The most popular datasets used for data selection until now have been RedPajama (Skill-it, (Chen et al., 2023c), SlimPajama (Shen et al., 2024), data mixing laws (Ye et al., 2024)), The Pile (ODM (Albalak et al., 2023a), DSIR (Xie et al., 2023b), DoReMi (Xic al., 2023a), Selection for encoders (Feng et al., 2022)), and C4 (DsDm (Engstrom el al., 2024)), each with pros and cons. C4 (Raffel et al., 2020) (750Gb of data) is the oldest and contains only weberawled data. C4 generally leads to models with lower benchmark scores the other 2 datasets, but this makes it easy first testing ground for data selection methods since there is increased room for improvement. The Pile (Gao et al., 2020) (800Gb of data) dataset contains slightly more data than C4, but only ~28% of the total dataset comes from web scrapes and the remaining 72% comes from a wide variety of domains including books, scientific articles, and Github. This combination of well-defined sources makes the Pile a good candidate for research on data mixing. Lastly, RedPajama (IT tokens) is the newest and the largest of the commonly used datasets, and was originally intended be an open-source ro-crcation of the datasct used to train Llama-2. Recently, RedPajama-2 (30T tokens) was relcased which may be an even better resource for lesting data selection. RedPajama-2 comes with precomputed quality signals including those implemented for C4 and Ref,nedWeb, allowing for experimentation on selection mechanisms under controlled settings. Another beneficial aspect of RedPajama-2 is that it contains a huge amount of data, allowing researchers to filter the data down to varions scales, ranging from billions to trillions of tokens.
27

---

Published in Transactions on Machine Learning Research (07/2024)
4 Data Selection for Instruction-Tuninn and Multitask Training
Instruction-tuning and multitask training are methods for addressing the mismatch between the pretraining data distribution and downstream use cases. Multitask train- ing is a method where the model is trained on a wide variety of supervised tasks with the goal of performing all the training tasks and possibly generalizing to unseen tasks. Recently, enstruction-tunigg has become a dom- inant training paradigm where the model is trained on pairs of [Instruction, Output), where Instruction de- notes a human instruction for the model, and Output is the desired output, or an example of a desirable output. The goal of instruction tuning is to encourage the model to generate outputs in a way that is more controllable and helpful for the user (Zhang et al., 2023c).
Learning Stage
Selection Objective
Pretraining
Aediae Rax/Tealryyy
Instruction tunine
Model Perforrance
Aligrment
[fidincy
Co-Content Learning
Teskospeciff fie Fins-turing
Selection Efilency
Fikkation ieagety
Figure 5: Data selection: instruction-tuiing: Generally, the most important objectives when selecting data for instruction-tuuingg are model performance and evaluation integrity.
Both multitask training and instruction-tuning generally assume that the model has been pretrained and has already acquired fundamental language capabilities. The intent of models that have been trained with either multitask training or cestruction-tunigg is, broadly speaking, to handle a very wide variety of possible inputs for downstream use cases as either classification or generative models. Thus, the aim of data selection for these settings is typically focused on collecting a wider varicty of data, and diversifying the data that already exists.
Diversification by scaling tasks and datasets. As model sizes have scaled up and model capacity has correspondingly improved, distribution diversification has proven to be an effective method of improving model generalization, for both specialized tasks and general purpose model performance (Wei et al., 2021). Methods of multitask training (e.g., MT-DNN (Liu et al., 2019), Muppet (Aghajanyan et al., 2021), T5 (Raffel et al., 2020), ExT5 (Aribandi et al., 2021)) demonstrated the benefits of training on range of datasets and tasks, selected for their perceived high-quality curation. Subsequently, number of works (Khashabi et al., 2020; McCann et al., 2018; Keskar et al., 2019) proposed unifying the format of any language task into questions and answers, elfectively adjusting the data point characteristics, with the goal of better matching the train and test distributions and expanding the number of tasks a model could learn in tandem, further improving model generalization. Based on this premise, Natural Instructions (Mishra et al., 2021), FLAN 2021 (Wei et al., 2021), and P3 (Sanh et al., 2022) scaled up the number and diversity of training tasks, distinguished by templated instructions- now known as instruction luning.
At this stage, data selection was based mainly availability across open-sourced dataset repositories (e.g., Hugging Face and TensorFlow Datasets), and pressure to scale, rather than any principled selection criteria. To extend this scaling beyond the openly available data, data augmentation techniques such as translation (Muennighoff et al., 2022b; Yong et al., 2022; Dhole et al., 2021; Singh et al., 2024; Üstün et al., 2024), input inversion (Min et al., 2022; Longpre et al., 2023a), and negative examples (Wang et al., 2022b) have successfully been used to further enrich the data variety. A number of works also increase data diversity by templatizing data in multiple prompt formats, including zero-shot, few-shot, and chain-of-thought prompts (Chung et al., 2022; Iyer et al., 2022). These works demonstrate that task variety and template variety both drive greater generalization, though Wei et al. (2021) report easier training tasks, such as sentiment analysis, contribute less.
Manual and heuristic-based diversification. While scaling tasks has been an effective method of diversifying data, naively including all data possible can lead to severely imbalanced datascts and has motivated the use of better mixing and balancing techniques. For example, Wei el al. (2021) impose a maximum number of sub-dataset examples to prevent large datasets overwhelming the training mix. Additionally, Tyer et al. (2022) experiment with a range of maximum values, finding that including at most 512 data points from each sub-dataset achieves the best results across held-out test sets. By setting an upper limit to the number of samples from any individual sub-dataset, these methods improve the data. diversity
28

---

Published in Transactions on Machine Learning Research (07/2024)
by reducing the distribution density around arbitrary regions of the sample space, as defined by individual sub-datasets.
Iyer et al. (2022); Longpre et al. (2023a); Chung et al. (2022); Wang et al. (2024) achieve strong performance improvements by varying the proportions of sub-mixtures with a manual grid search, but this can be very costly. Furthermore, Wang et al. (2024) and Tvison et al. (2023b) show that none of the available mixtures are by default best on all evaluations. This strongly motivates the need to design and explore data selection methods in the instruction-tuning setting.
Model-based diversification. More recently, instruction-tunin data has been further diversified through the use of existing models using conversation as the unified data format. Training data can be generated directly by a model, often from the OpenAI API, creating and expanding examples iteratively (Honovich et al., 2022; Wang et al., 2023e; Xu et al., 2023; Taori et al., 2023; Lee et al., 2023; Kim et al., 2023). Wang et al. (2023c) begin with a small seed pool of example tasks, which the model uses to iteratively generate new lasks and instances of that. lask, followed by filters to ensure that data is "high-quality" and to remove duplicated data. EVOL-INSTRUCT methods enhance these techniques to iteratively re-write training instances to be more complex and challenging (Xu et al., 2023). Lu et al. (2023a) itcratively trains on and produces its own training data with self-refinement techniques. These methods are shown to produce a diverse array of prompts that can be guided to match the distribution of certain real world tasks, domains, and topics which are not covered in existing datasets such as P3, FLAN, or the dataset used to train OPT-IML.
A survey of synthetic data (asks showed that they tend to focus on longer-form responses than what exists in academic datasets, well as tasks related to brainstorming, planning, explanation, and creativity (Longpre et al., 2023b). The advent of synthetic data has also spurred developers to filter out system-specific outputs, such as disclaimers and refusals (e.g., "I'm sorry, but([..]=16 from OpenAl models).
Data efficiency. As larger and more diverse instruction tuning collections have become available, the benefits of scale seem to have diminished, and in response recent methods have explored a variety of utility functions aimed at improving data efficiency while focusing on quality, diversity, difficulty, and complexity as components of the utility. LIMA demonstrated strong results with only 1000 highly curated, diverse training instances meant to approximate real user prompts with high-quality responses (Zhou et al., 2023). Similarly, the best performing variants of Open Assistant are high-quality subsets, as graded by human reviewers (Köpf et al., 2023; Muennighoff et al., 2023a; Zhuo et al., 2024). The previous works use human judgments to dotermine quality, but new work has developed methods to automatically measure and sub-sclect for instruction data quality and diversity. For instance, Cao el al. (2023) use natural language indicators lengths, perplexity, and textual lexical diversity) with BlendSearch to measure and identify the highest quality data points. Similarly, InsTag (Lu et al., 2023b) labels instructions based on semantics and intentions. On the other hand, Lian et al. (2023) improve data efficiency through the use of a model-based quality filtering. They perform a pass over the FLAN dataset and remove datapoints where the answer appears to be incorrect (according to GPT-4),, then train a model on this reduced dataset and find they get a model with similar quality level, but only 2/3 the compute. Similarly, Chen et al. (2023b) use CParGPT to score data points and use this score utility function to select a subset of data, improving data efficiency compared to using the full Alpaca (Taori et al., 2023) dataset. Additionally, Li et al. (2023a) introduce the Instruction-Followiii Difficully metric which allows a model to self-select the data points which are most important. Data Efficient Instruction Tuning for Alignment (DEITA) surveys InsTag and other prior methods that mcasure data complexity, quality, and diversity, and propose unified data selection strategy that samples high quality examples, while being diversity-aware (Liu et al., 2023b). Lastly, Kung et al. (2023) introduce Active Instruction Tuning, a framework that identifies informative tasks to improve generalization, using prompt uncertainty as the utility function. These methods demonstrate a growing focus on automated mcasures of quality selection, often dynamically as part of the model training process.
(such as
Data selection is increasingly guided by data governance, liability, and other concerns Data selection is not always guided only by performance considerations. A large-scalc audit of popular instruction tuning datasets shows a marked rise in datasets with non-commercial terms and licenses (Longpre el al., 2023b).
lohttpe://hugginggaacs
http:///wwgaggaaacce///////a///a htt.//attsetttttekmiiia wttpaiummm/PPPP4P-P. Cleaned
29

---

Published in Transactions on Machine Learning Research (07/2024)
Whereas non-commercial restrictions applied to 30% of datasets in the prior decade, 61% of 2023 datasets had such restrictions. Similarly, new lawsuits (Saveri et al., 2023) and pushback from creators (Chayka, 2023) over alleged copyright infringement has become central concern for developers navigating legal risk in their training data selection. The logal outcomes and enforceability of such restrictions remain uncertain, and may vary by jurisdiction. Additionally, data selection may guided increasingly by factors related multlinguality, target task type, or domain, as prior work has shown no size fits all for instruction tuning data (Wang al., 2024). To address some of these concerns, the Data Provenance Initiative (Longpre et al., 2023b) has provided tools to search and filter for datasets according to license, terms, and characteristies criteria.
30

---

Published in Transactions on Machine Learning Research (07/2024)
5 Data Selection for Preference Fine-tuning: Alignment
Various aligument methods, which include Reinforcement Learning from Human Feedback (RLHF), RL from AI Feodback (RLAIF), and Direct Preference Optimization (DPO), involve the integration of human preferences into model behavior. This training process is designed to steer model responses to be generally more helpful and less harmful, whereas in other training phases such as pretraining or instruction-Luning, these preference signals may not be clearly defined within the utility function. These methods, grouped under Preference Fine-tuning (PreFT), typically follow instruction-tuiing in the training pipeline of large generative models. The format of this data is often trios of (Prompt; Chosen response, Rejected response), where a Prompt is an instruction or other request from a user, Chosen is the preferred answer and Rejected the less- preferred answer. Some methods work on groups of responses to a prompt, but they are less prevalent, and recently, Ethayarajh et al. (2024) proposed Kahneman-Tvorsky Optimization (KTO) which requires only the chosen or rejected continuation, but not both. The benefit of PreFT methods is to remove the requirement of strict next-token or accuracy-based predictions from ML models and integrate other, sometimes nuanced, qualitative signals instead.
Learning Stage
Selection Objective o
Pretraining
Rediar Rox/Todiityy
Instruction tuning
Algemeet
Andoerraze
Firlewyy
In-Content Learring
Tesksppecilic Fine-turing
Seletiion bllonney
Erikatton stteht
Figure 6: Data selection: alignment. Often, the most important objectives when selecting data for aligument is reducing bias and toxicily and model performance.
RLHP and RLAIF methods include the important step of independent training of a reward model for active selection or weighting of samples during training. A reward model is a separate model fine-tuned to take in any input, such as text, and output a scalar reward indicating the utility (as represented by the probability that the text would be chosen). Data selection methods can be applied via training a reward model or the downstream policy, except for Direct Preference Optimization (DPO) methods that directly optimize the reward model and policy jointly from one set of data (Rafailov et al., 2023).
Data selection methods for PreFT are very nascent and are often focused on obtaining signal on specific capabilities and evaluations from the model. Tn this setting, the primary approaches to data selection are manual filtering, model-based evaluation, and reward model re-weighting (c.g., rejoction sampling). The simplicity of the data selection methods rellects the relative underspeeification of the goals of preference training, where basic questions are actively being studied on how best to collect preference data, and for what purpose (Lambert et al., 2023a; Casper et al., 2023). This area is also studied in Liu et al. (2023b) primarily for aligning instruction-tuned models, where the authors focus three axes: complexity, quality, and diversity.
Manual and heuristic selection. Initial methods for collecting and filtering data for PreFT involved manually selecting data and using heuristic utility functions. Multiple works have indicated the use of external organizations to procure their data (Touvron et al., 2023b; Nakano et al., 2021; Bai et al., 2022b; Rajani et al., 2023), where detailed instructions are written by researchers and sent to data-labeling company. However, it is common practice for data-labeling companies to filter the data internally before sending it to the model training organization and these processes are not documented beyond general notions of controlling for diversity and quality. Furthermore, this process is not reproducible. Ethayarajh et al. (2022) filter Reddit data based on the number of votes received on comments and posts in the comment chain when creating the Stanford Human Preferences (SHP) dataset. Similarly, Bai et al. (2022a) and Lambert et al. (2023b) weigh data points based on a measure of minimum engagement from users when creating their respective datasets, UltraFeedback, a popular dataset used to train Zephyr-B (Tunstall et al., 2023) and Tulu 2 (Ivison et al., 2023b), implements multi-aspect filtering process to create a preference dataset focused on instruction following, helpfulness, truthfulness, and honesty (Cui et al., 2023a). Further manual filtering has been performed on this data, and others such as Orea DPO pairs!" 17 improve the performance by verifying accuracy (Bartolome et al., 2023). The authors perform stratified
chat models such as
17 http::///mngghgfffrrrres/ssstttt
31

---

Published in Transactions on Machine Learning Research (07/2024)
sampling across variety of existing preference datasets, filter for length, and decontamintate their data. Additionally, they use set of 17 models to obtain diversity in their dataset.
Model-based selection. While human oversight can be higher quality, the ability of models to accurately label undesirable (e-g., incorrect) data has improved significantly, leading to an increased usage of language models to assign utility. For example, the Phi series of models (Gunasekar et al., 2023) use filtering to achieve "textbook quality" data from the web, as performed by a custom LM-basod classifier, combined with synthotic data generated from Ap models. UltraFeedback (Cui et al., 2023a) and Nectar (Zhu et al., 2023) use GPT-4 to filter and/or rank the responses for quality. Critic models (Shepherd (Wang et al., 2023a) and Prometheus (Kim et al., 2023)) are used to critique model responses and identify diverse errors, and can be used to rank data points for quality. An example of using a model to filter data is by asking an LLM if an edited piece of text fixed a critique, and if the answer is no, the datapoint is removed, as done in Castricato et al. (2024). They are used as a part of the RLAIF cycle, where the critic is used as a reward model, to rank multiple model outpuls. As this area of methods matures, it is likely that such critic models will form a central part of the data selection process by simply removing erroneous data.
Reward model re-weighting. A number of works have also demonstrated that using reward model directly to assign utility to candidate data points is viable data selection method (Bai et al., 2022b; Touvron et al., 2023b; Yuan et al., 2023). In rejection sampling (RS), or Best-of-N sampling (BoN) as used in Nakano et al. (2021); Gao et al. (2023a), n candidate oulpuls are sampled from the generalive model and the reward model selects the best candidate. Touvron et al. (2023b) use rejection sampling as part of a continued instruction-tuning by selecting the top percentage of their instruction datasct to continue training on, while Yuan et al. (2023) use rejection sampling over reasoning paths to improve mathematical abilities, where reasoning path length is key factor. They evaluate the reasoning steps programmtically in a Python environment and use the candidate completions from multiple models while fine-tuning single model. Liu et al. (2023a) improve on standard rejection sampling, inspired by the unsupervised RL training loop, to have a critic model consistently evaluate data in the RLHF process, gating which data is passed to the model and offering critiques on the rejected data. Beirami et al. (2024) demonstrated that a popular KL, distance measuring the impact of Best-of-N sampling is actually bounded by KL distance rather than a scaling law on the number of samples scored over.
Pace et al. (2024) propose West-of-N sampling, method of utilizing synthetic data to improve the reward model, which can in turn improve the base generative model with RS or BoN techniques. West-of-N sampling takes the synthetically generated candidate outputs (as in rejection sampling) and extracts the best and worst candidates as a pair of preference data. This data is then used to iteratively fine-tune the reward model, which can be used for many of the other methods in this section. Similar to West-of-N is the method of Self-rewarding LLMs (Yuan el al., 2024b), where instead of using the synthetic pairwise data to train a reward model, the data is iteratively used to directly update the policy with direct preference optimization (DPO) (Rafailov et al., 2023). However, self-rewarded models have been shown to be biased towards their own rewards (Xu et al., 2024a) and, similarly to other methods which utilize synthetic data, it is crucial to ensure that the data generating process yields diverse set of data to reduce the chances of mode pol.
Emerging methods have applied reward-weighted data selection within a feedback loop, more akin to earlier forms of RLHF, such as with DPO models (Yuan et al., 2024b), or methods imitating self-play with less clear data selection methods (Chen et al., 2024). For example, Liu et al. (2024b) performs a pjjation-sammping-llit optimization with statistical weightings predicting the likelihood of whether a completion came from the optimal policy or base SFT policy to improve downstream preference optimization similar to DPO or Sequence Likelihood Calibration (SLiC) (Zhao et al., 2022). Especially in preference-base based alignment, the border between synthetic data generation and data selection are often blurred as best practices are determined.
32

---

Published in Transactions on Machine Learning Research (07 /204)
6 Data Selection for In-Context Learning
In-context learning (ICL) is a widely used prompting paradigm for language models (T.Ms). Instead of being used to fine-tune the LM, a few demonstration examples are given as a prompt to guide the language model to per- form a similar prediction task on the imput query (Brown et al., 2020a). In-context learning is known to be sensitive to the choice (Perez et al., 2021), and even ordering (Lu et al., 2022), of the demonstrations. To improve the ICL performance without training the potentially large LM, many recent papers have worked on constructing better in- context. demonstrations by: selecting an optimal ordering from a fixed set of demonstrations, selecting from large set of labeled data, or strategically annotating a small set of unlabeled data.
Learning Stage
Selection Objective
Pretraining
Instruction tunnig
Modd
Aligmment
In Contoat Learning
Tasksspeciic Fine-turing
Selection Effclency
Exakation stageit
Figure 7: Data selection: in-context learning. Generally, the most important selection objectives for in-context learning are dala efficiency and model performance.
Demonstration reordering. Lu et al. (2022) point out that the ordering of demonstrations significantly affects the in-context learning performance, and propose to reorder the demonstration using the entropy of the predicted labels as a utility function.
Demonstration selection. Suppose we have a set of annotated data that cannot all fit into the prompt of a pretrained language model, and we also do not want to finc-tune the potentially very large language model due a limited budget. How do we select the set of data points to be used as in-context demonstrations that leads to optimal performance? As discussed below, some methods select fixed set of demonstrations, while others select demonstrations for each individual test data point using either retrieval-based methods or sampling- basedd based methods.
Some works have focused on selecting a fixed set of demonstrations that achieves the best average performance across all testing inputs sampled from a target task. For example, Zhang et al. (2022a) use reinforcement learning techniques to learn a policy that assigns utility to demonstrations for use on previously unseen tasks. A different approach, proposed by Wang al. (2023b), is to use a smaller pre-trained language model to learn task-defined latent tokens and then select the examples that can best reconstruct the latent tokens.
Many other works propose retrieval-based methods, where they train demonstration retriever that can select demonstrations specifically tailored to each individual test data point. A straightforward method is retrieving the most similar examples to the testing input by cosine similarity of sentence embeddings (Liu et al., 2022). Gao et al. (2023b) further enhance this approach by retrieving demonstrations whose label lies in top-2 zero-shot predictions of the testing input. Rubin et al. (2022) score the similarity-based retrieved candidate demonstrations by their one-shot in-context learning performance for sequence-to-sequene tasks, then train a retriever with the collected scores. Li et al. (2023c) advance this framework through unified training across various datasets. Hashimoto et al. (2023) analyze the effect of different utility functions in training demonstration retrievers, and propose that incremental utility, which is the 1-shot utility score minus the (-shot utility score, is better than directly using the 1-shot utility score. Additionally, Iter al. (2023) show that the cross entropy difference between candidate demonstrations and the test example is a promising selection utility. Ye et al. (2023) improve upon the previous works by scoring a set of demonstrations instead of only one demonstration. Finally, when prompting a model for semantic parsing, Levy et al. (2023) propose to select diverse demonstrations that collectively cover all of the structures required for the output program. For a comprehensive survey on this line of work, please see Xu et al. (2024b).
There are also (iterative) campling-based methods that do not require training a retriever to select the demonstrations. The most straightforward approach would be to uniformly sample a set of demonstrations from the candidate set as used by Min et al. (2022). Chang & Jia (2022) train a datamodel (Ilyas et al., 2022) to predict the validation performance for set of chosen demonstrations and select the demonstrations with the
33

---

Published in Transactions on Machine Learning Research (07/2024)
highest validation performance. Similarly, Nguyen & Wong (2023) propose constructing validation set and evaluating each training data point by contrasting the validation performance of prompts with and without that data point, and formalizing the performance difference as influence. Li & Qiu (2023) iteratively select in-context demonstrations by an informativencss score, which is the language model prediction probability difference between one-shot and zero-shot inpuls, and a diversity score; which discounts semantically similar examples. Gupta ct al. (2023) propose select the most informative set of demonstrations, instead of individual informative demonstrations, with a greedy algorithm to maximize the coverage of the chosen demonstrations. Similarly, Xu & Zhang (2024) iteratively refine the selected set of demonstrations by computing the misconfidence score, which is the ratio between the highest probability that the language model assigned to any incorrect label, and the output probability of the correct label.
Selective annotation. SU et al. (2023) first propose the selective annotation setting: given an unlabeled dataset and a fixed budget, the goal is to select the most informative examples for annotation, which are used to further select demonstrations for each testing input to maximize ICL performance. In this way, we aim minimize the human annotation effort in creating labeled datasets. We also maximize the advantage of in-context learning that only small number of labeled examples are needed as demonstrations.
Various methods for selecting the annotation set have been proposed, usually based on the difficulty and diversity of the unlabeled examples. Then, a sentence-transforee (Reimers & Gurevych, 2019) is usually employed to compute the cosine similarity between cach annotated example and the testing input, then the most similar examples are selected as the ICL demonstrations.
SU al. (2023) propose VOTE-K, which constructs graph by applying k-NN over the transfce-rannsoormerr embedded (Reimers & Gurevych, 2019) unlabeled examples, to select examples for annotation. They also try to use exponential score to discount examples that are close to the already selected examples. Zhang et al (2023b) also contract graph over the unlabeled examples by k-NN. They use a diffusion process to quantify the influence of unlabeled subsets and a greedy algorithm to conduct the final selection. Their proposed method, IDEAL, achieves better performance under lower time consumption than VOTE-K. Mavromatis et al. (2023) combines diversity and uncertainty sampling, by assuming each wrongly predicted demonstration is most useful in predicting inputs from its semantic neighborhood, therefore formalizing demonstration selection max coverage problem. Wu et al. (2023) propose an interactive data annotation system, that iteratively samples unlabeled examples with spek-fpe patterns and then let the human annotators correct the LLM-suggosted annotations.
34

---

Published in Transactions on Machine Learning Research (07 2204)
7 Data Selection for Task- Fine-tuning
Fine-tuning a model on a specific target task is a very different learning setting from pretraining, instruction- luning, or RLHF, but the data selection methods that apply are not vastly different. In some ways, selecting data for a specific target task can be casier than the pre- vious settings. First, because there is only one target task, the target distribution will generally be more narrow than in either pretraining, instruction-tuuing, or multi- task learning. Also, task-specific mun-ttnnng is generally easier to evaluate because the target distribution is more narrow, the expected use cases are more clear, and suc- cess has more straightforward definition, leading to a less ambiguous evaluation than the previously discussed settings.
Learning Stage
Selection Objective
Pretraining
Instruction tuning
Madel Porferrance
Aligrment
Eflanny
In-Conteat Learring
Teckospecfic Fine-turing
Selution Ellcarey
Emkaaton Iaggtt
Figure Data selection: task-specific fine- tuning. Commonly; the most important objec- tives when selecting data for fine-tuning are model performance, data efficiency, and evaluation in- legrily-
Data selection for lask-specific fine-tuning can be roughly divided by whether the goal is to match the target distribution or to diversify the existing data distribution. The first setting, where the goal is to match a target distribution, is particularly beneficial in data-limited situations such as few-shot learning (Albalak et al., 2022b; Ivison et al., 2023a; Albalak et al., 2023b). For example, there may be very little data for the target task (the task we want the model to perform); but we do have access to a wide variety and large quantity of auxiliary data that we can utilize. The second setting, where the goal is to diversify the data distribution, can further be divided into two settings, where the goal is either to improve data efficiency (Lang et al., 2022; Maharana et al., 2024), or improve the model's robustness (Haghtalab et al., 2022; Sagawa et al., 2020; Bejan et al., 2023).
Selecting from auxiliary data to improve generalization. There have been a number of works that utilize auxiliary data (i.e. data from a source other than the target task) to improve model's ability to generalize to the target distribution. In these settings, the goal of data selection is to determine the optimal auxiliary data that improves a model's performance on the target distribution, and because we have access to data from the target distribution the utility function for these methods is often a form of similarity between the auxiliary and target data. These methods tend to have some similaritics with domain-specific data selection for pretraining data ($3.4) with the main difference being that in a line-tuning setting, more computationally expensive utility functions are feasible.
The availability of pretrained models has been huge benefit to the ability of models to generalize fine-tuning setlings, allowing information learned during pretraining to be transferred and applied to the downstream task. A number of earlier works have taken this step further and perform an intermediate training step of training on auxiliary data between pretraining and fine-tuning, often relying on empirical evidence to determine which auxiliary data is best (Phang et al., 2018; Pruksachatkun et al., 2020; Iter & Grangier, 2021; Grangier & Tter, 2022).
Phang al. (2018) introduce the Supplementary Training on Intermediate Labeled-data Tasks (STILTs) method where the goal is to fine-tune model for tasks in the GLUE dataset (Wang et al., 2018a). They manually select textual entailment (MNLI (Williams et al., 2018), SNLI (Bowman et al., 2015)), paraphrase detection (8) and fako-sentence detection as their auxiliary tasks specifically because they are similar to the tasks in the GLUE dataset. Similarly, Pruksachatkur el al. (2020) evaluate the transfer between 11 intermediate auxiliary datasets and 10 target tasks, but also evaluate the models on 25 probing tasks to try and understand what skills are being learned from the auxiliary data. These methods provided carly efforts at understanding task transfer, but the data selection was performed entirely by human decisions on similarity to the target. task.
While Phang et al. (2018) and Pruksachatkun et al. (2020) focus on task transfer from the auxiliary to target data, Iter & Grangier (2021) and Grangier & Iter (2022) focus on domain adaptation from the auxiliary to
18nttps://data wwt./oost///eeeeeeeeee MatassetReeeeat Question- Pairs
35

---

Published in Transactions on Machine Learning Research (07/2024)
target data. Iter& & Grangier (2021) find that the best data selection method for domain adaptation uses domain classifier as the utility function, model that is trained to discriminate between data from the target distribution and from a general distribution, similar to methods on domain-specific selection for pretraining ($3.4). Grangier & Iter (2022) further explore the use of three separate data selection methods: importance sampling, contrastive data selection, and influence functions, finding that all three methods select for very similar data points.
More recently, methods that select from auxiliary data train on both the auxiiiary and target data simulta- neously. For example, FETA (Albalak et al., 2022b) performs comprehensive task transfer study similar to Pruksachatkun et al. (2020), training model on all auxiliary-target pairs, and finding that similarity between the auxiliary and target task's label-space (e.g., binary classification, multi-class classification, etc.) can be used as a good heuristic utility function. However, using the label-space is only possible when the structure of the target task is very clear, and only useful when the label-space is fairly unique. Additionally, the structure of a task does not fully describe the data, and this method is not feasible at large scales as it requires human judgment. DEFT-FEN (Ivison et. al., 2023a) improves on the ideas from FETA by using a model-based utility function to select the 500 auxiliary data points that are nearest neighbors to the target dataset. SKILL-IT (Chen et al., 2023c) also builds on the ideas from FETA (Albalak et al., 2022b), defining tasks as different skills that a model can learn, and learning a pairwise correlation between skills by training on each pair of skills to find when skill A benefits skill B. Based on this, they create a graph of skills ordered by dependencies and define a data mixing algorithm that incorporates the loss on the target task. However, training a separate model on every pairwise combination of tasks may be very inefficient. Albalak al. (2023b) improves upon the inefficiencies of SKILI-ITT by formulating the auxiliary data selection problem as multi-armed bandit, where the utility function is an aggregated reward function of the cosine similarity and magnitude similarity between gradients of batches from the auxiliary and target datasets. By formulating the auxiliary data selection problem as a multi-armed bandit, their algorithm runs online without requiring any knowledge of the auxiliary data prior to training. While Albalak et al. (2023a) design a gradient-basod utility function for data mixing, Xia et al. (2024b) use a similar gradient-based utility to assign values to individual auxiliary data points. To improve the efficiency of assigning utilities to each data point, they only calculate gradients for the parameter-effiient LoRA (Hu et al., 2022) modules and further reduce the gradient dimension through random projection.
Improving task-specific data efficiency. Another setting where data selection can be applied beneficially for task-specific fine-tuning is when wish to select subset of the target dataset to improve data elficiency. Methods in this category have similar motivation to fuzzy data deduplication from $3.5, but because the target datasets when performing fine-tuning are significantly smaller than pretraining corpora, the methods used here can apply more complex, computationally expensive utility functions.
Maharana et al. (2024) propose a method of diversifying data based on the expected difficulty. They represent. the dataset as a graph, where individual nodes (data points) are connected to their k-nearest neighbors in representation space and use variability (Swayamdipta et al., 2020) the utility function for each individual data point. Next, they perform forward message passing process which propagates information on the ut utility function of a data point to its neighbors. Finally, they perform data selection through an iterative reverse message passing process, where at each step the highest scoring data point is kept, and all of its neighbors get
to encourage the selection of diverse data points. Lang et al. (2022) also use a graph representation as part of their utility function, determining which weakly-labeled data points to keep basod on their nearest neighbors, and the label of their nearest neighbors. They only keep data points whose nearest neighbors have mostly the same label, thereby using the representations geometry to identify accurate subsets and removing ambiguous data points. Bejan et al. (2023) develop the task-agnostic self-influence score as a utility function for filtering data (and designing curriculum). The intuition behind the self-influcnce score is: if not training on data point z00 leads to a significantly higher loss on that data point, then there must not be many supporting data points around (+() and it is likely to be an outlier, and therefore a low-quality data point. They suggest that self-influence functions can identify data with label noise, out-of-distribution data, data that is ambiguous, and difficult to learn samples, and removing such data can improve the models performance.
36

---

Published in Transactions on Machine Learning Research (07/2024)
Many methods of improving data efficiency exist in general machine learning settings (e.g., data pruning and coreset selection), but most have exclusively been developed and evaluated in the computer vision domain. Azeemi et al. (2023) is one of the limited tests of data pruning on language tasks, where they observe that selecting for difficult to learn examples leads to improved model performance, while reducing the dataset size by up to 60%. The many other works on data pruning and coresel selection (further discussed $9) may provide effective methods of improving data efficiency for task-specific fine-tuning, but could require some alterations to work for NLP tasks.
Selecting data for robustness. Finally, important problem that data selection be applied to is improving model's robustness by ensuring that model is not biased towards any particular subpopulation of data and to avoid relying on spurious correlations. Because these methods concern themselves with subgroups of data, they share similarities with data mixing ($3.8). Sagawa et al. (2020) introduce group distributionally robust optimization (CROUP DRO) to train models that minimize the worst-case loss over groups in the training data. As their utility function, they use a notion of the generalization gap (the difference between training and validation loss) and prioritize training on data from those subgroups which have a large generalization gap. Haghtalab et al. (2022) also aim to minimize the worst-casc error of model across subgroups, while also aiming to improve the data elficiency by minimizing the nunber of samples required for each subgroup. They formulate the multi-distribution learning problem as zero-sum game and aim to learn the minimax equilibrium by introducing an online learning algorithm, Resampling- based Multi-DDsttrbutian Learning (R-MDL) that uses estimates of the training error and generalization error as the utility.
Considerations for task-specific fine-tuning. Compared with data selection for pretraining, the goals of data selection for task-specific fine-tuning are very similar: improving data efficiency, selecting a diverse set of samples, and reducing model's bias towards any particular subgroup. However, the task-specific datasets used in fine-tuning are often small enough that methods with high computational requirements are plausible. Though many of the methods presented in this section may be beneficial for pretraining (or other settings), they may be unreasonably costly to use on web-scale data. For example, Azcemi et al. (2023) requires training a model on the full dataset prior to determining which data points to keep/remove, and Haghtalab al. (2022) assumes that the model will be validated on each subgroup at every step of the learning algorithm. Although these ideas may be computationally prohibitive to be used as-is, it may be possible to adjust them to work at larger scales by reducing the frequency of updates, using smaller models proxics.
37

---

Published in Transactions on Machine Learning Research (07/2024)
8 Data Selection for Other Domains
This survey focuses data selection for language models, but data selection is active area of research in computer vision, vision-and-language and broadly in machine learning. Some of the data selection methods covered by this survey are domain-specific (e.g., the lotal number of characters in 2!0), while others are domain agnostic (e.g., selecting data most similar to a target distribution). Therefore, some of the methods previously presented in this work may be more or less applicable in other domains. Similar to language models, data selection methods have been applied at multiple stages of model training in other domains (e-g-, vision-language model pretraining) and for similar purposes (e.g., data efficiency, model performance, and reducing bias). In this section we will briefly discuss some of the prominent uses of data selection in non-language domains and compare and contrast them with the previously discussed methods.
Data selection for pretraining, Of particular interest has been the risc in pretrained vision-language models, which has followed closely after the success of large language models (Radford et al., 2021; Schuhmann et al., 2022; Zhu et al., 2024). For example, Schuhmann et al. (2022) and Zhu et al. (2024) design datasets for vision-language pretraining using some of the same methods and ideas presented in $3 as well as some methods that are specific to their languala data. Schuhmann al. (2022) collect web-text along with images that contain an alt-tezt in order to create data with image-text pairs. They utilize some heuristic filters the text, similar to those from $3, but they also remove data, with less than 5kb of image data, well as image-lexi pairs that have cosine similarity less than 0.28 for english data and 0.26 for other languages (image and text separately encoded using OpenAI's ViT-B/31 /32 CLIP model). The similarity step removed significant quantitics of data, reducing the original corpus from around 50 billion images to 6 billion. While LAION-5B is a corpus entirely of image-text pairs, Multimodal C4 (Zhu et al., 2024) introduces a dataset of interleaved sentences and images, where each data point can contain multiple sentences and images. Zhu et al. (2024) use preprocessed English-only C4 dataset and gather image files the source websites for C4. They perform image-decuplication step to ensure that a single image is not overrepresented within single document or across the entire corpus. Then, they discard images with a height or width smaller than 150 pixels, which accounts for many small icons and navigation buttons, as well as images with aspect ratios greater than 2 or less than 0.5 (removes many banners at the top/bottom or sides of websites). Finally, to interleave images into the existing text, they use cosine similarity between images and individual texts, interleaving only the images which have a similarity score of at least 0.15. Another unique filtering done for this domain is a step of removing images that contain faces.
that were found on
Recently, Bruce et al. (2024) collect dataset of publicly available internet videos and train a foundation world model to generate action-controllable worlds given image inputs. They filter their dataset using a mixture of hcuristics (e.g. vidco titles must contain certain keywords such as "speedrun" or "playthrough" ") and quality filtering. Specifically, they hand label 10,000 videos and train a classifier on their labeled data.
These methods mainly filter data with heuristic approaches, and have the same drawbacks as heuristic approaches for languago datasets; namely, the difficulty of validating a method, and slow iteration time as it requires training multiple models. Dala Fillering Nelworks aim to overcome these weaknesses by filtering data not with heurstics, but with trained network (Fang et al., 2024). While Fang et al. (2024) demonstrate that it is possible to train filtering network that can select data points which lead to a better model, in a cruel twist, they find that data quality is key to training a good filtering network, leading to a circular problem.
In addition to data collection and filtering, data mixing is also a topic of concern for non-language pretrain- ing. Piergiovanni et al. (2023) propose an online data mixing method for language models that mixes variety of tasks together (e.g., image captioning or visual question answering). Their method, dynamic difficulty sampling, calculates the loss of each task t, Li, and weights each task proportionally with the total
L. This mixing method maintains some similarities to the online data mixing methods from language model pretraining (Xia et al., 2024a; Albalak al., 2023a; Schioppa et al., 2023)-
loss L, getting the final sampling weight as S. L'
38

---

Published in Transactions on Machine Learning Research (07/2024)
Data selection for task-specific fine-tuning. The popularity of task-specific fine-tuning across modalities and domains has led to many data selection methods that larget this setting. For the most part, the methods of data selection for non-language domains are quite similar to the methods for language. Generally speaking, these methods use the output of a model as the representation upon which the utility function is built. In an early work on support vector machines, Wang el. al. (2005) devise a method of selecting data that reduces the training set size while maintaining generalization by filtering out data points which do not affect the final decision function. This idea resonates in the language domain, with similarities to methods that use loss (Albalak et al., 2023a) or methods that attempt to estimate data difficulty (Mindermann et al., 2022). A more recent direction has been how to best utilize synthetic data, and Yuan et al. (2024a) demonstrate that generating large quantities of synthetic images and selecting data using distribution matching (matching the synthetic data to the larget data using maximum mean discrepancy) leads to models that perform well for image classification with synthetic data only, as well as when mixing real and synthetic data.
Adaptive batch selection. A number of methods have been developed to improve the data selection on the level of individual batches, sometimes called data difficulty estimation. One of the motivations of these methods is that they use the current model's state to determine which data to select, but this can add significant computational overhead. Additionally, because these methods are run alongside the model training, their cost is not amortized across training runs as it would be for static datasots. Chang et al. (2017) propose two methods for ACTIVE BIAS that emphasize training on data points where the model is uncertain, based on their prediction variance or closeness to the decision threshold. Similarly, Song et al. (2020) propose RECENCY BIAS, a method that further emphasizes training on uncertain data points that were predicted inconsistently in recent iterations of training. Finally, Mindermann et al. (2022) present Reducible Holdout Loss Selection (RHO-Loss), a method of adaptive batch selection that, intuitively, avoids selecting data points that are redundant, noisy, or less relevant. RHO-Loss was validated on a very small sample of NLP tasks, task-specific fine-tuning on SST2 (Socher et al., 2013) and CoLA (Warstadt et al., 2019), but has not been demonstrated to work in other learning settings, and Kaddour et al. (2023) show that it is not computationally efficient.
Adaptive batch selection methods are similar to online data selection, except that they are often performed on the level of individual data points, rather than groups, domains, or tasks. This particular direction has found minimal success with LLMs, in particular for pretraining (Kaddour et al., 2023), but the success of such methods in other domains suggests that it would be worthwhile adapting these methods to language.
Bias, fairness, and robustness. Reducing a models bias towards or away from any particular class is a long-standing challenge in machine learning, related to the desire to create fair datasets and improve a model's robustness to distribution shifts and nut-of-distributin data.
There have been a number of recent works that aim to prepare fair datasets. Biswas & Rajan (2021) analyze stage-specific fairness (SF) which measures fairness metrics with and without certain data preprocessing stages. González- Zelaya et al. (2023) introduced FAIRPIPES, a heuristic approach to find an optimal data preparation pipeline configuration. A number of works have also developed software tools for tracing distribution distortions linked to fairness in the data preparation workflow (Yang et al., 2020a; Grafberger et al., 2021; Schelter et al., 2019).
Additionally, there exist fairness-aware balancing techniques (Yan et al., 2020; Sonoda, 2023; Chakraborty et al., 2021; Salazar et al., 2021; Yang et al., 2020b) as well as reweighting/reg wreiightinngrrsamlinngg resamplling methods aimed at ensuring fairness (Kamiran & Calders, 2012; Jiang & Nachum, 2020). In NLP, using balanced training for fairness has been explored (Wang et al., 2019; Han et al., 2022). Jeong et al. (2022) showed that different proportions of demographics can lead to nontrivial differences in fairness. To alleviate the challenge of gathering additional data, recent research has focused on designing adversarial networks. These networks are developed to acquire a balanced representation, which helps in minimizing the adverse effects of sensitive attributes (Madras et al., 2018; Adel et al., 2019).
Learning reprosentations that are robust to distribution shifts, or out-of-distribution data, is very challenging, and a very active area of rescarch. Deng et al. (2023) propose Progressive Data Expansion (PDE), a method for learning robust representations by progressively expanding the training dataset. While naively training on
39

---

Published in Transactions on Machine Learning Research (07/2024)
all data creates models that are susceptible to learning non-generalizable spurious features, by progressively expanding the training dataset, Deng et al. (2023) demonstrate that PDF improves the model's robustness for a better worst-group performance. Nguyen et al. (2022) demonstrate, on six publicly available image-text datascts, that each individual dataset can lead to robustness to certain distribution shifts, but no individual dataset dominates the robustness. They explore the interactions between datasets by combining multiple datasets and find that this does not lead to an overall better model, rather dilutes the robustness from each component dataset. Their results demonstrate that simply gathering large amounts of data is not optimal.
Increasing the rate of data sclection rescarch. Recently, the importance of data selection and quality has gained notoriety, and thus few efforts have been developed to try and lower the barrier to entry for research to be done. One such effort is DataComp (Gadre et al., 2023), benchmark for multimodal dataset design. DataComp provides two tracks for participation: the filtering track, and the track. The filtering track of particular interest as it gives participants a common dataset (12.8B image-text pairs from Common Crawl), and the participants" goal is to determine the best subset to train a model on. Additionally, Gadre al. (2023) provide over 300 baseline experiments, and find that, similar to the language-only setting, smaller, more selective dataset can lead to models that generalize better than larger datasets. Of course, this still comes with the samne limitations as all previously discussed sections, where the exact evaluation setting plays a large role in what data is "best". Specifically, DataComp evaluates models' abilities to perform image classification and retrieval tasks, but does not evaluate generative capabilities. This suggests that distribution matching methods may fare well on DataComp, while an evaluation of generative capabilities may prefer data selection methods that favor distribution diversification. DataPerf (Mazumder al., 2023) provides another entry for data selection research. The DataPerf benchmark contains 4 data selection settings: vision, specch, debugging, and language. The various settings allow rescarchers to test their algorithms for selection, augmentation, quality assesment, and cleaning. Mazumder et al. (2023) find that automated methods tended to perform better than manual cleaning or filtering; specifically, they found that submissions succeeded with automated methods for recognizing noisy images and labels, identifying mislabeled images, correcting class imbalance, and selecting and enhancing images from the long-tailed distribution of classes.
By providing a shared resource where researchers can test their methods, benchmarks and competitions like these can help to increase the pace of research, well as giving a direct comparison between methods. Specifically, DataComp has evaluated 21 submissions and DataPerf has evaluated 54 submissions. 20 Some of the promising methods from these benchmarks may be useful for language domains, but their transfer may be limited due lo their domain-specific nature. Benchmarking data-centric mothods, and data selection in particular, for the language domain has not been fully utilized and should be in the future to accelerate the pace of data selection research.
19 As of 02/06/2024, found at https:///vee /ttt::///ww.dttacommpp..
daw.comp. wataaaccm..alleaaeeeeeeeee ai/leaderboard html.
20 Across vision, speech, debugging, and language tasks, as of 02/06/2022 /2024, found https://www..stapprt org/
40

---

Published in Transactions on Machine Learning Research (07/2024)
9 Related Topics
Data cleaning. As discussed in sections 3.2 and 3.5, some utility functions can determine both whether an entire data point should be removed from the dataset and also whether a chunk of text should be removed from a data point. Data cleaning refers to the latter scenario. While some of the selection methods described in this work can also be used for cleaning pretraining dała, there are additional data cleaning methods not covered here. For example, in supervised settings data can be labeled incorrectly and methods for detecting incorrectly labeled data often assume that such data are outliers, utilizing outlier detection methods to clean (or remove) the incorrectly labeled data (Abedjan et al., 2016; Narayan et al., 2022). Data cleaning can also refer to imputing missing values or finding inconsistencies across data points (see Li et al. (2021) for more details on data cleaning in non-language domain data).
Dataset distillation and coreset selection. Dataset distillation (also known as dalaset condensation) and coreset selection (also known as data pruning) have been extensively studied for reducing the size of training data. Dataset distillation aims to synthesize a small dataset such that the model trained on that synthetic dataset achieve performance akin to one trained on the full dataset. Dataset distillation is first introduced by Wang et al. (2018b), where the model weights are treated as a function of the distilled dataset (outer optimization) and the distilled dataset is optimized (inner optimization) so that randomly initialized models trained on it with one-step gradient descont could achieve good performance. Such a bi-level meta-learning framework often incurs substantial computational costs. To improve the scalability and effectiveness of dataset distillation, subsequent research (Zhao et al., 2020; Lee et al., 2022b; Zhao & Bilen, 2021) has employed gradient matching as the supervision signal, aligning the gradients of model parameters trained on the original dataset and the distilled dataset. Moreover, researchers have proposed techniques such as feature distribution matching (Wang et al., 2022a; Zhao & Bilen, 2023; Zhao et al., 2023a), training trajectory matching (Cazenavette et al., 2022; Cui et al., 2023b; Guo et al., 2024), and kernel methods (Nguyen et al., 2020; 2021; Zhou et al., 2022).
In contrast, the objective of coreset selection (Mirzasoleiman et al., 2020a; Borsos et al., 2020; Killamsetty et al., 2021b; Guo et al., 2022; Killamsetty et al., 2021a) is to select a subset from the given datasct so that model trained it achieves optimal performance, which better aligns with the topics discussed in this paper. Many coreset selection methods design series of scores and rank the data instances accordingly. Within this realm, geometry-based corcset selection methods selects the examples based on their distances from k-means cluster centers (Chen et al., 2012; Sener & Savarese, 2018; Sorscher et al., 2022; Xia et al., 2023). Additionally, predictive uncertainty and model loss metrics can also serve as criteria for ranking data instances (Coleman et al., 2019; Pleiss et al., 2020; Paul et al., 2021; Zheng et al., 2022).
There also exist coreset selection methods that are optimization-drivenn CRAIG (Mirzasoleiman et al., 2020a) optimizes the coreset selection that the sum of the gradients the subsets closely match the gradients the full dataset. This approach has been extended to handle noisy data settings in the subsequent work (Mirzasoleiman et al., 2020b). GLISTER (Killamsetty et al., 2021b) and GRAD-MATCH (Killamsetty et al., 2021a) focus on coreset selection on noisy and imbalanced datasets and propose to match the gradient of the corcsct with the training of a held-out validation set. The ADACORE (Pooladzandi et al., 2022) method incorporates the second-order gradient information for better gradient matching. Notably, the techniques of coreset selection have found application in LLM fine-tuning (Liu et al., 2023b; Xia et al., 2024b). For example, the LESS (Xia et al., 2024b) method selects influential data for supervised fine-tuning based on the similarity of gradients evaluated on the coresct and the validation set.
Data attribution and valuation. Data valuation and attribution focus on understanding how each training data point affects a model's predictions. Methods for valuation and attribution can help to identify potentially mislabeled data, identify brittle predictions, quantify the affect of a training data point on an evaluation data point, and even quantify train-test contamination. These methods tend to be computationally expensive, although recent progress has improved their efficiency.
In recent work, Ghorbani & Zou (2019) propose DATA SHAPLEY, a principled framework for data valuation, where the goal is to quantify the value of each data point to a predictors performance. The DATA SHAPLEY
41

---

Published in Transactions on Machine Learning Research (07/2024)
value for each data point can then be used the utility function in data selection method for either distribution matching or diversification purposes. Nlyas et al. (2022) present DATAMODELS, framework used to predict a models performance on a data point x(1), given that the model is trained on a known subset of training data. They demonstrate that even using a linear function as the predictor, their datamodelling Tramework is capable of accurately predicting model putputs and counterfactuals, as well as encoding a metric for data point similarity. Similarly, Park et al. (2023) propose TRAK, a data attribution mothod that can make accurate counterfactual predictions, trying to answer the question "why did my model make this prediction". They show that TRAK is plausible method for determining which training data was most impactful on model's prediction.
Data augmentation. Data augmentation methods are used to generate new data by modifying the existing data points in a dataset. Similar to data selection, data augmentation also aims to shift the training distribution. Data selection is often used to improve the data coverage in domains where there is limited data (Li et al., 2022b; Chen et al., 2023a) and can be used in conjunction with existing data selection methods (e.g., domain-specific selection in $3.4 or "selecting from auxiliary data to improve generalization" in $7). For example, Albalak et al. (2022a) use a policy gradient along with information from an auxiliary task to improve their main model. However, the quality of the coverage produced from augmentation may be limited, as the semantics of augmented data can diverge from the underlying distribution of language, leading to an undesirable distribution shift and possibly a decrease in performance (Li et al., 2022b; Chen et al., 2023a). Thus, care should be taken when performing data augmentation to ensure that the augmentations do not stray too far from the desired distribution. Nonetheless, data augmentation can still be a useful tool, and we refer readers to existing surveys on the many methods for augmentation in NLP (Chen et al., 2023a; Feng et al., 2021), code (Zhuo et al., 2023), and computer vision (Yang et al., 2022).
Data curation. Data curation is a set of processes surrounding the discovery, organization, integration, annotation, cleaning, storage and maintenance of data (McLure et al., 2014; Freitas & Curry, 2016; Thiru- muruganathan et al., 2020). Some of the methods that are used for curation overlap with those from data selection (e.g., cleaning and filtering), but there are also methods from data selection which are not included in data curation (e.g., heuristic filters, data mixing, and filtering toxic and explicit content). Data selection is, generally speaking, a process that occurs after data curation, once a practitioner is prepared to train a model. After the data has been curated, then data selection methods can be used to create dataset for the desired purpose.
Curriculum learning. While data selection aims to determine which data points a model should train or evaluate on, and how many times they may be repeated for training, curriculum learning aims to improve a model through an orthogonal goal. Curriculum learning is often motivated by the idea that the training data should be increasing in complexity according to the models current capability, and aims to determine when a data point should be trained on. Curriculum learning has found limited adoption in the natural language domain. For example, Graves et al. (2017) use multi-armed bandits to determine an appropriate curriculum for language modeling with LSTMss Xu et al. (2020) demonstrate that curricula can be used to improve learning in natural language understanding tasks for a BERT model. More recently, Fan & Jaggi (2023) propose irreducible curriculum for language model pretraining, which prioritizes training on samples with higher learnability (motivated by RHO-Loss (Mindermann et al., 2022)). Curriculum learning is one aspect of data-centric Al that has not yet seen a large adoption in large language models, in part due to its limited success.
Active learning. Active learning is a long-studied problem that aims to answer the question: given unlabeled data points, how should fixed annotation budget be used to optimally train a model for supervised tasks, and has been applied to wide variety of settings ranging from text classification (Lewis & Gale, 1994) to in-context learning (Wu et al., 2023). Active learning is closely related to data selection as both fields consider the question of how to find the optimal data to train a model, but focus on different settings. Unlike data selection, active learning explicitly focuses on supervised training. Please see the available surveys on active learning in NLP (Zhang et al., 2022b), and general deep learning (Ren et al., 2021; Zhan et al., 2022; Tharwat & Schenck, 2023) for a deeper discussion on the topic.
42

---

Published in Transactions on Machine Learning Research (07/ (204))
10
Discussion
10.1 Test set decontamination
Understanding a model's capabilities requires evaluation on a lair benchmark, but the training of language models on web-scale data has raised concerns and speculation about the possible inclusion of evaluation benchmarks in training data. Given the scale of data it is not feasible to manually check, so automated methods have been designed to detect the existence of evaluation data in the training set (Rae et al., 2021; Li et al., 2022a; 2023b; Magmusson et al., 2023; Elazar et al., 2024), as well as detecting models that have already been trained on evaluation data (Carlini et al., 2018; Jagielski, 2023; Marone & Van Durme, 2024). Jacovi et al. (2023) on the other hand, suggest three strategies for reducing the risks of contamination. Data decontamination is an important step in ensuring the integrity of model evaluation, and is included in the discussion section because this critical step is relevant, and should be roquired, for all stages of model training.
Detecting test set contamination with n-grams. Test set contamination methods often follow a similar form to deduplication methods. When decontaminating MassiveText, Rae et al. (2021) compute the 13-gram Jaccard similarity between train and test documents (as originally proposed by Lee et al. (2022a)) and remove train documents that have a Jaccard similarity over 0.8, the same exact method that they use to detect duplicated documents. Similarly, Brown et al. (2020a) and Gao et al. (2020) use 13-gram overlap filtering to detect contamination in their respective training sels. To make decontamination easier, some tools have been developed. For example, CarperAl released software for decontaminating evaluation data from a training datasct using MINHASHL.SH." 21 Additionally, Marone & Van Durme (2024) propose Data Portraits, method of investigating the data that a model was trained on. Data portraits are an artifact that records training data, allows downstream inspection, and enables answering questions about test set leakage and model plagiarism through novel membership testing tool, strided bloom filter. In addition to the new method, they release a demo of their tool which uses 50-character grams to detect overlap between training data and an input sequence. Elazar et. al. (2024) indexed several corpora using elastic-seareh, a reverse index, and used it to detect exact matches of multiple test-sets used for evaluation. Finally, Merrill et al. (2024) built an efficient automtaton named CDAWG (Inenaga et al., 2005), that allows a flexible n-gram match to the longest suffix that matches an input string in the (indexed) training data.
Protecting test data with canaries. A canary string a unique sequence of characters that is included within a dataset, or document, used to empirically evaluate, or audit, whether a language model was trained on that document (Carlini et al., 2018). By including canary strings within benchmark, they can protect the integrity of an evaluation metric, demonstrating the likelihood of a model having been trained on that evaluation benchmark. For example, BIG-bench (Srivastava et al., 2023) includes a canary GUID string in each task to allow for detecting whether models were trained on the evaluation data. 23 To detect canaries, Carlini et al. (2018) propose the secret sharer framework for efficiently extracting unique, secret sequences such as personally identifiable information, and demonstrate that such information can be extracted from model. Jagielski (2023) expand the secret sharer framework, and demonstrate how to interpret. canary exposure by relating it to membership inference attacks and differential privacy.
Detecting models trained on test data. Proving that black-box models have trained on data point can be very challenging, but also important to ensure that their results can be trusted. Because their data is not disclosed, it is impossible to know with certainty what their training data was, but some methods have been proposed as efforts to address this issue. Shi et al. (2024) propose min-k% prob, using the intuition that unseen examples are more likely to contain outlier words with low probabilities according to the model, whereas unseen examples are less likely to exhibit such low probability words. While this is not explicitly a method of decontamination, it is useful tool for determining how contaminated a pretrained models data was. This tool can be used for benchmark example contamination detection, privacy auditing for machine unlearning, and copyrighted text detection. Oren et al. (2024) present another method for detecting
21 CarperAl decontamination CitHub repository chttpe://dataporttee org/ 23pIG-bench canary GUID README.
43

---

Published in Transactions on Machine Learning Research (07/2024)
contaminated data based on the observation that models have tendency memorize the order of examples as they exist in the dataset, and provide provable guarantees of test set contamination through framework of statistical tests.
Decontaminating code data. Decontamination in code data is particularly important, as Li et al. (2022a) suggest, that competitors coding competitions lend to publish their solutions online after a compelition. One solution to this problem is to use a temporal split when dividing data into training and evaluation. To ensure that test data does not exist in the training dataset, the training dataset can be constructed only from files that originate prior to the test competition. This ensures that the training data includes only information that would be available for human participants of the competition. In a more traditional method, StarCoder (Li et al., 2023b) decontaminates The Stack by removing files that contained either exact matching docstrings or solutions from the HumanEval (Chen el al., 2021) or MBPP (Austin et al., 2021) datasets, docstrings from APPS (Hendrycks et al., 2021), questions from GSM8K (Cobbe et al., 2021), or prompts from DS1000 (Lai et al., 2023).
10.2 Tradeoffs between memorization and generalization
While most methods of data selection discussed here, data deduplication in particular, suggest that memo- rization is a negative model property, this is not always the case. There are truths about the world which always hold and memorization is indeed a desirable property in those cases. For example, when answering the question "what is the formula for Pythagoras' theorem?" we expect a model's answer to deviate very minimally from "a² + b2 2" and scrious deviation can have real world consequences if such a model was used in an education setting. However, generalization is still desirable in this situation as changing the variable names should still be recognized as correct (e-g-, "s} + s2 s3"). Furthermore, memorization is also desirable for code data, such as specific API calls in tool usage for LLMs (Schick et al., 2023; Pan et al., 2023), and syntax for coding languages (Chen et al., 2021; Li et al., 2022a: Allal et al., 2023). Memorization is desirable in many other contexts as well, and Brown et al. (2021) have theoretically proven that in many cases memorization is a requirement of any reasonably accurate training algorithm.
On the other hand, it has been demonstrated that. not only do models memorize the desirable information, but also the undesirable information such as personally identifiable information, phone numbers, credit card information and much more. Carlini et al. (2020) demonstrate this with a simple attack for extracting verbatim sequences of an LLMs training data, finding that although data points from the training set do not have noticeably lower loss than test data points on average, there are some individual examples that are indeed memorized. Biderman et al. (2023a) study whether it is possible to predict which sequences from the training data will be memorized by using smaller models, or partially trained models. Carlini et al. (2023) demonstrate that memorization grows significantly with three factors: the capacity of a model, the number of times a sample is repeated in the training data, and the number of tokens used to prompt the model to emit the memorized data. While the third factor, the number of tokens in prompt, is not in the control of model developers, the first two are. Model developers can choose to use smaller models to satisfy the needs for given use case, rather than relying on massive models which generalize across many use cases, but tend to memorize data. Furthermore, deduplication has been demonstrated as a strong method for minimizing memorization. However, as previously discussed, deduplication needs to be intelligently applied so that "good" memorization can occur (e.g., facts), while reducing "bad" memorization (e.g., PIT).
10.3 There's no free lunch for filtering
Overly filtering the data can lead to undesirable biases in the model. In particular, Dodge et al. (2021) find that blocklist filtering used to create C4 (Raffel et al., 2020) disproportionately removes text written by, and about, minority individuals. Additionally, Welbl et al. (2021) find that removing toxic content (as determined by the Jigsaw Perspective API24) can lead to worse evaluation loss and disproportionately affects texts about and by marginalized groups. This finding was also replicated by Longpre et al. (2023c). These
24ttttps:/ httpppss//ppeepeetivaa...ccm
44

---

Published in Transactions on Machine Learning Research (07/2024)
works demonstrate the need for filtering strategies with improved precision (remove less text that is desirable) and recall (remove more text that is undesirable).
10.4 Tools for data selection
When performing data selection, first step should generally be to understand the raw dataset, and a number of tools have been developed to assist in data exploration and auditing. For example, researchers at A12 developed a search engine over the C4 dataset, 25 and Piktus et al. (2023) create a search engine over C4, the Pile, ROOTS, and text captions from LAION. 26 Elazar et al. (2024) describe their What's in my big data tool, which can search for and count n-grams in a number of popular pretraining corpora. Additionally, their online demo contains tools for exploring the internet domains included in datasets, a visualization of the amount of text overlap between corpora, and a visualization to understand the length of documents in each corpus." 27 Finally, Liu al. (2024a) developed the INFINI-CRAM which uses sulfix arrays as a method for language modeling, ultimately demonstrating competitive performance on the dodPajama, the Pile, and Dolma corpora in addition to being able to quickly search for n-grams within any of those corpora. 28
Once the data has been explored and audited, the next step of data selection is to devise the appropriate selection mechanisms and apply them over the data. Some of the methods discussed in this work have been implemented as open-sourced tools, allowing anyone to easily make use of them. The CCNet pipeline (Wenzek et al., 2020) is a commonly used tool for downloading and cleaning Common Crawl data. The CCNet. pipeline performs deduplication at the line level, performs language identification with a fastText linear classifier, and can train an n-gram language model to calculate perplexity per document. Soldaini et al. (2024) open-source tools for identifying non-language text, language identification, personally identifiable information, toxic text, text quality, and duplicated data." 30 Xie et al. (2023b) provide a pip-installable tool for selecting data with a distribution similar to a target dataset, that can be used either for domain-specift selection, or for quality filtering.* 31 Lee et al. (2022a) provide the code to run their ExactSubstr deduplication," 32 and Computer (2023) provide tools for computing exact duplicates with a bloom filter, and fuzzy deduplication with locality sensitive hashing. 33 Additionally, the tools provided by Computer (2023) contain code for calculating a wide variety of data signals including heuristics, quality, duplication, and toxicity. Additionally, CarperAl provide pip-installable package for decontaminating datasets from nunber of openly available benchmarks. 34 Finally, Penedo et al. (2024) provide a planform-agnottic python library that can filter and deduplicate data efficiently.
For a more comprehensive and up-to-date list of available tools, see the Foundation Model Development Cheatsheet (Longpre et al., 2024).
10.5 Considerations when applying data selection to your setting
The first step in determining appropriate actions for data selection is to determine whether you are in a setting where you wish to increase the distribution coverage from your data, or if you wish to select a subset from your data. For pretraining, the goal will usually be to increase coverage, unless the model has a well-defined downstream use case (e.g., for medical domain), in which case the goal can be to select subset. of general case. For multitask training and instruction tuning, the goal
data which is most similar to the downstream is generally to increase coverage in order to ultimately produce capable general purpose model. Additionally, depending on the number of parameters in the model you plan to train, you may have a range for the desired number of training tokens according to your compute budget (Kaplan et al., 2020; Hoffmann et al., 2022;
httpps: htattp::/ce-searcc..o apps. allenai. org/
26 code at httpp: http://ggtttbbcccc con/huggingtace/ gaaa, demonstration at functioning of 02/07/2024
http:://ww.....pppcc//////pcaaaa is not
"httpp:///iibd wwt.:aaa.dpp.............aa.aaaa
awwemai.
gran
htttta///gghuuu.comm//ccoroeeeee net
45

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Penedo al., 2023). The various filter sensitivities will need to be adjusted depending the mumber of tokens in the raw data in order to achieve the desired number of training tokens. For task-specific fine-tuning, the setting depends on how large the task-specific dataset is. Generally, if the dataset is relatively small, the goal of data selection will be to increase coverage over the expected test distribution. However, the lask-specific dataset is very large then models can plausibly model the distribution with fewer data points by removing redundant information from the dataset, and costs can be reduced.
The next step in the decision making depends how much information you have about the test distribution, this will determine which utility functions are available to you. Depending whether you have data sampled from the test distribution, you may be able to find an appropriate model-based utility function, otherwise you will need to determine appropriate heuristics. Furthermore, the available utility functions may limited by the amount of compute available.
Cost is another considcration when determining how to apply data filtering. Cost can be calculated money, time, effort. The ordering of filters used can have great impact on the final cost. An appropriate choice of filtering order is from least expensive to most expensive, that the more expensive selection methods will be executed on fewer data points. Additionally, Coleman et al. (2020) demonstrated that data selection methods requiring model can be improved in efficiency by using proxy models much smaller than the actual model being trained, to further reduce costs.
46

---

Published in Transactions on Machine Learning Research (07/2024)
11 Future Directions: Challenges and Opportunities
11.1 Accelerating research on data selection
Research on data can be a very slow and tedious process, potentially requiring significant investments of time, compute, and human effort. Here we describe four directions of future research that can reduce the effort required and the barriers to entry for rescarch in data selection.
Scaling down. One direction that can enable more researchers to participate, as well as allowing for faster iteration, is scaling down the current. dataset sizes and models. For example, Kaddour (2023) demonstrate that it is possible to train a model on 745x less data, but lose only 1.9% performance on GLUE (Wang et al., 2018a) and 2.5% performance on super-natural instructions (Wang et al., 2022b). Additionally, Polo et al. (2024) show that wide variety of evaluation benchmarks can be distilled into only 100 data points, while maintaining an accuracy within 2% of the true accuracy. One area that is still uncertain is whether the results from small models scale up to larger models, or if there is any way to predict the results on large models, based on experiments from small models. This is an active area of research.
Metrics that directly evaluate data. One very high impact, but challenging, way to accelerate research on data selection is to develop metries that directly evaluate the data chosen by a selection method, thereby removing (or reducing) the need for expensive model training. Recalling the definition of data selection in $2.2, the goal of data selection mothod is inherently tied it's ability to minimize or maximize the objective function of a trained model. The dependence of data selection on model training significantly increases the iteration time for experiments, reducing the amount of experimentation that researchers can perform. 35
Preliminary works have been done in this direction, including some works on data attribution and valua- tion (Ghorbani & Zou, 2019; llyas et al., 2022), and data measurements (Mitchell et al., 2022). Works on data valuation and attribution (discussed in $9) attempt to better understand how each data point affects a model's predictions, but may be used as inspiration for methods that directly evaluate the data itself. Additionally, there are number of metrics that directly measure the intrinsic characteristics of data including distance, density, diversity, tendencies (mean, median, mode), and association (correlation, mutual information), but these measures have yet to be connected with model's performance on downstream data (Mitchell et al., 2022).
Developing metrics that directly evaluate data can significantly decrease the time spent on method development and increase the amount of exploratory work possible. Furthermore, removing the need to train models (which can be very costly) can allow for lover-resourced organizations and individuals to contribute to the field, improving diversity and inclusion in the process of large model development.
Data-centric benchmarks and challenges. Another high-impact direction for increasing the rate of rescarch on data selection the development of data-centric benchmarks and challenges. One hurdle of advancing data selection is that works are done across a wide variely of settings and cannot be directly compared. Specifically, much of data selection research using heuristic filters is performed part of collecting new dataset. This makes it impossible to compare selection methods with another dataset as they may contain different quantities of data, and sometimes even use varying model architectures and sizes. In other domains, such as vision and vision+ challenges and benchmarks have been developed that provide well-founded evaluation settings that allows researchers from many institutions to participate in a combined effort to push the frontier knowledge and capabilities in data selection (discussed " "Increasing the rate of data selection research" " in $8). Creating similar challenges for language models can go a long way in encouraging progress.
One challenge for developing benchmarks is there is no current consensus for what the appropriate loy settings are for data selection research. With the exception of the Loose track from BabyLM (Warstadt et al., 2023), no prior benchmarks or challenges have been created for data selection (BabyLM is also not specifically for
For reference, training billion parameter model 50 billion tokens takes roughly days train A100 GPUs using GPT-NeoX (Andonian et al., 2023; Albalak et al., 2023a).
47

---

Published in Transactions on Machine Learning Research (07/2024)
data selection, but rather to improve the speed of development architectures and hyperparameters for language models). The DATACOME (Gadre et al., 2023) and DATAPERR (Mazumder et al., 2023) challenges can provide inspiration for creating similar challenged and future benchmarks in language.
Due to the large number of training settings (pretraining, interucctiontuuinng alignment, in-context, task- specific) and goals (general-purpose, knowledge-intenssiv, chat, etc.) for language models, there are a wide variety of benchmarks and challenges that would be beneficial for the community. Creating challenges and benchmarks allows for direct comparison between methods, lowers the bar for entry into the field, and encourages open progress on this important task.
Open-sourcing tools and best practices. As the field develops in the open, new rules and best practices will emerge. Open sourcing the tools that implement these best practices is crucial. The combination of new best practices being developed in the open, and the open-sourcing of such tools can significantly reduce the overhead required to get started into research on data. This also allows researchers to focus on specific components of the pipeline, further accelerating research progress.
11.2 Better understanding the properties of the target distribution
Having a clear goal is crucial to defining and developing successful data selection methods. Using the terminology and conceptual framework developed in this work can improve the understanding of what data selection methods may be appropriate depending on the properties of the desired larget distribution. For example, the goal of data selection is very clear for task-specific fine-tuning, but less clear for preference fine- tuning (PreF"T). Because the definition of success is less specific for PreFT,, developers will likely benefit from data diversification methods. For example, it may be possible to utilize existing auxiliary data for preferences by drawing inspiration from methods in $7 ("Selecting from auxiliary data to improve generalization' or improving robustness across preference groups by utilizing methods for robustness (97 "Selecting data for robustness" $8 "Bias, fairness, and robustncss").
11.3 Shifting compute time from model training to data processing
It is now well known that training data plays a very significant role in the performance a model, therefore creating an increased desirability spend more compute on data processing and selection (and possibly away from model training). To improve beyond what is possible only through heuristics, it may be necessary utilize more expensive methods.
One example of how this may be accomplished is through the use of methods which have previously only been used in low-data settings (c.g., alignment) and applying them to scttings with greater amounts of data (e.g., pretraining). For example, methods inspired by model-based filtering, or reward model-reweighttnng techniques can be used to improve the "quality" of pretraining data. Additionally, techniques for generating synthetic data can be used to create more diverse training data (Bradley et al., 2023) and methods previously used only for rod-teaming (Samvelyan et al., 2024) can be used to generate training data for more robust models. As compute gets cheaper, it becomes more feasible to spend the cost. of directly applying the existing methods. However, another avenue worth exploring is using cheap approxinations for utility functions, such as computing similarity using bag-of-n-grame grams and hash functions rather than from model representations.
Another method for improving data that would require large amounts of compute is additive data selection. Currently, most data selection methods (especially for pretraining) remove undesirable content from data points. However, it possible that in the future it will be beneficial to have additive data selection methods. For example, would be possible to use a language model to infill information within a data point. One example would be to include explanations for some phrases: e.g., "the basketball player wasn't. 't able to move due to the double dribble rule so he passed the ball to his teammate" where the italicized text has been added by an infilling model.
48

---

Published in Transactions on Machine Learning Research (07/2024)
12 Conclusion
In this work, we have compiled an in-depth review on the current state of data selection and present unified framework with which to consider and compare the wide variety of methods. This survey has described the current. best practices and considerations when selecting data for training a language model. Additionally, the unified framework has allowed us to define and describe some potentially fruitful future directions of research which we have highlighted throughout the paper. However, there is much more room for innovation and improvement in data selection.
Acknowledgments
This material is based on work that is partially funded by an unrestricted gift from Google. This work was supported by the National Science Foundation award #2048122. The views expressed are those of the authors and do not reflect the official policy or position of the US government. The work BH and SC is partially supported by the UC Santa Barbara IEE IGSB SW Impact Grant. We want to thank Luca Soldaini for sharing their figure that was adapted into Figure We are also thankful to Lucy Li for sharing her summary of different filtering mechanisms used for language models of recent years. Some of the figures include icons from the Noun Project:
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint 2Xix:2303.005400 2023.
Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. llyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang. Detecting data errors: Where are we and what needs to be done? Proc. VLDB Endow., 9(12):993-1004, aug 2016. ISSN 2150-8097. doi: 00.14778/2994509229944 URL https:///oi. org/ 10. 14778/2994500 29119///999940092999511.
2994518.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877, 2023.
Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller. One-network adversarial fairness. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2412 2420, 2019.
Amit Agarwal, Hema Swetha Koppula, Krishna P. Leela, Krishna Prasad Chitrapura, Sachin Garg, Pavan Ku- mar GM,, Chittaranjan Haty, Anirban Roy, and Amit Sasturkar. Url normalization for de-duplication of web pages. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09, pp. 1987-1990, New York, NY, USA, 2009. Association for Computing Machinery. ISBN URL https://dol. org/10. 10. 1145/1645953, 1646283.
9781605581123, doi: 00.1145/1645953.000
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5799-5811, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1 20655/v1/20211111 http://////21...mmmmmmme https://aclantholoog org/2022 emmlp-main. 468.
URL
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. HTLM: Hyper-text pre-training and prompting of language models. In International Conference on Learning Representations, 2022. URL https: /ttp::///ee//eeeeeeeeeee wttp://oommmiieepp-Piiii.
Alon Albalak, Varun Embar, Yi-Lin Tuan, Lise Getoor, and William Yang Wang. D-REX: Dialogue relation extraction with explanations. In Bing Liu, Alexandros Papangelis, Stefan Ultes, Abhinav Rastogi, Yun-Nung Chen, Georgios Spithourakis, Elnaz Nouri, and Weiyan Shi (eds.), Proceedings of the 4th Workshop on NLP
49

---

Published in Transactions on Machine Learning Research (07/2024)
for Conversational AI, pp. 34-46, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v1 /006////1/2022.1/44covaa....... URL https://aclanthhooog wrg/2022 n1p4convai-11.
Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang- FETA: A benchmark for few-sample task transfer in open-domain dialogue. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Melhods in Natural Language Processing, pp. 10936 10953, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguisties. doi: 10.18653 w1/2222.2222222 emnlp- wwwlp-main.niiie URL https://aclantholl. org/2022. emnlp-man main. 751.
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang- Efficient online data mixing for language model pre-training, arXiv preprint ar arx::23312.044000 2023a.
Alon Albalak, Colin Raffel, and William Yang Wang. Improving fow-shot generalization by exploring and exploiting auxiliary data. In seirth- Conference on Neural Information Processing Systems, 2023b. URL https://openrreviet nett/foruuniddd-DDDXed.tl
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, al. Santacoder: don't reach for the stars! Xiv preprint Xrx::2200 000988, 2023.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songs, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NcoX: Large Scale Autoregressive Language Modeling PyTorch, 2023. URL https://wwez http:///wwwggtth..boe. www/co/m/////oeeeee www.eeeeereeggttteeeeeeeee
Vamsi Aribandi, Yi Tay, Tal Schustor, Jinfong Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mchta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. arxi preprint arXim:211 10010 2021.
Peter Auer, Nicolò bandit problem. SIAM Journal on Computing, 32(1):48-77 77,, 2002.
Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint XrXi::2108.0 07732, 2021.
Amittai Axclrod. Cynical selection of language model training data. arXiv preprint arXiv:
2017.
Abdul Azeemi, Ihsan Qazi, and Agha Raza. Data pruning for efficient model pruning in neural machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Länguislies: FMNIP 2023, pp. 236 236-246, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.fnndi findings-emnlp.16 URL https://aclantholo.g .rgg/ 2023. f See3.findings-ee indings-emmll..f 18.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human foedback. arXiv preprint wxXi:22244008622 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldic, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint or orXiv:2212.080 08003, 2022b.
Alvaro Bartolome, Gabriel Martin, and Daniel Vila. Notus. https://gtthaa..cmmargg////
2023.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' D'Amour, Jacob Fisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint ar 401.::2401.01879, 2024.
50

---

Published in Transactions on Machine Learning Research (07/2024)
Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets. In Houda. Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10107-10121, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023. 112..emnlp-main..5 URL bettps:/aclanttiee httphhoooooooooo/2333333 emnlp-main. 625.
Stella Biderman, USVSN Sai Prashanth, Lintang Sulawika, Hailey Schoelkopf, Quentin Gregory Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. In Thirty-scventhh Conference on Neural Information Processing Systems, 2023a. URL attps://openreviee htt/forrumid=IqQDviie
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Authony, Herbie Bradley, Kyle O'Brien, Erie Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2397 2430. PMLR, 23 229 Jul 2023b. URL attps://proceedii mlr press//22ss//// wttp:pss///2///bddrrraaaaat htmm..
Sumon Biswas and Hridesh Rajan. Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline. In Proceedings of the 29th ACM joint meeting on Europeon software engineering conference and symposium on the foundations of software engineering, pp. 981-993, 2021.
Su Lin Blodgett, Lisa. Green, and Brendan O'Connor. Demographic dialectal variation in social media: A case study of African-Americal English. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1119-1130, Austin, Texas, November 2016. Association for Computational Linguistios. doi: 10. 10.18653/v1/D16-11 URL attps://aclanthologg/ hatpa/antoooogorr///1-----1-1 1ttg/166-110.....
Burton H. Bloom. Space/time trade-offs hash coding with allowable errors. Commun. ACM, :3(2):422 426, jul 1970. ISSN 0001-0782. doi: 01555/62666.3060 URL https:///oi..rg// 1145/3666666.... 362692.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index. ar preprint arXiv:2310.12911. 2023.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in newral information processing systems, 13:14879-14888 2020.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluss Mårquez, Chris Callison-Buree and Jian Su (ods.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 682-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 18653//11/D55-077 URL https://acllatthooogy.ooot
Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through ai feedback. arXiv preprint wrXiu:2310.13035 2023.
A.Z. Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complerity of SEQUENCES 1997 (Cat. No.97TB100171), pp. 21-29, 1997. doi: 11L109/SEQUEN.1997 199.66990997666600000
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM
SIGACT Symposium on Theory of Computing, STOC 2021, pp. 123-132, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380539. doi: 10. 10.1145/34069553451110 URL https://dbi.orr//0.. http://gg//..11////255...... 3451131..
51

---

Published in Transactions on Machine Learning Research (07/2024)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020a. URL https: pptpe///eepoig....... latperrfflesseeeeeeeeeeeeeeereer wttp:/feleeepppeeeeeeeeee wttee///lll//////////ee http:///020000111111111100000000 http:///0660000046111111100e0ea.
Paper pdf..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Nly Sutskever, and Dario Amodei. Language models are few-shot learners. NewrIPS, 2020b. URL attps://proon sttps://proceedings neurips cc/paper/2000/ 2/1e//1e/22//////////// 1457
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint ar Xix:24002 15391, 2024.
Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selection for large language models. Xiv preprint arXin:2307.06290 2023.
Nicholas Carlini, Chang Liu, Ular Erlingsson, Jernej Kos, and Dawn Xiaodong Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2018. URL https://api. semanticscholar wrg/CorpusID:11007550
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Berbert-Voss. Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2020. URL
https://api. semanticscholar wtt.ssccooaar.oooorruusssss.ss :29116229..
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviers http:///ppeeeeeeeeeeeeeetoraraaa
net/forrmmid--TaRRT 1ck.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel- Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: Survey Certification.
//openreview.o //openneviewnne///orrmmr wett///rorm/dddxx44akpp..p.
Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, and Stella Biderman. Suppressing pink elephants with direct principle feedback. arXio preprint orXiw:2402.088 2024.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus. arXio preprint arXi::2000.11 201.1401.1,, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 4750-4759, 2022.
Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. Bias in machine learning software: why? how? what to do? In Proceedings of the 29th ACM Joint Meeting on European Software Engineering
52

---

Published in Transactions on Machine Learning Research (07/2024)
Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, pp. 429 440, New York, NY, USA, August 2021. Association for Computing Machinery. ISBN 978-1-4503-856--6 doi: (0.1145/3469264434663377 URL https://dl. acm. wwwp/ogg////ooooo /ttp///....1///4166666..1 34615466
Haw-Shiuan Chang, Erik G. Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advonces in Newral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1002-1012, 2017. URL https //proceeeocce.... htmll.
weurips.cc/paper// wttpcccpppppe//./17..// htthhhhhhhhffaa/// 204011122222208000555333111 neurips.
Ting- Yun Chang and Robin Jia. Careful data curation stabilizes in-context learning. arXiv preprint
20t.22212.1076. 2022.
Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing, STOC '02, pp. 380-388 New York, NY, USA, 2002. Association for Computing Machinery. ISBN 1581134958 https://doi.. owg. 000.1115550997.555850
doi: 0411145550907750000 URL
Kyle Chayka. Is A.I. Art Stealing from Artists? The New Yorker, February 2023. ISSN 0028-792X. URL https://www.neworkerr nwwyorker........ http///u/wr/iiiittettoo//////aaa sttal------ weapiinng-o-aattiiiinn Sec-
tion: infinite scroll.
Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. An Empirical Survey of Data Augmentation for Limited Data Learning in NLP. Transactions of the Association for Computational Linguistics, 11:191 211, 03 2023a. ISSN 2307-387X. doi: 10.1162/tacl a 00542. URL https://doi.o. 10. 1aao://aac_a_005a00
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint w:Xiv:2307088000 2023b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0339 2021.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Ré. Skill- it! data-driven skills framework for understanding and training language models. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Sys- tems, volume 36, pp. htt.:///eppppeeeeeeeeee/////aaae
Curran Associates, Inc., 2023c. URL https: Confereeeee pdff
meeriinggs.neeieeeeeeeeee neurips.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv:1203.3472, 2012.
arxiv preprint
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprinl 011:3300 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1 113, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling intruction-finetune language models. arXiv preprint xrX::22100.11410 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retchhro Nakano, et al. Training verifiers to solve math word problems. Xix preprint or Xin:21 10. 14168, 2021.
53

---

Published in Transactions on Machine Learning Research (07/2024)
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasolciman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2019.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovee, and Matei Zaharia. Selection via proxy; Efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL https://openreviiet wwt. net:fforuu'1d-+tug200007
Together Computer. Redpajama: an open dataset for training large language models, October 2023. URL
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'cchb-BBu,, E. Fox, and R. Garnett (cds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
wwweeeeddinnn.uuuip.... cc/pacer. pdf.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation lcarning at scale. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Compulational Lingwistics, pp. 8440 8451, Online, July 2020. Association for Computational Linguistios. doi: 20/653/v112220..a org/2020 acl-main. 747.
URL
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.0187) 2023a.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference on Machine Learning, pp.
PMI.R, 2023b.
Emilia David. Ai image training dataset. found to include child sexual abuse imagery.
The Verge, Decomber 2023. URL https://www https://www.theverge comm/2033112/2//4000111 generative-ai ooaaae--laaooccrrgggololeeeeaaaa
7:57 AM PST.
Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrrevie wwtp://ormmiidd-e/////e
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. URL https://aclaathhooogy org/N19-140 1423.
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al. Nl-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arxim:2112 027221, 2021.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Iaarco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1ttp:///111221.emmmp/.
URL pttttp::/aalaanhooll/
wrgg/o21...../. em001.mmnl--minn... 98.
Nan Du Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellsters Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient. scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on
54

---

Published in Transactions on Machine Learning Research (07/2024)
Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMLR, 17-23 Jul 2022. URL. https: https://proceeddings wwriggs.mm...... mwt.prres/vv62//ee html.
Yanai Elazar, Akshita Bhagia, lan Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Conference on Leurning Representations, 2024.
István Endrédy and Attila Novák. More Effective Boilerplate Removal the GoldMiner Algorithm. Polibits Research journal on Computer science and computer engineering with applications, 1(48):79-83) 2013. ISSN 1870-9044. URL http://pollbitsa.eellbe
com/2013_44 48.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401. 12926, 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1988-6000 PMLR, 17-23 Jul 2022. URL https: //proceedings. mlr preess//112//thhaaaaaaaaaa htmm.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
Simin Fan and Martin Jaggi. Trreducible curriculum for language model pretraining. arXio preprint ar brXiv:2310.15389 2023.
Simin Fan, Mattoo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arxiv preprinl, 2023.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference Learning Representations, 2024. URL /ttps://openrewwe forr KAK6ngZZ0.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 968 988, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1, /0002222///2211.../ findings-acl.84, URL https: / haw.ttholaattttooo..oooo org/ 2021.f www.fiinnnnaal....
Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document selection for efficient encoder pretraining. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9522-9530 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ /v1 /2/0222eemlpmmma. 647. URL https: //t.a/aaaaannntc/cc/o/m hrg/2022. emnle-manaaeeeeee mww..aain..........
Andró Freitas and Edward Curry. Big data curation. New horizons for a data-driven economy: A roadmap for usage and exploilation of big data in Europe, pp. 87 118, 2016.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In scarch of the next generation of multimodal datascts. In Thirty-seuenth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //tppr////perrvieee.. http:///aaaaeeeeeeeeeeee/rr////a wwtt:foormmi.dddaaaaooo.....
55

---

Published in Transactions on Machine Learning Research (07/2024)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint wrXic:2101.00022 2020.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model ovcroptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'22 JMLR.org, 2023a. URL https: a/d..acm.oo... wttg/oo///iooooooooooo/og /ttg/////....555/111888.888888 www..//66118...3611111 3611884
Lingyu Gao, Aditi Chaudhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, and Michael Ben- dersky. Ambiguity-aware in-context learning with large language models. arXiv preprint arXiv:2309.070 2023b.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning, In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2242-2251. PMLR, 09 15 Jun 2019. URL btttps://proceeding: petteeeediinnmmm....pp... www.mlr.pp.... wwtss//97// ppress/v9/ghhorbaii100 htgggghoorbri..o html.
Vladimiro González- Zelaya, Julián Salas, Dennis Prangle, and Paolo Missier. Preprocessing matters: Au- tomated pipeline selection for fair classification. In International Conference Modeling Decisions for Artificial Intelligence, pp. 202 213. Springer, 2023.
Stefan Grasberger, Julia Stoyanovich, and Sebastian Schelter. Lightweight inspection of data preprocessing in native machine learning pipelines. In Conference on Innovative Data Systems Research (CIDR), 2021.
David Grangier and Dan Tter. The trade-offs of domain adaptation for neural language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3802-3813, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 00.18653/v1/20222 acl- acl- acl-longg266 264. URL https: //aclanthology org//022....../.ooooo. acl -ong.loog..4 264.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://aclaathollgg 1tt.a//aannoooooooooo//111100000 org/L18-1550.
Alex Graves, Mare G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukeuoghu. Automated curriculum learning for neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1311-1320. PMLR, 06- 11 Aug 2017. URL https::/procceeee/eee
mlr prres/v70// prress/vo//rrvvsl77e... graves17a.htm.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafford, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint 0rXiv:2402.008.0 008888 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint ar Xix: 11644, 2023.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coresct selection in deep learning. In International Conference on Dalabase and Expert Systems Applications, pp. 181-195. Springer, 2022.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless datasct distillation via aifficulty-llggned trajectory matching. In The Twelfth International Conference on Learning Representations, 2024.
56

---

Published in Transactions on Machine Learning Research (07/2024)
Shivanshu Gupta, Sameer Singh, and Matt Gardner. Coverage-basee based example selection for in-context learning. arXiv preprinl or 2rx1:23055.149 2023.
Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. Whose language counts high quality? measuring language ideologies in text data selection. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2562 2580, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/vl /te////2222222222 111e222mmllpmaaa mttopmmain.... URL https://aclanthholog. org/2022. emnlp-main. main.
Nika Haghtalab, Michael Jordan, and Eric Zhao. On-demand sampling: Learning optimally from multiple distributions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 406-419. Cur- ran Associates, Inc., 2022. URL 02917 /teTT//ce22444222229999663368889 Pappeeee-sss---p-----e--e-e Conf cronference.ppe. pdf.
http:/ceeednnnnnnenneeeeeeeeeeee neurips. cc/paper wtless/aper///22///5555
neurips.cc/paper
Xudong Han, Timothy Baldwin, and Trevor Cohn. Balancing out bias: Achieving fairness through balanced training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11335- 11350, 2022.
Kazuma Hashimoto, Karthik Raman, and Michael Bendersky. Take one step al time to know incremental utility of demonstration: An analysis reranking for few-shot in-context learning. arv preprint arXiv:2311.09619 2023.
Kenneth Hcafield. KenLM: Faster and smaller language model queries. In Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan (eds.), Proceedings of the Sirth Workshop on Stalistical Machine Translation, pp. 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL wrg/W11-2123
httpp:///aclanthologg htwphhhhooogoooo/WW1..
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring cod- ing challenge competence with apps. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Dalasels and Benchmarks, volume 1. Cur- htw.rceeddiinnnnnnniis...... neurips.cc/paper files/ /22///// rowad..-round..ppp pdf.
ran, 2021. URL
Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint Xrviv:2203. 15556, 2022.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint ar arXi::221..0000 096689, 2022.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: h/tper///pprreevee httpppaarrrrreeeeeeeoro/ra//aaaa net:/forum//dnnZ2eWeerrr
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Understanding predictions with data and data with prodictions. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9525 9587. html. Shunsuke Inenaga Hiromasa Hoshino, Ayumi Shinohara, Masayuki Takeda, Setsuo Arikawa, Giancarlo Mauri, and Giulio Pavesi. On-line construction of compact directed acyelic word graphs. Discrete Applied Mathematics, 146(2):156-17) 179, 2005. ISSN 0166-218X. doi:
PMLR, 17 23 Jul 2022. URL
mlr. wtt.://ww........ee.eeeememmee creeas/v122/lyya/
2004.04..
57

---

Published in Transactions on Machine Learning Research (07/2024)
012.
http:///c.ooo.ooooooo////o/o//oa 12th Annual
Symposium on Combinatorial Pattern Matching.
Dan Iter and David Grangier. On the complementarity of data selection and fine tuning for domain adaptation. arviv preprinl arXix:209000550 2021.
Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, and Chenguang Zhu. In-context demonstration selection with cross entropy difference. In Houda Bouamor, Juan Pino, and Kalika Bali (ods.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1150-1162, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v /tt/222.2.2nnnneeemee https://aclantholl.. .rg//023...000 findings-enti emmlp..mmpp.......m 81.
URL
Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. In Anna Rogers, Jordan Boyd-Grabes Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compulational Linguisties: ACI. 2023, pp. 9036-9061, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 1118655/1122223...0 fin.ing. URL https: //aclanthology. org/2023. iindings-acl... ac1.576.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradoop Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. Xáv preprint arXiv:8311.10000 2023b.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212. 12017, 2022. URL https: //ar:v/a/.vv///// www.:orgggbbaa... htabs//222....
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg, Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5075 5084, Singapore, December 2023. Association for Computational Linguistics doi: 10.18653/v1 122//3/v1/222.eer// www.p-main.n...... URL https:// httt::s//aclaahholooy org/2023. emnlp-main.300
Matthew Jagielski. A nole on interpreting canary exposure. arXie preprinl urXiv:2306.001: 2023.
Tim Janson, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. Perplexed by quality: A perplexity- based method for adult and harmful content detection in multilingual heterogeneous web data. arXiv preprint ar 2211 1 10440, 2022.
Haewon Jeong, Michael D Wu, Nilanjana Dasgupta, Muriel Médard, and Flavio Calmon. Who gets the benefit of the doubt? racial bias in machine learning algorithms applied secondary school math education. Tn Moth Al for Education: Bridging the Gap Between Research and Smart Education, AIED, 2022.
Heinrich Jiang and Ofrr Nachum. Tdentifying and correcting label bias machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 702 712. PMLR, 2020.
C. St. John and N. R. Draper. D-optimality for regression designs: A review. Technometrics, 17(1):15-22) 23, 1975. doi: 10.1080/0044 004401706.1971104488888 00401706. 1975.104
URL https://www. http://wwww.aaa.ooooo/ooeo.ooo onlffonnin... httpm//e..m///////////......
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXie preprint arXiw:1612.0300 2016.
Jean Kaddour. The minipile challenge for data-officient language models. arXiv preprint arXi:2204408844 2023.
Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner. No train no gain: Revisiting efficient training algorithms for transformer-basee language models. In seirth- Conference on Neural Information Processing Systems, 2023. URL htttaa///pppparrree..ee////eeeer www.://ooormm.ddddb........
58

---

Published in Transactions on Machine Learning Research (07/2024)
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1-33) 2012.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697-10707. PMLR, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 0rXiv:2001.08361 2020. URL attps://arxiv. pprggabs//0101... 08361.
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering, text classification, and regression via span extraction. arXiv preprint ar arXiv:1904.09266, 2019.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UnifiedQA: Crossing format. boundaries with a single QA system. In Findings of the As- sociation for Computational Linguisties: EMNLP 2020, 2020. URL https: //apaaaaaaanttggyyoooom.. .wwooog..o.g..oooooooooooo.ooooo org/2022 2020. findings-emnlp. 171.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference Machine Learning, pp. 5464 5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramaniar Ganesh Ramakrishnan, and Rishabh Iyer. Glister: General- ization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, al. Prometheus: Inducing fine-grand fino-grained evaluation capability in language models. arXiv preprinl arxiv:2300.00 08,,49 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations democratizing large language model alignment. arXiv preprint arXiv:2304.070 07327, 2023.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulii--Orhikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda. Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguisties, 10:50-72, 2022. doi: 10. (0.1162/tacl a 00444. URL https://aclantte
org//o/g//22 tac1-1.4.
Sncha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al. Madlad-400: A multilingual and document-level large audited dataset. arXiv preprint arXiv:2309.0,4 04662, 2023.
Po-Nien Kung, Fan Yin, Di WU, Kai-Wei Chang, and Nanyun Peng. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. arXiv preprint xrXto:2311.00258, 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Danicl Fried, Sida Wang, and Tao Yu. Ds-1000: a natural and reliable benchmark for data science code generation. Tn Proceedings of the 40th International Conference on Machine Learning, ICML '2. JMR.Rorrg 2023.
59

---

Published in Transactions on Machine Learning Research (07/2024)
Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human leedback. Xiv preprint arXic:2310. 13595, 2023a.
Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023b. URL https://huggingfaat co/datasets/Huggeeeeees htg.geetts//gggggaaaaaei.ee
stack- srrchange-reeferncces.
Hunter Lang, Aravindan Vijayaraghavan, and David Sontag. Training subset selection for weak supervision. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 16023 16036. Cur- ran Associates, Inc., 2022. URL https://proceeddnng www..is....p....... neurips.cccppperr htc/ppppr_iiee/ppppr///000/ ffless//apprr//22///ii/// 66770aa4eaaa0ffff555111aaa oorpaaaaearraaaaaaaaaao-eoororr
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafcai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepereq, Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The bigscience roots corpus: A 1.6tb composite multilingual dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 31809- 31826. Curran Associates, Inc., 2022. URL htttps://proceeedinggg wwwp:iiiiiinnnnuun..oo.m cctprppppcccpppprr.prppppppppp wwtpp/pppepe.eeeepppeeeeeeeeeeee files/paper/ 2022/5hee69922227777445333382222
BerdddBeehmaraee.eaaa .df..
Dong-Ho Ho Lee, Jay Pujara, Mohit Sewak, Ryen White, and Sujay Jauhar. Making large language models better data creators. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023
Singapore, December Conference on Empirical Methods in Natural Language Processing, pp. emnlp-main.945 URL https: 2023. Association for Computational Linguistics. doi: 10. 20.18833/v1/200 2tt/3////2223..mmmmmaall //aclanthology. 2rt.ttooooggyoooo22223.3.3 emnlp-main. 948.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424 8445, Dublin, Ireland, May 2022a. Association for Computational Linguisties. doi: 10.18653/vv /202l1/2222222/222 lo77lo..777. URL https://aclanttoot .rrg/ 2022. acl long. 577.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pp. 11852-12366 PMLR, 2022b.
Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annuaa Mesting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1401-1422, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL https://aclanthollgg org/2023.aa acl-ooo...looon..... 78.
David D. Lewis and William A. Galc. A soquential algorithm for training text classifiers. In Bruce W. Croft. and C. J. van Rijsbergen (eds.), SIGIR '94, pp. 3 12, London, 1994. Springer London. ISBN 978-1-4441--099-5.
Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sola Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vodant Misra. Solving quantitative reasoning problems with language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL https: h/tpp://///peerviee... http://////ppeeeeeeeeeee////aaaa
60

---

Published in Transactions on Machine Learning Research (07/2024)
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting 1lm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. 2023a.
Peng Li, Xi Rao, Jennifer Blase, Yue Zhang, Xu Chu, and Ce Zhang, Cleanml: A study for evaluating the impact of data cleaning on ml classification tasks. In 2021 IEEE 97th International Conference on Data Engineering (ICDE), pp. 13 24, 2021. doi: 10.110//1CDE513300
Raymond Li, Loubna. Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirien Joao Montciro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Damitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023b. Reproducibility Certification.
ISSN 2835-8856. URL https: httthhht::////eeeeeeeeeee //tp//eeeeeeeee/////////e wet. wwt.:/ee//uuniie httee//ooommiddooo044aa
Xiaonan Li and Xipeng Qiu. Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.10 1335999,.,,399 2023.
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 4644-4668, Toronto, Canada, July 2023c. Association for Computational URL https: hatas//thaaaaaggg...../..m/ wwwpogyyy.og//23o3////// acl-long 256.
Linguistics. doi:
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Tgor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6)24): 1092 1097, December 2022a. ISSN 1095-9203, doi: 10.1126 as112//scenneeeag11555. URL http://ax.doi.ooo 10. ww11/11///////// /ss/sscieccce abq1158.
Zekun Li, Hong Wang, Alon Albalak, Yingrui Yang, Jing Qian, Shiyang Li, and Xifeng Yan. Making something out of nothing: Building robust task-oriented dialogue systems from scratch. In Alera Prize TaskBol Challenge 1 Proceedings, 2022b. URL https: //wp::///www.. www.aaaww....ineeaaaaaaaaaaeeeee /ttp://///.e/pppeeeeeeo/eeee//// www.uuuuw...ttttgo..oooorarrrare
from-scratch
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023. URL
https::/htttpss//uugggaaas http://ffaaa....pppeeeeeeeeee
Hao Liu, Matei Zaharia, and Pieter Abbeel. Exploration with principles for diverse ai supervision. arXiv preprint ar orXiv:2310.0 0.201000000. 08809, 2023a.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: (t.18853/w1/2022..d6 1.100. URL https://aclanthology org/ 2ww1222222eeelo111111 Meelio-1.10.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yojin Choi, and Hannanch Hajishirzi. Infini- gram: Scaling unbounded n-gram language models to trillion tokens. arxiv preprint 171777 2024a.
61

---

Published in Transactions on Machine Learning Research (07/2024)
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024b. URL https:// /ttps:://peenrview..u tetp://.eetttrrrmmr//ooossssrr
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312. 2023b.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Compatational Linguistics, pp. 4487-4496, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1 //aclanthology org/P19-1441.
URL https:
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint 2rxi::2001. 13688, 2023a.
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marmu, Damien Silco, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXiv preprint Xiv:223300. 16787, 2023b.
Shayne Longpre; Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint orXiv:2305.13169 2023c.
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, et al. The responsible foundation model development cheatsheet: A review of tools & resources. arXiv preprint arXiv:2406.1 16746, 2024.
Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu. Self: Language-driven self-ovolution for large langnage model. arXiv preprint arXiv:2310.000 0.00333, 2023a.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2023b.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedingg of the 60th Annual Meeling of the Association for Compulational Linguisties (Volume 1: Long Papers), pp. 1086-8098, Dublin, Treland, May 2022. Association .gg.ggg///oooooooooooo..o.
for Computational Linguistics. doi: 10.18653/ /t1/2/2222--2le acl-long.5556 556. URL 2022. 1022.ac1-an acl-acllll mocl-long.550 556.
Alexandra Luccioni and Joseph Viviano. What's in the box? analysis of undesirable content in the Common Crawl corpus. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 182 189, Online, August. 2021. Association for Computational Linguistics. doi: 10.18653/v1 10l./2/clalssoooootoeoo //aclanthology. acl 24.
URL https:
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Manna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al. Fingpt: Large generative models for a small language. arXiv preprinl ar Xiv:223311 1.03311.0000 056400 2023.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 3384-38 3393. PMT.R, 2018.
62

---

Published in Transactions on Machine Learning Research (07/2024)
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Flazar, Kyle Lo, et al. Paloma: A benchmark for evaluating language model fit. Xix preprint arXiv:2312. 10523, 2023.
Adyasha Maharana, Prateck Yadav, and Mohit Bansal. mattbbb(D)) pruning: Message passing for balancing diversity difficulty in data pruning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviev http://wwww...eeeooror/o/ttaaaaa
Udi Manber and Gene Myers. Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing, 22(5):935-945 948, 1993. doi: 10. 10.1137/0222058. URL
Mare Marone and Benjamin Van Durme. Data portraits: Recording foundation model training data. Advances in Neural Information Processing Systems, 36, 2024.
Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, Huzefa Rangwala, Christos Faloutsos, and George Karypis. Which examples to annotate for in-context learning? towards effective and efficient selection. XrXiv preprint ar Xix:23300220046, 2023.
Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William A Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Will Cukierski, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjao Chen, Mehul Smriti Raje, Max Bartolo, Sabri Eyuboglu, Amirala Ghorbani, Emnett Daniel Goodman, Addison Howard, Oana Inel, Tariq Kane, Christine Kirkpatrick, D. Sculley, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Young, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Y. Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi. Dataperf: Benchmarks for data-centric AI development. In Thirty-sevente Conference on Neural Information Processing Systems Datasets and Benchmarks Track, net mett/foruuniidLLFTTgr
2023. URL
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint Xi:: 08730, 2018.
Merinda McLure, Allison V Level, Catherine I, Cranston, Beth Ochlerts, and Mike Culbertson. Data curation: A study of researcher practices and needs. portal: Libraries and the Academy, 14(2):139 164, 2014.
William Merrill, Noah A. Smith, and Yanai Elazar. Evaluating n-gram novelty of language models using rasty-dawg. arXiv preprint 2rXi:2240611000 2024.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICT: Learning to learn in context. In Marine Carpuat, Marie-Catherins Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, Seattle, United States, July 2022. Association for Computational Tinguistics. doi: 10.18653/v v1 /20222222mmmm wwwp22nnaaclnnn caaacclmmin.2000 main.201. URL https: //alanthooo org/2022 naacl-main. 201.
Shervin Minace, Tomas Mikolov, Narjes Nikzad, Mcysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprinl ar 2rXiv22402.06196 2024.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15630 15649. PMLR, 17 23 Jul 2022. URL
mir. mlr. www.ppreesssv.e.eeee hetpe/vvv2//mmii mindmmmmnnammm222........ html.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Corescts for data-cfficient training of machine learning models. In International Conference on Machine Learning, pp. 6950 6960. PMT.R, 2020a.
63

---

Published in Transactions on Machine Learning Research (07/2024)
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovee. Coresets for robust training of deep neural networks against noisy labels. Advances in Newral Information Processing Systems, 33:11465-11444 2020b.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-tast generalization via natural language crowdsourcing instructions. arXi preprint arXie:2044.000 2021.
Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major Ezinwannc Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kicla. Mcasuring data. arxiv preprinl ar arxiv:222..00 05129, 2022.
Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Jan Hajië, Sandra Carberry, Stephen Clark, and Joakim Nivre (eds.), Proceedings of the ACL 2010 Conference Short Papers, pp. 220 224, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology wtwp//ogyy.r///02000000 2041.10-2041.
Chenghao Mou. Large-scale near-deduplication behind bigcode, May 2023. URL https:/hhuggingfareeco// blog/dedup. Accessed: 2023-12-06.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprinl urXim:2202.08904, 2022.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arxiv preprint orXi::2200003316, 2022a.
Niklas Mucnnighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:221 11.0211,,,78 2022b.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. Xiv preprint ar arXiv:2308.00 07124, 2023a.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrainee language models. In Thirby- seventh Conference on Neural Information Processing Systems, 2023b. URL /ttps://openreview. .ett/ forum?id-j5ButrEJJ33.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amaupreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXim:2402.0990 09906, 2024.
Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012. ISBN 0262018020.
Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112 09.9933, 2021.
Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré. Can foundation models wrangle your data? arXiv preprint arXim:2205.0900 2022.
Tai Nguyon and Eric Wong. In-context example selection with influences. arXiv preprint arXiw:2302.11042 2023.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of CLIP. Tn Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreviers net/forum? Ld=.TCBaVFPSC..
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridgo-regression. International Conference on Learning Representations, 2020.
64

---

Published in Transactions on Machine Learning Research (07/2024)
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lec. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-511 5198, 2021.
Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust language modeling. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.PIJCCNLP)) pp. 4227-4237, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. (..18833/1I/DI-1111 1432. URL https://accaantholooyy org/D19-145 1432.
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test sct contamination for black-box language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //pp/////ep/eeeeeeeem/ee
Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald Lüngen, and Caroline Iliadi (eds.), Proceedings of the Workshop on Challenges in the Management of Large Corpora, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pp. 9 16, Mannheim, 2019. Instiiz- für Deutsche Sprache, doi: 10.14618 (a16118//ds-pub-000 URL http: r/wnbnnerroovinngggggeeeoo de/urn :nn:: de: bsz 90231-
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human leedback. arXiv preprinl Xiv: 02155, 2022. URL https://arxiv.ot http://ariiv.o///a/////////// wtgg/bs/2203333333 021555.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12086 2024.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (ods.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3806-3824, Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/v1/20 f findings- emnlm 248.
URL https: http::////caaantollgy..oooo
Sung Min Park, Kristian Georgiev, Andrew Nlyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model bchavior at scale. ar preprint arXiv:2303. 140003.111188 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advonces in Neural Information Processing Systems, 2021. URL https://openrrvvexx net.riiriiiee/ruu id=Uj7pF-D-Yr DvT.T.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data only. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 79155-79172. Curran Associates, Inc., 2023. URI. https: https://proceeeddngs neureeeedddnnnnennee..eee....eee nwwrips.cc.ppperrr files/ http://00//////aa22ccc00333111e2
_nd Bww.eeennchhhh.eeee pdf.
Guilherme Penedo, Alessandro Cappelli, Thomas Wolf, and Mario Sasko. Datatrove: large scale data processing, 2024. URL https://githu:: http//////////gggggooooo/ooggngn www.///ggingaac//eaataaaaaaaaoae datatrove.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL http:////pppaeeeeeeeeee//rnnrnna
rRh4T.
65

---

Published in Transactions on Machine Learning Research (07/2024)
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. NAACI, 2018. URL https://aclanttol.. org/N18-1202. 1202..
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks, arXiv preprint arXiv:1811.01088 2018.
AJ Piergiovanni, Weicheng Kuo, Wei Li, and Anelia Angelova. Dynamic pretraining of vision-language models. In ICLR 2023 Workshop Multimodal Representation Learning: Perks and Pitfalls, 2023. URL
https://openrevieu http://pppnrrvi..neeeeeeeeeeeeee www/ffouumnciieoommmeeeeeooe foo.urrmddmmm(WNNNN....
Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf, Stella Biderman, Martin Potthast, and Jimmy Lin. GAIA search: Hugging face and pyserini interoperability for NLP training data exploration. In Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 588 598, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: /10///33/1/2233acl//ee
URL
org/2023. hel-demo.3
Geoff Pleiss, Tianyi Zhang, Ethan Flenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044 17056, 2020.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating Iims with fewer examples. arXiv preprint 149999,,..49,, 2024.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order corescts for data- efficient machine learning. In International Conference on Machine Learning, pp. 17848 17869. PMLR, 2022.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediato-tane task transfer learning with pretrained language models: When and why does work? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5231 5247, Online, July 2020. Association for Computational Linguisties. doi: 10.18653/v //20///11/20.0 acl-main.467, URL attps://aclanttaee 0rgg////0.ooogy.../0000....00 acl-main m67
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training, 2018. URL https://app semanticscollar
Segmaaateccccolo..ooooo///oraaaa orgg/orppsID:49911900
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are sem.amiccccnnnnnnnnnrr wwt.oogg///orr.... ortpaaroo//ooopop1110005533 unsupervised multitask learners, 2019a. URL https://appp htt.np/ps///pp...maaaaaaaooooola
unsupervised
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019b. URL https:///4mucfppesy/a cloudfront. bet/better-languaee
saagggaaaeemddelleaeeee models are
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. ar preprint 2r 12446 2021.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint 20x::23055.82900 2023.
66

---

Published in Transactions on Machine Learning Research (07/2024)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435. URL https://jmlr. httpmml.oogggppppeeeeeeee otggppppers//voluue2222000777770 20-074, 0d.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https:
co http:///aaattt///gggggaa//aaa/aa
2023.
Nils Reimers and Iryna Gurevych. Sentence- BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empórical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/1vl / URL https
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM Comput. Sure., 54(9), oct 2021. TSSN 0360-0300. doi: 10.1145/3472291. URL https://doi org/10. 1145/3472299
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Marine Carpuat, Maric-Catherins Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (cds.), Proceedings of the 2022 Conference of the North American Chapler of the Associalion for Compulational Lánguislics: Human Language Technologies, pp. 2655 2671, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ v1 /2022.22222 nww.lnaaallllceee maacl-main.11 191. URL /ettps:/aclanttiee 191.
org/2022. maacl-main.
Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. Language modelling with pixels. In The Eleventh International Conference on Learning Representations, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openreviee pot/forrunidd=yy(u..aa
Teresa Salazar, Miriam Scoane Santos, Helder Araújo, and Pedro Henriques Abren. FAWOS: Pwire Oversampling Algorithm Based on Distributions of Sensitive Attributes. TEEE Access, 9:81370-811 2021. ISSN 2169-3536. doi: 10.
Conference Name: IEEE Access.
Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow teaming: Open-endod generation of diverse adversarial prompts. arXiv preprint orXiv:2402.1683 16822, 2024.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Represenlations, 2022. URL https: //oeereviie http://ppaaaaaeeeeeeeee///aaaaap nett//orrummttt-9/tt00000
Joseph R. Saveri, Cadio Zirpoli, Christopher K.L. Young. and Kathleen J. McMahon. Paul tremblay, mona awad vs. openai, inc., et al., 2023. URL sttogs///torrgg...orrrtiiseee cww/recap/goo uscourts cand. mand.4148222 414822/gov 2gov. uscourts. cand. 414822. 1. 0 1_111.1p0000 pdf. Case 3:23-ev-03223-AMMG Document:
courtlistenerr
Filed 2//2/2233 UNITED STATES DISTRICT COURT, NORTHERN DISTRICT OF CALIFORNIA, SAN FRANCISCO DIVISION.
Teven Le Scao. Scaling multilingual language models under constroined data. PhD thesis, Université de Lorraine, 2023.
67

---

Published in Transactions on Machine Learning Research (07/2024)
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Flsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train if you have one million gpu hours? arXiv preprint (xXix:2210015424, 2022.
Scbastian Schelter, Yuxuan He, Jatin Khilnani, and Julia Stoyanovich. Fairprep: Promoting data a first-class citizen in studies on lairness-enhancing interventions. arXiv preprint arXiv:1911 125877, 2019.
Timo Schick, Jane Dwivedi-Yin Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreviier
Andrea Schioppa, Xavier Garcia, and Orhan Firat. Cross-lingual supervision improves large language models pre-training, orXiv preprinl ar Xiv:2300 1 11778, 2023.
Jacob Schreiber, Jeffrey Bilmes, and William Stafford Noble. Apricot: Submodular selection for data summarization in python. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R. Kundurthy, Katherine Crowson, Ludwig Schmidt,, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sizth Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreviet
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A corc-set approach. In International Conference on Learning Represenlalions, 2018. URL https://openreviai dowp///eeeeiiiieeerreorrrrrrr net/forum?id= H1aIuk-RW.
Abhinav Sethy, Panayiotis G. Georgiou, and Shrikanth Narayanan. Text data acquisition for domain-specific language models. In Dan Jurafsky and Eric Gaussier (eds.), Proceedings of the 2006 Conference on Empirical Melhods in Natural Language Processing, pp. 382 389, Sydney, Australia, July 2006. Association for Computational Linguistics. URL
org/w06-1100 1645.
Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-dc: Understanding data combinations for Ilm training, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL /ttps://openreviev net/ met/forummideee formm//ii-zNgr300OUN.
Zhouxing Shi, Nicholas Carlini, Ananth Balashankar, Ludwig Schmidt, Cho-Jui Hsieh, Alex Beutel, and Yao Qin. Effoctive robustness against natural distribution shifts for models with different training data. In Thirby-seventh Conference on Newral Information Processing Systems, 2023. URL https: //openreview //openreview.nee/foorooeee www.///oorr.dd-PAEEEEEE..
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya datasct: An open-accoss collection for multilingual instruction luning. arXie preprint arXiv:2402.0660 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
Luca Soldaini, Rodncy Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. ar Xie preprint or 00000000 2024.
68

---

Published in Transactions on Machine Learning Research (07/2024)
Hwanjun Song, Minseok Kim, Sundong Kim, and Jac-Gil Lee. Carpe diem, seize the samples uncertain "at the moment' for adaptive batch selection. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, pp. 1385 1394, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450368599. doi: 0114455/33453113411 10. 10.1111//334511111 001.40551.3111889 3411898.
URL https://doi.oo
Ryosuke Sonoda. Fair oversampling technique using heterogeneous clusters. Information Sciences, 640:1 190:9, September 2023. ISSN 0020-0255. doi: 10.1016/j.ins 2023. URL https://ww..sciencddiii http///iaaee/rrtttaa////////o/o/
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openrevieer ntt/ffruum (oooounniiddUSSSPPPPy)
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Auh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandor, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loc, Barret Zoph, Bartlomicj Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch,, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Silco, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coclho Mollo, Diyi Yang. Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez- Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista. Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch- Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Tan Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B3 Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Mouis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros- Colónn Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Kamirez-Quintane Marie Tolkiehn,
Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochane
69

---

Published in Transactions on Machine Learning Research (07/2024)
Mario Giulianclli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng. Nathan Andrew Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnos, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo- Hwan Lee, Spencer Torene, Sriharsha. Hatwar, Stanislas Dehaene, Stefan Divie, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofci Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prablu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilitics of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httpps//opperrviiet http://ppeeeeeeeee.eeeeforrrraaa
Hongjin SU, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrrview net/forummid=q//hlly77
Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. Detecting personal information in training corpora: an analysis. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 208-220, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v 110/5/3//1/222.trusss// /2203...rusttnpp..... URL https://aclanthhe 18.
org/2023, trustalp-1.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Datasct cartography: Mapping and diagnosing datascts with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275-9293, Online, November URL https:
2020. Association for Computational Linguistics. doi: 10.18653/v wtt0///ecm0m//////m.. haaclantholog..orr///20 argg20000.mmmp-main. 746.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xucchen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instructioon-oolloiing Ilama model. https://github.ce tatau-lab/stanford _alpaca, 2023.
70

---

Published in Transactions on Machine Learning Research (07/2024)
Alaa Tharwat and Wolfram Schenck. A survey active learning: State-of-the--ar, practical challenges and research directions. Mathemalics, 14), 2023. ISSN 2227-7390. doi: 10:33900maat111040890 URL https://ww.mmpp.comm/// wttpp..coo/22777733//114448333
Saravanan Thirumuruganathan, Nan Tang, Mourad Ouzzani, and AnHai Doan. Data curation with deep learning. In EDBT, pp. 277 286, 2020.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving Ilm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36, 2024.
Mads Toftrup, Søren Asger Sørensen, Manuel R. Ciosici, and Tra Assent. A reproduction of apple's bi- directional LSTM models for language identification in short strings. In lonut-Teodor Sorodoe, Madhumita Sushil, Ece Takmaz, and Eneko Agiree (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp- 36 36-42, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v (0553/1//// aclanthology org/2021. eacl-srw.6.
URL https://
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint urxw:2232. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXic preprint or Xiw:2307.00288, 2023b.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.160) 2023.
Ahmet Ustün, Viraat Aryabumi, Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ovi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint 242::242.07827, 2024.
Daan van Esch, Tamar Lucassen, Sebastian Ruder, Isaac Caswell, and Clara Rivera. Writing system and speaker metadata for 2,800+ language varieties. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Enaluation Conference, pp. 5035-5044 Marseille, France, June 2022. European Language Resources Association. URL https: https:///acaantholgg.
org/2022.00 1rec-1.538.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackborNLPP: Analyzing and Interpreting Neural Networks for NLP, pp. 353 355, Brussels, Belgium, November 2018a. Association for Computational Linguistics. doi: 10. 10.18653/vvl //1////1////// v1/W18-5446. URL https://aclanttholo. www//o/-g//o/oo.oooooooo B4446.
Jigang Wang, Predrag Neskovic, and Leon N. Cooper. Training data selection for support vector machines. In Lipo Wang, Ke Chen, and Yew Soon Ong (eds.), Advances in Natural Computation, pp. 554-564, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31853-8.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12196-12205, 2022a.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datascts are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IFFF/CVF international conference on computer vision, pp. 5310-5311 2019.
71

---

Published in Transactions on Machine Learning Research (07/2024)
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasumuru, Jane Dwivedi- Yuivedi-YY, Olga Golovneva, Luke Zettlemoyer, Maryam Pazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXic:2308.00 200::2308004992, 2023a.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexci A Efros. Dataset distillation. arxiv preprint arXiv:1811.10959, 2018b.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good retonstrations for in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL (Bc//forumm1dBBBO22ZZE3E
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Necraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super- NaturalInstructions:: Generalization via declarative instructions 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedingg of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. Association for Computational Linguistics. doi: (0.18853/v1/ 1122022..22222 wttp/22.eem/lppppp.p www.mmmain.n3eee main.340. URL https:// momnp-mmain...
Abu Dhabi, United Arab Emirates, December 2022b.
aclanthology.oo hwtpioooogyoorg///010
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Han- nanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds..) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484 13508, Toronto, 754. URL
Canada, July 2023c. Association for Computational Linguistics. doi: 10. attps://aclanthol/ wwwpgyg....g......// acl-long. 754.
Yizhong Wang, Hamish Tvison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36, 2024.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Trans- actions of the Association for Computational Linguistics, 2019.
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challengo: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pp. 34, Singapore, December 2023. Association for Computational Linguistics. doi: 2036553/v1/223..om/ coa.:eccoooobbbbiiiee b.bylm.11 URL https: //aclanthology. org/2023. conll- babylm....
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Gun, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ICLR 2022, 2021. URL
https://openreview htt.:ppwwwrreeeeeeeeeeoooaaaaraa net/fforummrdrrg////roo.z..
Johannes Welbl, Amelia Glaeso, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeee Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Saari-Francciee Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447-2469, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 findings-emn) org/2021 waw.inngg-eenll...m 210.
210. URL https: httpss//aclanntoolog 20tpeaatthoooooyor//220010111111
72

---

Published in Transactions on Machine Learning Research (07/2024)
Guillaume Wenzek, Marie-Ame Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CONet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, Marscille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444. URL https://acee http:///aaaaaatttog..oooooo org/ wwwgg/22/....0// 1rec-1.499.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09730 09739, 2024.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: org/N18-1100
URL
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellic Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211. 1.f:00021.05100, 2022.
Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. Scattershot: Interactive in-context example curation for text transformation. In Proceedings of the 28th International Conference on Intelligent User Interaace,, IUI '23 pp. 353 367, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701061. doi: (..115/3388641 61.35580599 URL https://don http::///dooooo/11///// 1145/
3581641.35888 3584059.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, 2024a. CRL /ttps://openreviie wwt.reeweewwweeeeeeeeeeeeeeeeeee www.eeewwweieeeeeeeeeeeeeeeeeeee wwwmmidd00100daee000
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arxiv:2402. 04333, 2024b.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-cfficient deep learning. In The Eleventh International Conference on Learning Representations, 2023. URL /ttps://openzee
net ftee/fooommmttt75EE0000009
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding. arXie preprint orXiv:2309.099 07597, 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Shirth-seventh Conference on Neural Information Processing Systems, 2023a. URL http::///pppprrrreeeeeeee//oo/ss
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In shirby-seventh Conference on Neural Information Processing Systems, 2023b.
URL http::///ppppreeeeeeeeeeet//a/ae
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying language models risks marginalizing minority voices. arXin preprint orXin:2104.066 2021.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Curriculum learning for natural language understanding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (ods.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6095-6104, Online, July 2020. Association for Computational Linguisties. doi: 10.18653/vi 1.1663/v1//2020) 1/22000. acl-main.541 URL https: / http://aaaaaatoogy...ooo .wwoggg...rrgg/ee otg/ http://222....mmiaaa.... acl-main.54.. 542.
73

---

Published in Transactions on Machine Learning Research (07/2024)
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. or Xiv preprinl 1wXi:23304.1224, 2023.
Shangqing Xu and Chao Zhang, Misconfidence--aase demonstration selection for Ilm in-context learning. arXiv preprint orXiv:241106633 2024.
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models. ar preprint or Nrxi:220 11436, 2024a.
Xin Xu, Yue Liu, Panupong Pasupat, Mehran Kazemi, et al. In-context learning with retrieved demonstrations for language models: A survey, arXiv preprint orXic:2401.1122, 2024b.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, pp. 483 498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/vl 101///11///// /v122211111222/2 maacl- main.41. URL https://aclanthol..g
2222/222..22 naacl-main, 41.
Shen Yan, Hsien-te Kao, and Emilio Ferrara. Fair Class Balancing: Enhancing Model Fairness without Observing Sensitive Attributes. In Proceedings of the 29th ACM International Conference Information & Knowledge Management, CIKM '20, pp. 1715-1724, New York, NY, USA, October 2020. Association for Computing Machinery. ISBN 978- 1--503-6859-9 doi:
URL https: /tc.a::/ddl.amm.. acm.
org/doi 10. 1145//344/33111111 /t//////110..145333333311 34111998....111.0.....11 3411980.
Forest Yang, Mouhamadou Cisse, and Oluwasanmi Koyejo. Fairnoss with overlapping groups; a probabilistic perspective. Advances in Neurul Informalion Processing Systems, 33, 2020a.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Trunsparency, pp. 547-558, Barcelona Spain, Jamuary 2020b. ACM. ISBN 978-1-4503-6936- doi: 10.145//3511095.337770 URL https://ddl acm. wwwpooo/////ooooooooooooooooooot 1145// 3351095 3375709.
Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. Image data augmentation for deep learning: A survey. arXiv preprint arXiv:2204.088 08610, 2022.
Jiacheng Yo, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exomplars for in-context learning. Xiv preprint arXiv:2302.056i 2023.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance, 2024.
Zheng-Xin Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. Bloom+ 1: Adding language support to bloom for zero-shot prompting. arXiv preprint arXiv:2912.09593 2022.
Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. In The Twelfth International Conference Learning Representations, 2024a. URL
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10090, 2024b.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship learning mathematical reasoning with large language models. arXiv preprint or Xin:22008.00 01825, 2023.
74

---

Published in Transactions on Machine Learning Research (07/2024)
Xueying Zhan, Qingzhong Wang, Kuan hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B. Chan. A comparative survey of deep active learning, 2022.
Chen Zhang, Mingxu Tao, Quzhe Huang, Jiuheng Lin, Zhibin Chen, and Yansong Feng. Mc 2: A multilingual corpus of minority languages in china. arXiv preprint arXiv:2311.08.0 088448, 2023a.
Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling- Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Tdeal: Influence-driven selective annotations empower in-context. learners in large language models. arXiv preprint ar Xrv::23310 10873, 2023b.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint 2308 10792, 2023c.
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Yoar Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedinggs of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9134 9148, Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguistics. doi: 10.18653/v1 /ttp://///2222//mmmmmmo/ www.p-mminnnn....m.emmmmmmm //aclanthology. or2/2022. emnlp-main.62. 622.
URL https:
Zhisong Zhang, Emma Strubell, and Eduard Hovy. A survey of active learning for natural language processing.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 6166 6190, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10.18653 / htt.2222..mmllmmaii... www.p-mmannnnnn URL org/2022. emnlp-main 414.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 2514-6523 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2020.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023a.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ar preprint arXiv:2303.1822 01te22333.1222 2023b.
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Salch, and Peter J. Liu. Calibrating sequence likelihood improves conditional language generation. In The eleventh international conference learning representations, 2022.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representations, 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In sevrth Conference on Newral Information Processing Systems, 2023. URL htttss:/opperrviiee net//oormmtdd=<BM0Knn
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems, 35:9813 9827, 2022.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: 7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023.
75

---

Published in Transactions on Machine Learning Research (07/2024)
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjac Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advonces in Neural Information Processing Systems, 36, 2024.
Terry Yue Zhuo, Zhou Yang, Zhensu Sun, Yufei Wang, Li Li, Xiaoning Du, Zhenchang Xing, and David Lo. Data augmentation approaches for source code modcls: A survey. arXiv preprint 1rXiv:2305.19915, 2023.
Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter-cfficien instruction tuning code large language models. arxiv preprinl 4:10:2201.007788, 2024.
Daniel M Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arviv preprint arxiw11000 085993
2019.
76

---

Published in Transactions on Machine Learning Research (07 (27/2
A Heuristic Filtering Details
RefinedWeb (Penedo et al., 2023) hcuristics. Removes lines based on the following rules:
If the line is "mainly" composed of uppercase characters
the line is composed of only numerical characters
If the line is counter (e.g., "3 likes")
If the line contains single word
Note that some of these rules are underspecified, and the paper does not contain a threshold used to determine that a line "mainly" composed of uppercase characters. Additionally, they do not include information on how they determine that line contains a counter.
Furthermore, Penedo el al. (2023) also edit lines if they have fewer than 11 words and match of the following patterns:
At the beginning of the line (e.g., "sign-in")
At the end of the line (e.g, "Read more...)
Anywhere in the line (e.g., "items in cart")
Note again that these rules are also underspecified the exact patterns that were matched are not included in the manuscript.
MassiveText (Rae et al., 2021) heuristics. conditions:
Remove documents that do not meet all of the following
between 50 and 100,000 words
mean word length is between 3 and 10 characters
symbol-to-word ratio is less than 0.1 (for the symbols "#" and "...")
fewer than 90% of lines start with a bullet point
fewer than 30% of lines end with "..."
greater than 80% of words contain alphabetic character
contain at least two of the following "stop words": the, be, to, of, and, that, have, with
C4 (Raffel et al., 2020) heuristics Remove lines that:
do not end in terminal punctuation, including periods, exclamation marks, question marks, and end quotation marks.
have fewer than words
have the word "Javascript" because many of the scraped web pages had warning statements about Javascript being enabled
Remove documents that:
have fewer than sentences
77

---

Published in Transactions on Machine Learning Research (07 (07/2024)
have word from the "List of Dirty, Naughty, Obscene, or Otherwise Bad Words
have the placcholder phrase "lorem ipsum"
contain a curly bracket, "(" because it appears in many programming languages and they desired for C4 to contain only natural language
have any of the phrases "terms of use", "privacy policy", "cookic policy", "uses cookies", "use of cookies", or "use cookies" because they found many pages to have boilerplate policy notices
mC4 Xue et al. (2021) Heuristics mC4 is multilingual dataset, strongly motivated by the methods of the C4 dataset. They compare that while C4 can remove lines which do not end in an English terminal punctuation mark, that is not possible for a multilingual corpus. Instead, they apply a line length filter, which removes pages that:
contain fewer than three lines of text with 200 or more characters
have word from the "List of Dirty, Naughty, Obscene, Otherwise Bad Words
have a primary language confidence (determined by cld3 below 70%
MADLAD-400 (Kudugunta et al., 2023) Heuristics MADLAB-400 is a multilingual corpus, so some of the heuristics are unique to a multilingual setting, but many can shared with English-only corpora.
They remove pages that:
Contain fewer than lines with 200 or more characters (as in Xue et al. (2021))
have fewer than 5 sentences
Remove lines that:
contain the word "Javascript"
contain the placeholder text "lorem ipsum"
contain curly brackets ("(" and "]")
Additionally, Kudugunta et al. (2023) design set of characteristics that make document "questionable" If more than 20% of the sentences in document have one of the following qoesle characteristios, then the whole document is removed. The questionable characteristies are:
The predicted language ID of the sentence does not match the document-level language ID)
over 50% tokens in the sentence begin with a capital letter (only if the line has at least 12 tokens)
the sentence has less than 20 or more than 500 characters
Over 20% of the characters in the sentence match [0-90+/((>]
the sentence matches "cursed regex". A cursed regex a set of substrings and regexes that they found accounted for signifieant quantity of questionable content (fully specified in Appendix A.2 of Kudugunta et al. (2023)).
GPT-3 (Brown et al., 2020a) Heuristics None listed.
36 Shttes///ttabboom///DXOOU///// Daw.eeeayyyy-----O---tataa- -ateea---nnn---twiieeeeee Words 37 httpe///hhhbb.cmm//XXooO//////e Noueehty--sseeeeaandd
Ohseene-and- -oteed---eiiseee... Bad- Words
38 Shttss//gghhabbmm//gggg//aaaa
78

---

Published in Transactions on Machine Learning Research (07 (07/2024)
ROOTS Corpus Laurençon et al. (2022) Heuristics Laurençon et al. (2022) collect and select data
from multiple sources. First they collect code dataset from BigQuery and use the following heuristics:
remove files with fewer than 100 or greater than 200,000 characters
remove files with less than 15% or greater than 65% alphabetic characters
remove files with line lengths less than 20 greater than 1000, token length standard deviation of 3 or less
Laurençon et (2022) mention their use 8 different filters, and describe the details of each filter in a separate document but do not specify what thresholds they use for their selection mechanisms"! They use filters for:
Number of words
Character repetition ratio
Word repetition ratio
Special characters ratio
Closed class word ratio
Flagged word ratio
Language identification prediction score
Perplexily
GLaM (Du et al., 2022) Heuristics None listed.
Llama (Touvron et al., 2023a) Heuristics To train Llama, Touvron et al. (2023a) create corpus composed of domains: CommonCrawe C4, Github, Wikipedia, Books, ArXiv, and StackExchange. The processing of cach dataset done separately as follows (where details are provided):
English CommonCrawl- They use the CCNet pipcline (Wenzek et al., 2020) which does not perform any heuristic filtering.
GitHub "low quality" files were removed based on heuristics including the line length and proportion of alphanumeric characters. Additionally, boilerplate, such as headers, were removed using regular expressions.
Wikipedia hyporlinks, comments, and other formatting boilerplate was removed
ArXiv They follow the method of Lewkowyez et al. (2022) and remove everything prior to the first section well as the bibliography. They also remove all comments, inline-expanded definitions, and user-written macros.
StackExchange They remove all HTML tags, and sort the answers by score so that the highest scoring answer is closest to the original question.
hhttps://cloud.googee
determined by mapping values in parametera_ filtering_defaull default from this file: httper//gitluu..om// http:///gillbbbbooooiiieeeeeeeee watpssccncooorrshhopo/oooeo preparation htwbbbobbaaiiaaa pepprooessingg/trainnn///// oscar cleaning and filtering/ paralngg//ammmters.. filtering.py, but this has been confirmed.
41It possible that the sclection
mechanisms
bc
79

---

Published in Transactions on Machine Learning Research (07/2024)
Pile (Gao et al., 2020) Heuristics The Pile dataset consists of 22 separate sub-domains, each domain requiring it's own preprocessing. While the main goal of the Pile project was to gather data from wide variety of domains, they include some heuristic filters when collecting data. Code to replicate their dataset processing can be found at https:/ http:////Saaab.ooo///////////ee /Eteethrr////tteeppll....
They use the following filters for each domain (where details are provided):
Pile-CC- jusTexx (Endrédy & Novák, 2013) to extract the text from Common Crawl. They decide to use jusText based on visual inspection of the outputs compared with using Trafilatura, Newspaper, Goose3, and DragNet. No details on the heuristics are specified.
OpenWebText2 Newspaper was used to extract text instead of jusText to maintain consistency with
ArXiv - They use the TeX source file and pandoc to convert the files to Markdown, discarding papers that had errors during the conversion process. They remove any line that begins with "; ;", which indicates an html class in Markdown.
FreeLaw They use BeautifulSoup to extract raw text from CourtListener
NIH Award applications were removed if their abstract was too short, or missing. They also removed some administrative boilerplate (not specified).
HTLM (Aghajanyan et al., 2022) Heuristics HTLM utilizes unique heuristic because their data maintains some of the original HTML code from web documents in a simplified form called MiLL (MTHML). MHTML removes all sub-trees the HTML DOM that does not contain text of specific character size (128 for text elements, 64 for httpt/aabls//ppaaaa//a/ and removes all headers, footers, copyrights, forms and iFrames. Then, they combine consocutive <div> elements, and remove any attributes which are not class id attributes.
This preprocessing leads to data which has mix of HTML and text. Aghajanyan et al. (2022) only use single hcuristic filter, they remove any MHTML document that has a text-to-HTMe ratio of 0.46 or less.
HumanEval (Chen et al., 2021) Heuristics HumanEval is corpus of code repositories and has different considerations when filtering data. They remove files that:
have average line length greater than 100 characters
have a maximum line length greater than 1000 characters
were likely auto-generated (unspecified)
contain small percentage of alphamumeric characters (unspecified)
AlphaCode (Li et al., 2022a) Heuristics AlphaCode also creates dataset code repositories for training. They remove files that:
are larger than 1MB
have maximum line length greater than 1000 characters
were likely auto-generated (unspecified)
In addition to the training dataset, Li et al. (2022a) also create the CodeContests dataset to be used as an evaluation for competitive programming problems. These files contain problems, user solutions, and test. cases. For this set, they remove:
80

---

Published in Transactions on Machine Learning Research (07/2024)
submissions that do not pass any tests
tests where less than 10% of submissions produce nou-empty outputs
submissions that pass less than 10% of remaining tests
Minerva (Lewkowycz et al., 2022) Heuristics Lewkowyez et al. (2022) create dataset of 38.5B tokens combining webpages filtered for mathematics, arXiv papers, and general purpose natural language data. To create dataset of webpages with mathematics, they filter for pages that include the string "<math" or MaathJax-Element- Next, they extract the mathematical content from within the following html tags:
<scrppt type='nahhllate">
<scrppt "yp=""math/ascttmathh"""
<annotation eucoding"appllccttioo//rtr
81


Reviews:
[{"review_id": "8SO26FcXoU", "reviewer": "Reviewer_ccCq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis paper systematically and thoroughly reviews and summarizes the current technologies and challenges about data selections and presents a taxonomy to understand data selections. This paper can systematically accelerate the development of data processing and facilitate the community to better use data.\n\n**Weaknesses**\n\nAs a survey paper, I think this paper does not have any weaknesses. It will be better if authors can also present some suggestions about the integral process (which datasets, selection methods and so on) of data selection for training large language models, and give some statistical about the current dataset situations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically studies the data selection mechanism in language models across different stages, from pre-training to fine-tuning. It gives a thorough review to understand the techniques of data selection, including data quality, selection mechanism and so on. Generally, it summarizes a lot of findings and methodologies about data selection and facilitates the community to better choose data selection method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think this paper does not have any major request changes. It will be better to add some suggestions or examples for the community to create datasets for training large language models, and also give some details about current datasets for selection.", "broader_impact_concerns": "This paper does not have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RK9tq4E96j", "reviewer": "Reviewer_TCqY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* impressive breadth of literature review\n* great summaries of cited content; I felt the level of detail was just right for this purpose\n* the structure of the manuscript was clear and easy to follow\n* the illustrations were great and helped to guide the reader\n\n\n## Weaknesses\n* the focus of the survey was very fine grained; except for the high level structure given by the model development cycle of LLMs, it felt like there is not much structure between the leaves of the taxonomy of methods studied and the high level structure of the model development. Maybe it would help readers to try and have some more mid-level structure? I'm not sure whether that's possible and how the best solution would look like - but maybe there are commonalities between methods that could help to reduce the complexity? Or there could be, for each section/model development step some sort of ranking, of which methods are most popular or successful in terms of quantitative metrics? \n* related to the above point, I think the authors did a great job at objectively summarizing the field with a neutral tone and view. But I guess for many readers it would be helpful to have bit of guidance or insight into what works and what did not show concistent empirical improvements? In some paragraphs the authors mention that methods did not really get traction or are not really used anymore, but maybe an overview of those methods that did survive would be helpful? \n* I believe active learning is probably relevant as well and has been applied to NLP problems, too [Zhang et al., \"A Survey of Active Learning for Natural Language Processing\", EMNLP, 2022](https://aclanthology.org/2022.emnlp-main.414.pdf) - some references probably mention literature in that field, so sorry if I just missed that,  but it seems that could maybe fit in section 9.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an in-depth review on the current state of data selection in natural language processing (NLP). I very much enjoyed reading this outstandingly up-to-date and comprehensive overview of the state of the art in data centric language modeling. Given the importance of this topic I'm convinced that the value of contributions that aim at comprehensive surveys in this field are hard to underestimate. This holds true for all subcommunities of the ML field, but in NLP the topic deserves special attention: due to the vast amount of data and the fact that we currently do not really understand the complexity of commonly used training data sets (in contrast to, say, computer vision or tabular data sets), the research summarized in this manuscript is more heterogeneous and difficult to structure than data cleaning/selection methods in other fields. \n\nAs demonstrated by the number of references cited, structuing the complexity of this field is a very valuable contribution. \n\nWhen reading the manuscript I just had minor remarks that I'll list below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In order to guide readers a bit better through the amount of information provided in the survey it would be great if the authors could add a bit of mid-level structure (taxonomy or illustrations/rankings of methods) as mentioned above.", "broader_impact_concerns": "The authors do discuss societial impact of data set quality on LLMs, including concerns such as fairness and how it could be enforced via data set curation, as well as concerns on data protection.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PVWwFtDD0M", "reviewer": "Reviewer_zFED", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The survey covers a variety of recent approaches in data selection for LMs, from pretraining to task-specific fine-tuning.\n- In terms of the data pipeline, the survey presents a clear overview of existing work.\n- The survey provides further discussion and future research direction with challenges and opportunities, facilitating the application of data selection for LMs in practice.\n\n*Weaknesses*\n- **The motivation & problem scope** is a bit confusing. The survey starts with aiming to narrow the knowledge gap from organizations with resources to process large-scale data, and says “In this work, we focus on data selection for pretraining.” (in Section 2.3.4). However, the survey covers existing works across the data pipeline for LMs beyond pretraining. \n- **The link with the foundations of machine learning** is missing. The mathematical foundation for explaining why data selection theoretically works better for LMs would be helpful. The current presentation is too add-hoc in some degree. Although in Section 2, the authors formulate some conceptual framework for data selection, it is not that helpful to understand the foundation of the existing work.\n- The authors lack **the discussion with related surveys** in data selection for LMs or other deep models. Although Section 9 explains some data-focused related topics, it has too many technical details. I think the authors should provide some high-level explanation of how this survey differs from other related surveys in the data selection area, which can emphasize the contribution of this survey.\n- There is no mention of **popular datasets** used in practice. Providing the summary of publicly available datasets used in practice for validating data selection of LMs would be very useful for the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper concludes a hot area in machine learning community called data selection for language models (LMs). The survey covers many novel approaches. In particular, the paper categorizes existing data selection approaches according to the data processing pipeline, and provides comprehensive reviews for each data processing stage. The survey presents further discussions and future directions with challenges and opportunities in data selection for LMs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Major*\n- Provide mathematical foundations\n- Provide a discussion with the related surveys in data selection for LMs or other deep models\n- Provide a summary of popular datasets used in data selection for LMs \n(so far, refer to Weaknesses for detailed requests)\n\n- Provide a big table to summarize the important works for each data selection pipeline. Although the authors cover a wide range of existing works, the current version is too wordy and hard-to-follow. A big table showing the data pipeline, technical category, and properties of popular works would be very helpful to readers in organizing the existing works. \n- It would be nice to have a discussion that tells what kind of recent work shows state-of-the-art performance in each pipeline. Providing empirical tips regarding the performance would be very useful for the readers to facilitate their research.\n\n*Minor*\n- In Figure 4 (and other sections), I think “Heuristics” can be included in other categories such as language filtering or deduplication. Making the “Heuristics” as a separate category seems less convincing to me.\n- (typo) In Definition 2.4, \"all data possible data points\" should be \"all possible data points\"", "broader_impact_concerns": "No ethical concerns found", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper offers a valuable contribution by providing a comprehensive review of LLM pretraining data selection methods. Reviewers have praised the systematic and thorough nature of the survey. The authors have successfully addressed most concerns raised during the revision process. The reviewers also highlight the importance of this work in structuring the complexity of LLM pretraining data research.
Decision: Accept (poster)