Data Selection for Language Models via Importance Resampling
Authors: Sang Michael Xie
Shibani Santurkar
Tengyu Ma
Percy Liang
Venue: NeurIPS 2023 poster (2023)
Keywords: Language models
pretraining
data selection
fine-tuning
TL;DR: We present an importance resampling framework for selecting language model pretraining data that improves performance on GLUE and domain-specific tasks.

Abstract: Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT-3) and domain-specific (e.g., Codex) language models (LMs). We formalize this problem as selecting a subset of a large raw unlabeled dataset to match a desired target distribution given unlabeled target samples. Due to the scale and dimensionality of the raw text data, existing methods use simple heuristics or require human experts to manually curate data. Instead, we extend the classic importance resampling approach used in low-dimensions for LM data selection. We propose Data Selection with Importance Resampling (DSIR), an efficient and scalable framework that estimates importance weights in a reduced feature space for tractability and selects data with importance resampling according to these weights. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents from the full Pile dataset in 4.5 hours. To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space. Across 8 data selection methods (including expert selection), KL reduction on hashed n-gram features highly correlates with average downstream accuracy (r=0.82). When selecting data for continued pretraining on a specific domain, DSIR performs comparably to expert curation across 8 target distributions. When pretraining general-domain models (target is Wikipedia and books), DSIR improves over random selection and heuristic filtering baselines by 2--2.5% on the GLUE benchmark.

Data Selection for Language Models via Importance Resampling
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang
Department of Computer Science Stanford University
{xie, shibani, tengyuma, Satannn@osssstaffroroeoeo
Abstract
Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT- 3) and Nomain-speciff (e.g., Codex) language models (LMs). We formalize this problem selecting subset of large raw unlabeled dataset to match a desired tar- get distribution given unlabeled target samples. Due to the scale and dimensionality of the raw text data, existing methods use simple heuristics or require human experts to manually curate data. Instead, we extend the classic importance resampling approach used in low-dimensions for LM data selection. We propose Data Selection with Importance Resampling (DSIR), an efficient and scalable framework that estimates importance weights in reduced feature space for tractability and selects data with importance resampling according to these weights. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents from the full Pile dataset in 4.5 hours. To measure whether hashed n n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, data metric that measures the proximity between the selected pretraining data and the target on some feature space. Across data selection methods (including expert selection). KL reduction hashed n-gram features highly correlates with average downstream accuracy (r=00 0.82). When selecting specific domain, DSIR performs comparably
data for continued pretraining to expert curation across 8 target distributions. When pretraining general-doman models (target is Wikipedia and books), DSIR improves over random selection and heuristic filtering baselines by 2-2.5% on the GLUE benchmark."
Introduction
Given fixed compute budget, the choice of pretraining data is critical for the performance of language models (LMs) [18; 10; 24; 64; 27J. Existing works rely on heuristics to select training data. For example, GPT-3 [10] and PaLM [14] filter web data for examples that are closer to formal text from Wikipedia and books Specifically, they train a binary classifier to discriminate formal text from web data and select web examples that have predicted probability above noisy threshold [10: 18; 21]. However, heuristic classification does not guarantee that the selected data is distributed like formal text. As second example, domain-specifie LMs such Minerva [45] and Codex [11] (math and code LMs, respectively) employ domain- adaptive pretraining (DAPT) [24], where the model is initialized from base LM and continues to be pretrained domain-speciff dataset to achieve gains over the base LM on that domain. The domain-specific datasets are typically manually curated, but a framework for automating data selection could save effort and increase the amount of relevant training data.
proxy for high quality, method which we call heuristic classification.
Code, selected data, and pretrained models are available at https: /ttp:::///itaai http://ggttuuu.otoo.gge com/pmmo///ooo lambda baaadda/dedd deir.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Target data
Importance weight estimator
1. Estimate importance weights using raw + target data (simple bag-of- ngrams estimator)
Large raw dataset (e.g., The Pile)
2. Select data via importance resampling
Subset of raw data distributed like target
Figure 1: Given a large raw dataset such as The Pile [21] and a smaller target dataset (e.g., Wikipedia + books), we aim to select subset of the raw data that is distributed like the target in some feature space. Our method, DSIR, first estimates importance weights using raw and target data in an n-gram feature space. The importance weights are used resample subset of the raw dataset.
In this paper, we consider the problem of data selection; given a large and diverse raw dataset (c.g.. The Pile [21]) and a smaller dataset sampled from a desired target distribution, choose a subset the raw data that is distributed similarly to the target (Figure 1). While a natural approach is to resample the raw data according importance weights (importance resampling [68]), estimating importance weights high dimensional data such text is often statistically intractable [6; 75].
Instead, our Data Selection with Importance Resampling (DSIR) framework efficiently estimates importance weights over a featurization of the raw and target distributions (Section 3). Our framework first maps the raw and target data onto some feature space and resamples subset raw data according to importance weights computed in this feature space. DSIR is extensible via the choice of feature space and importance estimator, which specify what aspects of the data we care about.
What is a feature space that both allows for efficient computation and preserves aspects of the data that are relevant for the target? In Section 4, instantiate DSIR with hashed n- n-gram features, where n- n-grams are hashed onto a fixed number of buckets, for efficiency and scalability. The importance estimator parameterized by bag-of-words generative models on the hashed n-grams, learned by simply counting the the hash bucket frequencies. DSIR with hashed n-grams enables the selection of 100M documents from the full Pile dataset in 4.5 hours.
To evaluate how well hashed n- n-grams preserves the aspects of the data that are relevant for the target, in Section 6 we define KL reduction, data metric that measures how much selected dataset reduces the Bolllbakk-Leiblee (KL) divergence to the target (in some feature space) over random selection (KL((tagett random)- (target[/sellected)) We show in Section 5 that KL reduction highly correlates with average downstream performance (Pearson r 0.82) across data selection methods, including expert selection.
We consider selecting data from The Pile (1.6B examples) for continued pretraining of domain-specific LMS and training general-domain LMs from scratch. First, we select data for continued pretraining [24] of domain- -spaiic LMs in a controlled setting where the target samples are unlabeled training inputs from a downstream dataset (Section 5). We perform continued pretraining the selected data starting from RoBERTa [48] and evaluate by fine-tuning on the downstream dataset (whose unlabeled inputs were also used as the target for data selection). On 8 datasets from domains (CS papers, biomedical papers, news, reviews), DSIR improves over RoBERTa (no continued pretraining) by 2% on average and is even comparable to continued pretraining expert-curated data from Gururangan et al. [24].
For general-domain LMs (Section 7), the data selection target is formal, clean text from Wikipedia and books, following GPT-3 [10]. We train masked language model (MLM) from scratch the selected data and evaluate by fine-tuning on GLUE [82]. In controlled experiments, heuristic classification performs comparably to random sampling from The Pile, possibly because The Pile is already filtered

---

using heuristic classification. DSIR improves over both baselines by 2-2.5% on average on GLUE, We publish the selected dataset and the code for DSIR improve future LM pretraining.
2
Setup
Given a small number of target text examples z'j .i.......'' from a target distribution of interest and a large raw dataset 2 "N drawn from distribution q, we aim to select examples (k < N) from the raw dataset that are similar to the target.
Selection via heuristic classification. As a starting point, we first define the heuristic classification method used by GPT-3/The Pile/PaLM [10: 21; 14]. In heuristic classification, we train binary classifier f: X - [0.1] to output the probability that an input is sampled from the target distribution. The model is typically fasttext linear classifier on n- n-gram feature vectors (usually unigrams and bigrams) [30]. We initialize the feature vectors from pretrained fasttext word vectors. We use the trained classifier to estimate (xx;). the predicted probability that 2; is sampled from the target, for all raw examples. Example v; is selected if / (x:; > 3r, where 3, is sample from a Pareto distribution (typically with shape parameter a = 9 [10; 14)). Since each example is kept or discarded independently, to select a desired number of examples k, the process must either be repeated or a must be tuned. Heuristic classification selects examples from modes of the target distribution (high f(x;) )), which could lack diversity. To combat this, noise 3, added. However, is unclear how much noise to add and there no guarantees on the selected data distribution.
3 Data Selection with Importance Resampling
In the DSIR framework, we consider using importance resampling [68] to select examples that are distributed like the target. However, estimating importance weights high dimensional data like text is often statistically intractable without sufficient additional structure [6; 75; 23].
Instead, we employ importance resampling on a feature space Z that provides this structure. DSIR uses feature extractor h: > 2 to map the input. x to features z = ((x).. The induced raw and target feature distributions are (Jeet and Pleat, respectively. The goal is to select examples with features that are approximately distributed according to the target feature distribution Pleat Depending on the choice of feature extractor, DSIR focuses on different aspects of the input. For example, an n-gram feature extractor focuses on matching the n-gram frequencies of the selected data and the target.
Our framework consists of 3 steps:
1. Learn Pjear and Heer We learn two feature distributions Premt and Ireat using held-out featurized examples from the target and raw data, respectively.
2. Compute importance weights: We compute the importance weights w; featurized example 2; 100=h((a h(x;)) from the N raw examples.
(a)) Yiar (3.)
for each
3. Resample: Sample k examples without replacement from categorical distribution with
w. probabilities Σ m. Sampling without replacement avoids choosing the same example
multiple times, is more statistically efficient for importance resampling [23], and can be implemented efficiently with the Gumbel top-k trick [81; 38; 87; 39].
4 DSIR with Hashed N-gram Features
For efficiency and scalability, we instantiate DSIR with hashed n-gram features. Later Section 6, we test whether hashed n-gram features preserve the information needed to select data relevant to the target.
Hashed n-gram features. We consider hashed n n-gram features inspired from fasttext [30; 86]. Specifically, for each example we form a list of unigrams and bigrams, hash each n- n-grams into one of m buckets (m 10000 this paper), and return the counts of the hashed buckets in m-dimensional feature vector E N For example, if the text input is "Alice is eating", we form the list [Alice, is, eating, Alice is, is eating], hash each element of this list to get list of indices [1, 3, 3, 2, 0] and return the vector of counts for each index [1, 1, 1, 2, ... J. While the hashing
3

---

introduces some noise due to collisions, we find that this is simple and effective way to incorporate both unigram and bigram information.
Bag of hashed n-grams model. We parameterize the raw and target feature distributions Pteat and grent as bag-of- of--aamssnsssss models. The bag-of-ngrams model has parameters Y E A"", which is a vector of proba- bilities on the hash buckets that sums Under this model, the probability of feature vector EN is
P(z)) = >bb
(I)
where the bracket notation selects the corresponding index in the vector. Given some featurized examples 21...., 8o sampled from feature distribution, we estimate the parameters by counting:
9= 1 1
Speed benchmark on The Pile. To test the scalability of the framework, we benchmark DSIR with hashed n-gram features on selecting data from the full Pile dataset [21]. For this test, we do not preprocess the data other than decompressing the text files for faster I/O. We use hashed n-gram features with 10k buckets, fit the raw feature distribution with IB hashed indices from the Pile, and fit the target feature distribution with the full target dataset (ChemProt [41]). DSIR selects 100M documents from the full Pile dataset in 4.5 hours on I CPU node with 96 cores. Almost all of the time (4.36 hours) is spent computing the importance weights the raw dataset, while fitting the feature distributions (1 minutes) and resampling (6 minutes) were much faster. Increasing the number of CPU cores can further decrease the runtime.
Selecting Data for Domain-Specific Continued Pretraining
In this section, we use DSIR to select domain domain-specifie data for continued pretraining. We compare DSIR to 7 other data selection methods in this continued pretraining setting.
We select data for target distributions in the setting of Gururangan et al. [24], where we perform continued pretraining of domain-specific LMs. Here, the target is a specific downstream unlabeled data distribution and we select examples from The Pile (the raw data). For each downstream dataset, we select data for continued pretraining starting from RoBERTa [48] (see Appendix H). Following Gururangan et [24], we consider 8 downstream datasets across 4 domains: Computer Science papers (ACL-ARC [31]. Sci-ERC [51]), Biomedicine (ChemProt [41], RCT [16]) News (AGNews [92], HyperPartisan [34]), and Reviews (Helpfulness [53], IMDB [52]).
Setup.
Baselines. Beyond random selection (without replacement) and heuristic classification, also compare against manual curation [24] and a top-k variant of heuristic classification. In manual curation, simply fine-tune from domain-adaptio pretraining (DAPT) checkpoints [24], which are the result of continued pretraining on manually-
data. In top-k heuristic classification, we select the top-k-scoring examples according to the binary classifier used in heuristic classification. All methods select data from The Pile except for manual curation, which uses domain saaain-seeificc data sources [24].
We perform controlled comparison by equalizing the amount of LM training compute for all methods, measured by the number of tokens processed during training, following the compute budget in Gururangan et al. [24]. For random selection, heuristic classification, and DSIR using n-gram features (defined in Section 4), we control the number of selected examples (25M examples with fixed token length 256) and the training protocol. We standardize the fine-tuning for all models and average all results over random seeds (see Appendix H for details). All the models initialize from RoBERTa-base Before data selection via DSIR or heuristic classification, we remove extremely short (<40 words) or repetitive documents that tend to be uninformative (Appendix I).
Automatic data selection with DSIR can replace manual curation. between the data selection methods. To summarize:
Table shows the comparison
On average, DSIR improves over random selection by 1.2% and manually curated data (DAPT) by 0.3%, showing the potential to replace manual curation.

---

Table 1: F1 scores for continued pretraining from the RoBERTa checkpoint [48] 8 downstream datasets from 4 domains (CS, Biomed, News, and Reviews). Random selection, heuristic classification, and DSIR train 25M selected examples from The Pile. Heuristic classification and DSIR create a different pretraining dataset for every downstream dataset. All models (including DAPT [24]) use the same amount of training compute and results are averaged over 5 seeds, with standard deviations in subscripts. All datasets use macro-F1 except ChemProt and RCT, which use micro-F1.
ACL-ARC
Sci-RRC
ChemProt
RCT
HyperPartisan
AGNews
Helpfulness
IMDR
Avg
RoB (no continued pretrain)
66,80,
40.
82.31
86.68o.14
8885,
93.35
65.08
94 94.38c.11
82.20
Random selection
67.51
40.53,
83.146.52
66.85
86.42..a
93.52
68. 15
94.49c.2
Manaul DuraiionDAPPPP [24)
71.84a.
80.12) 57
84.17a.50
$7.1
87.23
93.61am1
68. 68.211.0r
95.08.
83.46
Heuristic classification
69.942.0
80.522..s
83.35
86. 78
85.71g
93.54
68 68.50
94 94.66,
82.88
Top-é Houristic classification
71
80.23
84.11o.s
93.67.11
69.18am
DSIR
72.862.71
80.44,.s8
85.51
87.140.s
87.01
93.622uu9
68 68.956.s
94.56p.5 50.54
83.76
Table 2: Fl scores for continued pretraining from RoBERTa, testing different components of heuristic classification and DSIR methods. For heuristic classification, replacing the Pareto noisy threshold with calibration and importance resampling (DSIR (fasttext discriminative)) improves F1. Generative importance weight estimators outperform discriminative importance weight estimators for DSIR. All results average over 5 seeds, with standard deviations in subscripts.
ACL-ARC
Sci-ERC
ChemProt
RCT
HyperPartisun
AGNews
Helpfulness
IMDB
Houristic classification
69.942us
80.52
83.35,ur
78c.17
85.71a.
93.54..1
68.50..7
94, 666.
82.88
DSIR (a- gramra generative)
72.86:
80.44.
85.51
87.14
87.01
93.62,,1
68.95..:
94.56e.31
83.76
DSIR discriminative)
68.16-.15
79.00, so
84.57am
87.09
89.18
93.54..14
68.41..5
94.95
DSIR (o- gram discriminative)
70.35 00
80.21o
85.03
87.04p 1s
85.49
93.74:
68.79 mz
94.84,
83.19
DSIR (unigram generative)
69.53
79.69..2
88. 88.24
87.05e.c
90.11
93.422.1e
68.55%.12
94.3%...
83.50
DSIR improves over heuristic classification by 0.9% and is comparable to top-k heuristic classification. We note that top-k heuristic classification not typically used in this setting, but we find that it may be particularly suited for Nomain-specific data selection, where diversity may less important than the general-domain setting.
Random selection improves by 0.4% on average over no continued pretraining all, showing that additional data generally improves the downstream Fl score. All the targeted data selection methods improve over random selection.
Discriminative importance weight estimators underperform generative estimators. We experiment with replacing components of DSIR in Table 2. First, we consider using the binary classifier from heuristic classification (which takes pretrained fasttext word vectors as input) as the thepotaane importance weight estimator in DSIR. For imput z4. the classifier predicts the probability of target f(z)). We use
this to estimate importance weights T- [=,) then resample according to these weights. This approach (DSIR (fasttext discriminative)) improves F1 by 0.3% over heuristic classification. However, this approach still underperforms DSIR by 0.6% on average, even with regularization and calibration.
f(ms)
We consider another discriminative version of DSIR that uses hashed n-gram features as input to a logistic regression binary classifier for importance weight estimation. This differs from heuristic classification, which initializes with pretrained fasttext feature vectors and fine-tunes the features along with the classifer. This approach (DSIR (n (n-gram discriminative)) underperforms DSIR by 0.7%, even with regularization and calibration. These results suggest that generative approach is better suited (or casier to tune) for importance resampling. However, the discriminative approaches still outperform random selection by 0.6%.
Selecting with n-grams improves downstream performance over unigrams. DSIR uses both unigram and bigram information to compute hashed n n-gram features. We ablate the role of bigram information in hashed n-grams by using hashed unigram features (with 10k buckets) for DSIR. In Table 2, we find that DSIR with unigram features underperforms DSIR with n-grams by 0.26%, though still achieving comparable F1 score manual curation. Overall, selecting data with unigrams is effective, but including bigrams further improves the relevance of the selected data.
Cross-domain analysis and the effect of the choice of pretraining data. DSIR assumes knowledge of the target distribution, but what happens if the target dataset is not representative of the target
5

---

Figure 2: FI scores of DSIR for all pairs of pretraining data target distributions (rows) and downstream tasks (columns). The cells are colored by its per-column ranking, with better rankings (higher F1 scores) having darker colors. While using the pretraining data selected specifically for the downstream task is typically strong, choosing the worst pretraining dataset for the downstream task reduces F1 by 6% average. All results averaged over 5 seeds.
Fine-tune dataset
HyperParticn
Helptalness
ACLARE
SoiENC
ChemPret
AONews
RCT
MOB8
Avg
ACL-ARC
72.86
80.08
83.32
86
87 00
93.05
67.8
94.0
03.31
CS
SckERG
73.37
80.44
83.73
87
77.17
93.44
68
93.89
82.2
Target distributton for pretrain data
ChemProt
70.14
77.84
65.51
86 81
90.61
93.47
67.67
94.21
83.28
Blomec
RCT
71.25
77.96
$3.96
87
77.86
93.25
06.53
93.49
81.43
HyperPortisan
65.55
80.52
82.25
86
87.01
93.29
67.25
94.37
82.1
AGNeww
79.69
Helpfulness
Avg 79.31
distribution? To test the effect of varying the target distribution on downstream performance, we consider every pair of pretraining dataset, which is selected by DSIR for target downstream task X. and downstream task Y. Figure provides the full matrix of results. We find 6% average drop in Fl when choose the worst pairing for each downstream task instead of matching the pretraining and downstream data. In the worst case, the F1-score on HyperPartisan drops by 30%. Thus, the choice of target distribution can have large effect downstream performance.
Pretraining data transfers better for targets within the same domain. In practice, we may have access some target datasets the relevant domain and hope to select pretraining data that can improve performance on other tasks in that domain. The 8 target/downstream tasks we use come from 4 domains, with 2 tasks from each domain. We define within-domal Fl as the average Fl the pairs of pretraining and fine-tuning data from the same domain, but excluding pairs where the pretraining data is selected for the fine-tuning task. We compute this by averaging the off-diagonal elements the x diagonal blocks of the matrix in Figure 2. We find that the within-domain Fl (82.9% is 1.7% higher on average than the cross-domain F] (81.2 12),, where the pretraining data selected for a target from a different domain.
KL Reduction on Hashed N-grams Predicts Downstream Performance
When designing a feature extractor for DSIR, how do we measure whether the features preserve the information for selecting relevant pretraining data? To answer this question, propose data metric, reduction, which measures how much data selection reduces distance to the target over random selection in feature space. We find that KI reduction on hashed n-gram features strongly correlates with downstream performance across various data selection methods, including those that do not involve n- n-grams, such manual curation.

---

Average downstream F1
0.82
RoBERTa baseline
DSIR (n-gram generative)
X
DSIR (unigram generative)
Random selection
83.0
Manual curation (DAPT)
Top-k Heuristic classfication
Heuristic classification
82.5
DSIR (fasttext discriminative)
DSIR (n-gram discriminative)
0.00
0.05
0.10
0.15
Average KL reduction (KL(target || random) (L(target selected))
Figure 3: Plot of average KL reduction on the n- n-gram feature space, defined how much the selected dataset reduces KI. divergence to the target distribution over just random sampling from The Pile, against average downstream score over the continued pretraining datasets in Table There strong correlation between KI. reduction and downstream performance (Pearson 0.822..
define KL reduction as the average reduction in empirical KL divergence from doing data selection over random selection over set of target feature distributions T:
KL reduction metric.
KL-reduction(p leat :eeat: 7)
Σ
(L (tea) KLL(Ptaa feat)
[T]
Prou
where y' feat is the empirical feature distribution of the selected data, Pteat is a empirical target feature distribution, Õreat is the empirical raw feature distribution. KL reduction depends on the raw distribution Trem and the set of target distributions T as hyperparameters. In our continued pretraining setting, Oreat the feature distribution of the Pile and consists of the feature distributions from the downstream tasks from Section 5.
KL reduction hashed n-grams predicts downstream performance. We show that when computed on the hashed n-gram feature space, KL reduction of a selected dataset highly correlates with the downstream performance a model trained on that data. Figure plots KL reduction against average downstream performance over 8 target distributions for data selection methods from The Pile [21], where the distribution parameters are estimated using 100k samples from each dataset. The average downstream Fl score is highly correlated with the KL reduction (Pearson r 0.82). This agrees with the results of Razeghi al. [65] for in-context learning [10] and extends the preliminary evidence from Gururangan et al. [24] on one selection method that better unigram overlap improves downstream performance. DSIR with hashed n-gram features achieves the highest KI. reduction and the best average downstream F1.
While some of the original pretraining datasets for DAPT [24] were not publicly available, downloaded the public versions as an approximation. Our results suggest that hashed n-gram features preserve most of the information needed for selecting data relevant to the target. Since KL reduction highly correlates with downstream performance and can be cheaply computed without training an LM, KL reduction can be used as a sanity check for future data selection methods.
7 Selecting Data for Training General-Domain LMs
In this section, we consider selecting formal text (as proxy for high-quality text) for training general-domain LMs from scratch. We use Wikipedia and books as the target distribution.
Baselines and setup. We compare the following methods for selecting data from the Pile: 1) Random selection, 2) Heuristic classification (GPT-3/Pile/PaLN) method), and 3) DSIR, As ablations, we consider top-k variants of heuristic classification and DSIR (take the top-k examples according to importance weights instead of resampling). We use each method to select 51 1.2M examples, which corresponds to 4 epochs with our compute budget. For heuristic classification and DSIR, we select 96% of the examples from domains excluding Wikipedia and books. This is done to reduce the bias towards selecting data from Wikipedia and books (the target distribution). We choose the other 4%
7

---

Table 3: Accuracies on the GLUE [82] dev set for stRe masked language model [17] trained on data selected from The Pile [21]. Following RoBERTa [48], for RTE, STS, and MRPC we fune-tune starting from the MNLI model instead of from scratch. DSIR outperforms heuristic classification (used by GPT-3 and PaLM) and random selection by over 2% on average. All results are averaged over seeds and standard deviations are in subscripts.
MNII
QNLI
QQP
RTE
SST-2
MRPC
Col.A
STS-B
Avg
Random selection
82.639.41
86. 26.90
89.576.30
67.37
90.050.41
87.40,
49.413.5
88.63,11
80.25
Heuristic classificution
82, 69.
85.950.79
89.77a.52
68.591.75
88.94u.3
86.03..
48.17533
88.62..22
79.85
Top-A Heuristic classfication
83.34..22
88.62224
89.89..1
70.04pm0
91.150,70
86.37_o0
53.022.50
89.30..1
81.47
DSIR Top-k DSIR
83.07
89.116.1
89,8 89.80
75.09
90 90.48u.57
87.70..6s
54,00
89.17
82.30
83.39
88 88.63
89.94..17
72.49
91.01u.79
16,18,
49. 49.90g.50
89.52,.21
81.38
uniformly from Wikipedia and books, and did not tune these proportions (Appendix F). We apply quality filter for extremely short or repetitive examples before heuristic classification and DSIR selection (Appendix J). For each dataset, we perform MLM pretraining for 50k steps with a large batch size (4096) and short token length (128), following Izsak et al. [28]. All the models use the BERT-base architecture [17]. We evaluate the models on the GLUE dev set, averaged over 5 fine-tuning runs [82]. Fine-tuning hyperparameters such as the number of epochs and batch size are fixed for each dataset, following reasonable defaults from the RoBERTa codebase [48].
DSIR qualitatively selects more formal text. Figure 4 shows the beginning characters of 20 random examples selected by random selection, heuristic classification, and DSIR. The random sample contains many code examples that are not similar to text from Wikipedia and books. Heuristic classification seems slightly too diverse, which suggests that the variance of the Pareto distribution added to the classifier scores may be too high. Note that we use the setting of the Pareto shape hyperparameter used in GPT-3 [10]. Qualitatively, DSIR selects the most formal text. By doing importance resampling to match the target distribution, DSIR trades off the relevance and diversity of the selected data automatically.
DSIR improves GLUE performance. Table 3 shows results on the GLUE dev set. DSIR achieves 82.3% average GLUE accuracy, improving over random selection by 2% and heuristic classification by 2.5%. Heuristic classification leads to 0.5% lower accuracy than random selection from The Pile. We hypothesize this is because The Pile has already been filtered once with heuristic classification.
Resampling outperforms top-k selection. Top-k heuristic classification and top-k DSIR have similar performance across datasets, with some tradeoffs compared to DSTR without top-k. DSIR without top-k is competitive with these variants on all datasets and achieves a 0.8-0.9% higher average over top-k variants. All the top accuracies across datasets are achieved by DSIR or top-k DSIR.
alignment, The value of Z-k simultaneously. To prove
in London and, like raid5 sfil by accident denied that the plaintiff Are celebrities the new made on the antimesenteric
when ship rats invaded. Stephon Gilmore'nBuufaala This story is a sequel BLM to Begin Massive Below, in alphabetical enforcement, the office O' O'Brien is Senior Lockwood, commander ten- ten-year-old girl. the suburbs, a regular American lawyersinCategory: state's 's chief exccutive. of scheduled program,
[Pphi]
WrlWe" Halpha
suggests that, nicotinic
THE AUTHORS OR COPYRIGHT
woodland species consumed they differ from gecko that someone may legit
H&E coupe over a length
(codingSttte SMModel
Five lessons in viral
console. logterr);
[m]_s) [b]__]]L_.S
lusepucckaee (eaeuuccageWwasyyyyng (wasysym)
Omega, which antomatically
e..". ..... True)
C1--01- -0-----1-2222 Cu2 128. 2(aa2)x))22)
Prison officials deposed plan see 30 more." Misdiagnosis of mosaic havent'l had issues when slowly
the given properties
GPI. Classpath Exception:
Guard unit was identifiably have to pul up the word Indiana head coach Archie a committee comprised of Filipacchi Media said Anna had the necessary a few days ago.
husband and I quit domating
The helical character of new TostWritel Line("
napkin.
In turn, this may trigger 50u20099l of 20w2009mM we are afraid of patrons. Linas Gylys, president
6-month period in 2014--15 15
"Alice Adams: An Intimate
index: m_ Stttertaart:
(a) Random selection
(h) Heuristic classification
(c) DSIR
Figure 4: Beginning characters of 20 random examples (each line is different example) selected by random selection, heuristic classification, and DSIR, where the target is formal text from Wikipedia + books. Qualitatively, DSIR selects more formal text than random selection and heuristic classification.
8

---

8
Related Work
Effect of pretraining data on LMs. The pretraining data has a large effect LM performance. Lee et Hermandezz al. [26] show that deduplicating data improves LMs, and Baevski et [3]; Yang al. [88] compare using large web corpus versus Wikipedia. Raffel et al. [64] shows that heuristically filtered data (filtering out short and duplicated examples) improves TS and Du et al. [18] shows that heuristic classification improves downstream few-shot performance for GLaM. We provide extensive controlled experiments comparing the effect of data selection methods on downstream performance.
Retrieval. Yao al. [89] use keyword-basee retrieval (BM25) to select data for semi-supervised learning. In preliminary tests, we found that out of 6.1M MM documents retrieved by BM25, there were only 1.8M unique documents (70% were exact duplicates). These duplicate examples can hurt performance [44; 26]. Selecting a desired number of unique documents involves oversampling and de-duplication. Instead, we consider top-k heuristic classification, which has similarities to cosine similarity- based retrieval (since heuristic classification uses an inner product score between pretrained word embeddings and a learned class vector) and avoids retrieving repeated examples.
Data selection in classical NLP. Moore-Lewis selection [56; 2: 20] takes the top-k examples in cross-entropy difference between n-gram LMs trained on target and raw data to score examples, which could over-sample examples from the mode of the target distribution. In Section 7, we found that top-k DSIR, which is form of Moore-Lewis selection with hashed n- n-gram LMs, underperforms DSIR by 0.9% on GLUE. DSIR naturally balances diversity and relevance for use in both domain-speciin and general- cases, since it uses importance resampling to match the target distribution. discrepancy measures [29; 69; 47] have also been used in selecting data in the domain adaptation setting. Overall, these methods do not consider importance resampling and do not address the gap between pretraining and downstream tasks: pretraining has a different objective to fine-tuning, pretraining uses unlabeled data that is not form-foddtted,, and the influence of pretraining data is separated from the final model by the fine-tuning step. Beyond the preliminary evidence that unigram similarity metrics are related to downstream performance in Gururangan et al. [24], we show gap,
comprehensively and quantitatively on 8 selection methods that despite the n-gram KL reduction on pretraining datasets highly correlates with downstream performance.
Data selection in deep learning. Many works show the importance of data selection in the supervised or semi-supervised learning setting in vision [76; 54; 33; 36; 35; 37; 83; 85; 61; 55; 15; 71; 5] and in language finetuning [4]. While most select image data from CIFAR or ImageNet, which have up to 1-10M examples, we consider selecting text data from The Pile. which has over 1.6B examples (of 128 chhitespace-delimitee words each). this scale, previous methods become quite expensive since they typically require running a neural network forward pass to get embeddings [71; 76; 37], taking gradients [35; 36; 83; 61; 55], or training reference model [54]. In contrast, we construct simple gram-based selection method that easily scales to internet-scale datasets. Coleman et al. [15] select data with high uncertainty under smaller proxy neural model. They do not consider using a target dataset for estimating importance weights. However, using a neural model could be a complementary strategy for importance resampling. Other works [70; 500 32] focus on choosing a subset that approximates training with the original dataset and require selecting data online during training. We aim to select a targeted dataset (once, before training) with different properties from the raw data (restricting the data to formal text or specific domain). Our work also differs from active learning methods [72; 19; 84; 90; 80), which query an annotator for more labeled data. Instead, we select data for eeff-super pretraining.
Importance weighting and domain adaptation. Many methods tackle the high- nigh-dimensional im- portance weight estimation problem [79; 67: 12; 13]. In particular, importance weighting classically used in domain adaptation [74; 78), where unlabeled target examples are used to adapt model trained on labeled source data, for reweighting the loss function. However, in many modern applications the source and target are often disjoint (e.g., sketches vs. natural images), causing undefined importance weights [62; 42; 73). We side-step nigh-dimensional importance weight estimation by instead working in reduced feature space where the support of the massive web corpus should cover the target.

---

Discussion and Limitations
Feature space for importance resampling. Finding an appropriate feature space important for DSIR. Although we find tight correlation between downstream performance and our data metric compute using hashed n-gram features, n-grams only capture superficial word-level overlap. Other feature extractors, such neural models, may produce features that better capture semantics. We consider a variant of DSIR which estimates importance weights on a neural feature space in Appendix B, and find that this variant also improves by 1-1.5% over random selection and heuristic classification on GLUE, but our preliminary version does not improve over DSIR with hashed n- n-gram features. However, extracting these features is much more computationally expensive (on the order of D times more FLOPs for D-p D-parameter neural model), and importance weight estimation on this continuous feature space may be more difficult.
Parameterization of the importance weight estimator. In principle, both generative and discriminative approaches to estimating the importance weights should work. In a discriminative approach, regularization and calibration should be used to combat overfitting and make the predicted probabilities useful for importance resampling. We find that generative approach requires less tuning and could also be better when the number of target examples is small, as Ng and Jordan [58] finds that Naive Bayes often performs better than logistic regression in low-sample regimes.
What is the right target distribution? When developing domain model such as Codex [11], the target dataset should be representative of the coding tasks we expect the model to be used However, it's unclear how exactly collect this dataset and how much to weight each task in the target distribution. Developing better procedures for collecting the target dataset can ultimately improve the data selected by DSIR. For general-domain LMs, follow GPT-3, the Pile, and Pal.M in using for- mal text from Wikipedia and books as proxy for high quality text [10; 21; 18; 14]. However, this is just heuristic. We leave the exploration of other target distributions for gemeral-domain LMs to future work.
Broader impacts. The impact of DSIR depends the properties of the target data. While DSIR could amplify biases present in the target examples, with the appropriate target data, DSIR can be used collect data that improve the training efficiency, alignment, bias of LMs [59; 4; 40]. These benefits could reduce the environmental impact of LMs [77; 43; 46; 60] and reduce their biases and risks [9; 1; 22: 8). For example, DSIR can be used to collect more data on underrepresented subpopulations and tune the model on this data to improve model fairness.
10 Conclusion
We provide cheap and scalable data selection framework based on importance resampling for improving the downstream performance of LMs. We also find data metric, KI. reduction, that strongly correlates with downstream performance and can provide a sanity check for data selection methods without training model. Our work provides a step in understanding the choice of pretraining data for downstream transfer in LMs.
11 Acknowledgements
We thank Neil Band, Hong Liu, Garrett Thomas, and anonymous reviewers for their feedback. This work was supported by an Open Philanthropy Project Award and NSF IIS 2211780. SMX was supported by NDSEG Fellowship. SS is supported by Open Philanthropy Graduate Fellowship.
References
[1] Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models, arXiv preprint arXiv:2101.05783, 2021.
[2] Amittai Axelrod. Cynical selection of language model training data. CoRR, abs/1 000.070922222 022.99, 2017. WRLhttp::// httpp:/arxii..... org/abs/ abs/ hbbb://.7.299. 02279.
10

---

[3] Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self- networks. arXiv, 2019.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, S. El-Showk, Nelson Elhage, Zac Hatfield- -Doddd, Danny Hernandez, Tris- tan Hume, Scott Johnston, S. Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, C. Olah, Benjamin Mann, and J. Kaplan, Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022.
[5] Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning (ICML), 2009.
[6] Thomas Bengtsson, Peter Bickel, and Bo Li. dfrre-of-dimmennionnaitt revisited: Collapse of the particle filter in very large scale systems. arXiv, 2008
[7] Steven Bird, Edward Loper, and Ewan Klein. Natural Language Processing with Python. O' R'Reilyy Media Inc., 2009.
[8] Su Lin Blodgett and Brendan OConnor. Racial disparity in natural language processing: A case study of social media Arican-Americn English. arXiv preprint arXiv:1007.00061,,2 2017.
[9]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Antoine Bosselut, Emma Brunskill, Erik Brynjolfssonn, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam. Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose F. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 108.07258, 2021.
Peter Henderson,
[10]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv: 14165, 2020.
[11] Mark Chen, Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv: 03374, 2021.
11

---

[12] Kristy Choi. Madeline Liao, and Stefano Ermon. Featurized density ratio estimation. Uncertainty in Artificial Intelligence (UAI), 2021.
[13] Kristy Choi, Chenlin Meng. Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
[14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, A. Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, B. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, M. Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya. S. Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. D. Luan, Hyeontaek Lim, Barret Zoph, A. Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, T. S. Pillai, Marie Pellat, Aitor Lewkowycz, E. Moreira, Rewon Child. Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, K. Meier-Hellsterr D. F.k, J. Dean, Slav Petrov, and Noah Fiedel. Ma Scaling language modeling with pathways. arXiv, 2022.
[15] Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang. Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations (ICLR), 2020.
[16] Franck Dernoncourt and Ji Young Lee. Pubmed 200k ret: dataset for sequential sentence classification in medical abstracts. I/CNLP, 2017.
[17] Jacob Devlin, Ming-Weo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: deep bidirectional transformers for language understanding. Associaaio for Computational Linguistics (ACL), pages 4171 1777714466,,.1 2019.
Pre-training of
[18] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, M. Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang. Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of exeeeesss.. arXiv, 2021.
K.
[19] Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active Learning for BERT: An Empirical Study. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7949-7962, Online, November 2020. Association for Computational Linguisties. doi: 10. 1/20200.mmmp-mmai..663. URL 2rg/2020..oo emnlp-main. 638.
[20] Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoe. Automatic document selection for efficient encoder pretraining, 2022. https:/// htt.aaawww.....aaaaaaa/aaaaaaaaa wrgg a2tpaa///./////////////aaaaee 10951.
[21] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The pile: An 800gb dataset of diverse text for language modeling. arXiv, 2020.
[22] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxieityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 1142,, 2020.
[23] Andrew Gelman and Xiao-Li Li Meng. Applied Bayesian modeling and causal inference from incomplete-datt data perspectives. Wiley Series in Probability and Statistics, 2004.
[24] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah Smith. Don't stop pretraining: adapt language models to domains and tasks. arXiv preprint 20044 2020.
12

---

[25] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In World Wide Web (WWW), 2016.
[26] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-oodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability of learning from repeated data. arXiv, 2022.
[27] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas. Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[28] Peter Izsak, Moshe Berchansky, and Omer Levy. How to train BERT with an academic budget. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[29] Jing Jiang and ChengXiang Zhai Instance weighting for domain adaptation in NLP. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264-271, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: http:/aclaatton aalaacalaantoooyy.oo
arg/P07-20 1034.
[30] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. European Chapter of the Association for Computational Linguistics (EACL),
2, 2017.
[31] David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics (TACL), 6, 2018
[32] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning (ICML), 2018.
[33] Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. IEEE/CVF Winter Conference on Applicatios of Computer Vision (WACV), 2019.
[34] Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Comey, Benno Stein, and Martin Potthast. Semeval2019 task 4: Hyperpartisan news detection. SemEval, 2019.
[35] Krishnateja Killamsetty, Durga S, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. GRAD- MATCH: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning (ICML), 2021.
[36] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
[37] Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi cami--uperrvied learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[38] Carolyn Kim, Ashish Sabharwal, and Stefano Ermon. Exact sampling with integer linear programs and random perturbations. In Association for the Advancement of Artificial Intelligence 2AAA),2016..
[39] Wouter Kool, Herke van Hoof, and Max Welling, Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. International Conference on Machine Learning (ICML), 2019.
13

---

[40] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences, 2023. 2023.
[41] Jens Kringelum, Sonny Kim Kjærulff, Søren Brunak, Ole Lund, Tudor 1. Oprea, and Olivier Taboureau. Chemprot-3.0: a global chemical biology diseases mapping. Database, 2016.
[42] Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning (ICML), 2020.
[43] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXi:: 2019.
[44] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Association for Computational Linguistics (ACL), 2022.
[45] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
[46] Anne-Laure Ligozat, Julien Lefévre, Aurélie Bugeau, and Jacques Combaz. Unraveling the hidden environmental impacts of Al solutions for environment. CoRR, /e/o///111.118000 2021. URL httpt:ttt//arat /ttp:///aavvvvv/a org/ abs /2100.210... 11822.
[47] Miaofeng Liu, Yan Song, Hongbin Zou, and Tong Zhang. Reinforced training data selection for domain adaptation. In Proceedings the 57th Annual Meeting of the Association for Compu- tational Linguistics. pages 1957-1968, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18633/v//199. 1189. URL httpst http:///ccaaanhoooooo// wwtp 1189.
[48] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907. 11621,169,,,000
[49] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. S2ore: The semantic scholar open research corpus. In Association.fo Computational Linguistics (ACL), 2020.
[50] Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. In International Conference on Learning Representations Workshop (ICLR), 2016.
[51] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
[52] Andrew L. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Association for Computational Linguistics (ACL), 2011.
[53] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations styles and substitutes. SIGIR, 2015.
[54] Sören Mindermann, Jan Brauner, Muhammed Razzak. Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning (ICML), 2022.
[55] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning (ICML), 2020.
[56] Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Pro- ceedings of the ACI. 2010 Conference Short Pupers, pages 220-224, Uppsala, Sweden, July 2010. Association for Computational Linguistics. https / h/ala///holcaallg///coo org/P10-2044 2041.
14

---

[57] David R. Musser. Introspective sorting and selection algorithms. Software: Experience, 27, 1999.
Practice and
[58] Andrew Y. Ng and Michael I. Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. In Advances in Neural Information Processing Systems (NeurIPS), 2002.
[59] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. J. Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, P. Welinder, P. Christiano, J. Leike, and Ryan J. Training language models to follow instructions with human feedback. arXiv, 2022.
Lowe.
[60] David A, Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. CoRR, abs/210k 104. 10350, 2021. URL https: //axxiv.// org/ abs 22044 10350.
[61] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. In Association for the Advancement of Artificial Intelligence (AAAT), 2021.
[62] Barbara Plank, Anders Johannsen, and Anders Søgaard. Importance weighting and unsupervised domain adaptation of POS taggers: a negative result. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 968 (1889973, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-11 1104. URL https: http:::/aaaaatthologe wrg.gy.......ooooe.... org/oo///o D-44/11411111 1104.
[63] John Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
[64] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019. 2019..
[65] Yasaman Razeghi, Robert Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv, 2022.
[66] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv . org/ abs/1908.. 10084.
[67] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. ArXiv, abs/2006. 12204, 2020.
[68] Donald B. Rubin. Using the SIR algorithm to simulate posterior distributions. Bayesian Statistics, 1988.
[69] Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with Bayesian optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 372-382, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.11653/ URL ttttps://acaanthool/ htt.eannnoooooooooo////-------
[70] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2015.
[71] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations (ICLR), 2018.
[72] Burr Settles. Active learning. Synthesis lectures on artificial intelligence and machine learning, 6, 2012.
[73] Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning (CMML),002.
15

---

[74] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the oog-likelhhod function. Journal of Statistical Planning and Inference. 90:227-244, 2000,
[75] Chris Snyder, Thomas Bengtsson, Peter Bickel, and Jeff Anderson. Obstacles high-dimensional particle filtering. Mathematical Advances in Data Assimilation (MADA), 2008.
[76] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv, 2022.
[77] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Compu- lational Linguistics, pages 3645-3631 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.1833/v1//199 1355. URI.ttpss:/ / ac aclannhologyy wwg/p19--111111 1355.
[78] Masashi Sugiyama, Matthias Krauledat, and Roburt Muller. Covariate shift adaptation by importance weighted eross validation. Journal of Machine Learning Research (JMLR), 8: 985-1005, 2007.
[79] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation machine learning. 2012.
[80] Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learning helps pretrained models learn the intended task, 2022.
[81] Tim Vieira. Gumbel-max trick and weighted reservoir sampling, 2014.
[82] Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019.
[83] Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In International Conference on Machine Learning (ICML), 2020.
[84] Xudong Wang, Long Lian, and Stella X Yu. Unsupervised selective labeling for more effective semi-supervised learning. In European Conference on Computer Vision, pages 427-445. Springer, 2022.
[85] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning (ICML), 2015.
(86] Kilian Weinherger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenherg. Feature hashing for large scale multitask learning. In International Conference on Machine Learning (ICML), 2009.
[87] Sang Michael Xie and Stefano Ermon. Reparameterizable subset sampling via continuous relaxations. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[88] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[89] Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NLP from scratch without large-scale pretraining: A simple and efficient framework. International Conference on Machine Learning (ICML), 2022.
[90] Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. Cold-start active learning through self-supervised language modeling. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online, November URL
2020. Association for Computational Linguistics. doi: 10. https://acclattholl/
20rg/2200-.. emnlp-main. 637.
[91] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in Neural Information Processing Systems (NeurIPS), pages 9054-9065, 2019.
16

---

[92] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text
classification, In Advances in Neural Information Processing Systems (NeurIPS), 2015.
17

---

A DSIR asymptotically selects from the target
We prove that DSIR selects examples with features distributed as the target as the raw dataset size goes to infinity, assuming that the importance weights are correct up constant factor.
Proposition A.1. Assume that the importance weights are proportional to the true importance weights P Then as the number of raw examples N goes infinity, the procedure returns i...d. samples with features distributed according to the target feature distribution Pfeat-
Proof. By assumption, we have importance weights wi that are proportional to the true importance weights, that w, C Piaales for the i-th source example for some constant C >>> O. First suppose that == Then,
(ux(2))
>
siml 1 2/W;
Prob. of sampling example with feature value
(3)
> 2j=1 W/
1= 1 1[2 =
N
Pice
geeat
(4)
Σ f=1 C pto (21)
1= 1 12 = Pize (2.)
(5)
Proe (ay
Oloar (zy)
For 2 1, we can similarly compute the probability of sampling the m-th example (m E: kt) as:
N-m++1
Pizal ( (2.)
22 2 Oiza (#:)
Prob. of sampling m-th example with feature value 2 m
(6)
+++1 Plus (zn)
Toar (zy)
where for notational convenience, we re-index the raw examples after selecting each example.
For each m E {L.....k), the numerator converges Ptear(2) N +000
N N-m+1
Preat (Zi)
N-m++ m++1
Prea (2)
Piea(())
Σ
1
2
- Peeat
Preat (2)
N m
Gent (zi)
N 1-++1
Vent (z)
Qieat (z)
(7)
since 2j (raw features) are sampled from Heat (raw feature distribution). For the same reason, the denominator converges to 1:
N N-m+1
Σ
Pient (zz)
- Eegias
Pient (zz)
=
(8)
N-m+1
Neea (2z)
Oeeat 2z)
Therefore the features the mn-h example sampled from Pient for all m (1,...,k).
Intuition from a simple example. DSIR uses importance resampling to better balance the tradeoff between relevance and diversity the samples converge to samples from the target distribution (Proposition A. while no such guarantee holds for top-k selection. For intuition using a simple example, consider a raw dataset We want to filter the raw dataset to have k = 10 flips from fair coin (the target distribution). The importance weights are 2.0.9 _ for the heads examples and 1 for the tails examples (the tails have
coin flips from biased coin, with 0.9n heads and 0.1n tails.
2-0.T
higher weight). If we select the top 1 flips according to the importance weight, we will select 10 tails, still resulting biased dataset. However, importance resampling balances this out, resulting in fair dataset in expectation as goes to infinity. We ran a simulation of the simple example with k= 10 and varying raw data sizes n to see how fast the resampled dataset converges to a fair dataset with the raw data size. For raw data sizes E (100,20 200,200,500 DSIR selects a dataset with (44%, 47%, 50%) heads respectively, averaged over 1000 trials. Thus, DSIR converges quickly to the desired target distribution. In all cases, top-k selects dataset with all tails.
18

---

Table 4: Accuracies on the GLUE [82] dev set for stRe masked language model [17] trained on data selected from The Pile [21]. Following RoBERTa [48], for RTE, STS, and MRPC we tune-tune starting from the MNLI model instead of from scratch. DSIR outperforms heuristic classification (used by GPT-3 and PaLM) and random selection by over 2% on average. All results are averaged over seeds and standard deviations are in subscripts.
MNII
QNLI
QQP
RTE
SST-2
MRPC
Col.A
STS-B
Avg
Random selection
82.639.41
86, 86.90p
89.57g.50
67.37
90.05%.41
87.40,
49.41g.57
88.63,11
80.25
Heuristic classificttio
82, 69..7
85.950.79
89.77a.52
68.591.75
88.94u.3
86.03..3
48.17a:3
88.62o.22
79.85
Top-A Heuristic classfication
83.34p.22
88.62p24
89.8%,.1
70.04pm9
91.15
86.37_.0
53.02a.50
89.30,,1
DSIR Top-k DSIR
83.07
89.11
89.8 89.80
75.09
90.48 90.488.57
87.70..6s
54,00
89.17
82.30
88 63
89.94..7
72.49
91.01..7
49. 49,90,.30
89.52..2
DSIR + Neural Teatures
83.44, 16
88 30
89.8 19.8
70.68g
90.50
87.55
52.58 o7
88, 1.,00
81.40
Hourrate casa casalicatioo
0SB
12-
Figure 5: Distribution of Pile data sources for datasets selected by Left: Random selection Middle: Heuristic classification and Right: DSIR. Heuristic classification and DSIR were restricted to select only 4% of its dataset from Wikipedia, Books3, BookCorpus2, and Gutenberg.
B DSIR with neural importance weight estimator
As preliminary study, we test an instantiation of DSIR with an importance weight estimator based on neural features. For each example, we extract embeddings from a (all fall-MiniM--6-- with dimension 384. We fit the generative models for the source and target feature space with 1000- and 50-component Gaussian mixture models respectively, with diagonal covariance structure. We use this to select pretraining data for training general-domain LMs from scratch. Table shows the results using this DSIR variant the last row. On average, DSIR with neural features improves by 1-1.5%+ over random selection and heuristic classification and is on par with top-k heuristic classification and top-k DSIR, but still underperforms DSIR with n-gram features. However, believe that many aspects of this preliminary pipeline could be improved or redesigned, and that using a neural model the importance weight estimator is a promising direction.
Transformer [66]
C Distribution of data sources for general-domain training
Figure shows the distribution of data sources (ArXiv, GitHub, News, etc.) from The Pile that were selected by random selection, heuristic classification, and DSIR. Heuristic classification and DSIR aim to select formal text that are similar to text from Wikipedia or books. Note that we restricted heuristic classification and DSIR select from data sources outside of Wikipedia and books sources (Books3, BookCorpus2, Gutenberg) for 96% of the dataset, while 2% is randomly selected from Wikipedia and the remaining 2% are selected from the book sources. DSIR seems to focus mostly selecting formal text from web data such as Pile-CC (which can still be quite varied), while the other methods select from a variety of sources.
D Distribution of data sources for continued pretraining
Figure shows the distribution of Pile data sources selected by DSIR for different target distributions. Each of the 4 columns represents a domain: CS papers, Biomedical text, News, and Reviews. The
19

---

Figure 6: Distribution of Pile data sources selected by DSIR for different target distributions, The four columns from left to right represent domains: CS papers, Biomedical text, News, and Reviews.
Average Downstream ream F1
83
CS: ACL-ARC
CS: SciERC
82
Biomed: ChemProt
Biomed: RCT
81
News: HyperPartisan
80
News: AGNews
Reviews: Helpfulness
79
Reviews: IMDB
-0.20 -0.15
-0.10
-0.05 0.00
Average KL reduction (KL(target) || random) KL(target selected))
Figure 7: Plot of KL reduction against average downstream F1 score of DSIR for the 8 continued pretraining datasets, where each point represents a different pretraining dataset (selected for a particular target). Pretraining data selected for CS-related target distributions tend to transfer well to datasets in other domains, while pretraining data selected for reviews transfers poorly.
distribution of data sources for target distributions from the same domain similar. When the target is a task from the CS domain, the distribution of data sources is the most diverse. Biomedical and news domains selected examples are from PubMed Abstracts and PubMed Central, and when the target is from the news domain, most of the selected examples are from web data (Pile-CC and Open Debnee)T)xt2),
particularly different; when the target is from the biomedical domain, most of the
Cross-domain KL reduction. Figure 7 plots the KL reduction against average downstream Fl for all datasets selected by DSIR (one dataset for each target). KL reduction is still a strong indicator of downstream performance in this case. Data selected using CS papers tend to transfer the best to other domains, while data selected using reviews hurts performance. This also shows that transfer between domains very asymmetric.
Figure 6, we show that the distribution of data sources selected by DSIR for CS targets is generally the most diverse, which could contributes to its strong performance on many domains. Intuitively, ACL-ARC (a dataset NL. papers) likely to contain more diverse set of topics than reviews.
20

---

Table 5: Continued pretraining results on the GLUE dev set when the target distribution is formal text. DSIR improves average GLUE performance by 0.4-0.7% over all baselines. All fine-tuning results are averaged over seeds. Following RoBERTa [48], for RTE, STS, and MRPC we fine-tune starting from the MNII model instead of from scratch.
MNLI
QNLI
QQP
RTE
SST-2
MRPC
CoLA
STS-B
Ave
BERT-base continued pretrain)
84,29
91 91.26..1s
90.23u..
s6.39
92.34u.51
86.422.m
56.36,.e
90.1
83.43
Random selection
83.82g.48
862.s
90.47u.30
26.03
92. 00.0s
87.211.4
59 39.002.07
90.320.3t
83.59
Heuristic classification
84.033.5a
90.47
90.460.50
76.751.4
880.44
86.03
56.034.22
90.30
83.24
DSIR
84.21
90.78,
90.45
18.34
92.0% 59
87.16
58.41s.s:
90.49%
83,99
E
Continued pretraining results when target is formal text
We also consider using the same datasets for continued pretraining, starting from the public BERT-base checkpoint. Here, all data selection methods improve over BERT-base on the GLUE dev set. Similarly to training from scratch, we find that heuristic classification slightly decreases performance compared to random selection (by 0.2% on average). DSIR improves over random selection by 0.4% and over BERT-base by 0.6%, achieving almost 84% on the GLUE dev set.
F
Data selection details
Data preprocessing. We select data from The Pile [21], which comes in 30 random chunks. We reserve chunk for validation purposes and only consider the last 29 chunks. We first divided the documents in The Pile into chunks of 128 "words", according to whitespace tokenization. These chunks define the examples that we do data selection on, totaling 1.7B examples. For heuristic classification and DSIR, we first apply a manual quality filter (Appendix J) and only consider the examples that pass the filter. Random selection selects from the unfiltered Pile.
Heuristic classification. We use bigram fasttext classification model [30], which first forms list of unigrams and bigrams, hashes them into a predefined number of tokens (2M this case), maps these tokens into learned feature vectors, and then learns a logistic regression model on top of averaged feature vectors across the model. We initialize the feature vectors from 300 dimensional pretrained subword fasttext vectors trained from Common Crawl. We use the fasttext hyperparameter autotuning functionality with a duration timeout of 30 minutes.
The classification model is trained on a balanced dataset of examples from The Pile validation set and examples from the target distribution (downstream unlabeled training inputs or Wikipedia/book text from The Pile validation set). We downsample the larger dataset of the two to create the balanced dataset. Each example is lowercased and stripped of newlines by first tokenizing using the NLTK word tokenizer and rejoining the words with spaces.
select raw example with probability P: predicted by the fasttext model if p: > 1 1 33, where {3; is sampled from a Pareto distribution with shape parameter 9. If the number of examples that do not cross the threshold is smaller than the desired number of examples k. then we repeat this process on the examples that were not chosen and continue to add to the dataset. After we have chosen at least & examples, we take i random samples without replacement from the chosen examples.
For noisy thresholding,
For top-k heuristic classification, we simply take the examples with the top-k predicted probabilities p:-
Importance resampling. Our importance resampling-be based methods use bag-of-words generative model text. We process each example by lowercasing and splitting into words using the WordPunct tokenizer from NLTK [7]. Following [30], we incorporate unigram and bigram information by hashing the unigrams and bigrams into 10k buckets, which defines a vocabulary of 10k "words" for the generative model. Both unigrams and bigrams are hashed into the same space of words. We learn two bag-of-worde models, one for the target and one for The Pile, using target data (downstream unlabeled training inputs or Wikipedia/book text from The Pile validation set) and Pile validation data. The parameters of the models are learned by simply counting the word frequencies across the dataset.
For unigram-based DSIR, we use the RoBERTa tokenizer [17], which allows us to avoid hashing. With bigrams, this is more difficult since we must consider 50000 pairs of tokens the RoBERTa vocabulary. Still, even in the unigram case we find that there are often tokens that are never seen in
21

---

Table 6: Hyperparameters for training general-domain LMs from scratch.
Architecture
BERT-base
Max token length
128
Batch size
4096
Learning rate
le-3 or 8e-4
Learning rate schedule
Linear
Weight decay
0.01
Warmup steps
3000
Total steps
50000
Optimizer
AdamW
Adam $1
0.9
Adam 32
0.999
Adam
le-8
GPUs
Titan RTX
Table 7: Hyperparameters for continued pretraining of general-domain LMs.
Architecture
BERT-base
Max token length
512
Batch size
2048
Learning rate
le-4
Learning rate schedule
Linear
Weight decay
0.01
Warmup steps
1440
Total steps
25000
Optimizer
AdamW
Adam 31
0,9
Adam $2
0.999
Adam
1e-8
GPUs
4 Titan RTX
the target dataset, so we smooth the MLE parameters by mixing with the uniform distribution over tokens with weight of 1e-5.
Implementation of importance resampling. We implement importance resampling with the Gumbel top-k trick [81; 38; 87; 39], which produces samples without replacement according to the softmax distribution of the given scores. In the Gumbel top-k procedure, we add IID standard Gumbel noise y; to each log-importance weight to produce score 8; =;= log (
(sx + 9t for each raw example.
We select the examples corresponding to the top k scores. Note that producing the log-likelihood ratios and adding independent Gumbel noise them can be trivially parallelized, and selecting top can be done in linear time with the introselect algorithm [57], implemented by numpy. argpart tittiiiii.....ii
Sampling data for coneral-domain LMs. To select dataset that is suitable for both pretraining from scratch token length 128 and continued pretraining with token length 512, we choose to first select 102.4M examples then concatenate every two examples to create 51.20 51.2M examples. This ensures token length of 512 without much padding. We train
that the examples are long enough for the importance weight estimator or fasttext classifier from The Pile validation set, where the target is Wikipedia+ BookCorpus2 Gutenberg Books3 and the raw data come from the rest of the data sources in The Pile. We first select 98.4M examples from non Wikipedia and book data, then randomly select 2M from Wikipedia and 0.66M each from BookCorpus2, Gutenberg, and Books3. We mix in some examples from Wikipedia and books to balance the distribution of sources and to reduce catastrophic forgetting in continued pretraining. After this, concatenate every two examples.
Details for ablations. We ablate top-k heuristic classification in Section 5 in two ways. First, we consider the original heuristic classification method, which takes classifier probabilities p: = (x,) for example and selects the example if p > 1 B, where 8 is Pareto random variable. Sccond,
22

---

Table 8: -ptaset-specific hyperparameters for fine-tuning LMs on GLUE, following best
hyperparameters from RoBERTa [48].
Epochs
Batch size
Learning rate
Continue from MNLI?
MNLI
10
32
le-5
RTE
10
16
2e-5
Y
MRPC
10
16
le-5
Y Y N
STSB
10
16
2e-5
COLA
10
16
le-5
QQP
10
32
le-5
NNN
SST2
10
32
le-5
N
QNLI
10
32
le-5
N
we consider heuristic classification with importance resampling by first calibrating the classifier's probabilities with Platt scaling [63] against validation set, then using the calibrated probabilities Ps to compute the importance weight log 1-p. Similarly to DSIR, we use the Gumbel top-k trick to select a subset using these importance weights.
Pi
We ablate the DSIR approach by replacing the generative importance weight estimator with a discrim- inative one. We use the same hashing method and define the features 10k-dimensional counts of the n-grams. We normalize each count vector to sum to 1. On top of these features, we train logistic regression classifier using the same dataset used train the fasttext classifier in heuristic classification. We tune an L2 regularization weight based on best held-out accuracy (we further split the validation set in half to create another held out set) in the binary classification task. Similarly as above, we calibrate the probabilities using Platt scaling and use the classifier probabilities to compute the importance weight.
G Training details for training general-domain LMs
Pretraining from seratch. Table 6 shows the hyperparameters for training general-domain LMs learning rate 1e-3. We use 8e-4 for DSIR since
from scratch. For all models except DSIR, we found that 1e-3 leads to divergence the training. We use 16 accumulation steps with 4 GPUs to achieve large batch size of 4096, following Izsak et al. [28]. Our hyperparameters result in compute budget of 26B tokens processed (128 x 4096 x 50000). Each training run takes about 50 hours. Our pretraining implementation is adapted from Yao et al. [89].
Continued pretraining (Appendix E). Table shows the hyperparameters for continued pretraining general-domain LMs. We continue pretraining from the BERT-base [17] checkpoint. During BERT training, they process 43B tokens. We process 26B tokens during training so that the total compute after continued pretraining is 69B tokens. Each continued pretraining takes about 60 hours.
Fine-tuning on GLUE.
We follow the hyperparameters used by RoBERTa [48] for fine-tuning on just the
GLUE (Tables 8 and 9). While RoBERTa searches over space of hyprparraetees,, hyperparameters set for each task from the RoBERTa code base. The fine-tuning for RTE, MRPC, and STSB continues from the fine-tuned model for MNLI, following Liu et al. [48]. We the default HuggingFace code for GLUE fine-tuning.
H Training details for continued pretraining of domain-specific LMs
Pretraining. Table 10 shows the hyperparameters for continued pretraining domain commain-specife speeific LMs. We choose the pretraining compute budget to equal the number of tokens processed in the DAPT models from Gururangan al. [24]. For all models, we first try pretraining with learning rate 5e-4, and if training diverges, we use 1e-4.
Fine-tuning. Table shows the hyperparameters for fine-tuning on domain-specific datasets. We use the fine-tuning code from Gururangan et al. [24] and follow their fine-tuning protocols. For datasets from CS/Biomed/News domains, we use a max token length of 256 to match pretraining length. For Reviews (IMDB and Helpfulness) datasets, use max token length of 12 since this
23

---

Table Shared hyperparameters for fine-tuning LMs on GLUE, following Liu al. [48].
Architecture
BERT-base 128 (from scratch) or 512 (continued pretrain)
Max length
Weight decay
0.1
Optimizer
AdamW
Adam 81
0.9
Adam 82
0.98
Adam
1c-6
Warmup ratio
0.06
LR schedule
Polynomial
Precision
FP16
GPUs
Titan RTX
Table 10: Hyperparameters for continued pretraining on domain-specific data.
Architecture
RoBERTa-bass
Max token length
256
Total steps
12500
Batch size
4096
Weight decay
0.01
Adam 31
0.9
Adam 33
0,999
Adam
1e-8
Warmup steps
720
LR schedule
Linear
Learning rate
5e-4 or le-4
GPUs
4 T'taan RTX
Table 11: Hyperparameters for fine-tuning on domain data.
Architecture
RoBERTa-baie
Max token length
256 or 512
Epochs
3 10
Patience
3 epochs
Batch size
4096
Weight decay
0.1
Optimizer
Adam W
Adam 31
0.9
Adam 32
0.98
Adam
le-6
Warmup ratio
0.06
LR schedule
Linear
GPUs
T'tan RTX
seems to change performance significantly. For DAPT models [24], we use max token length of
512 for all datasets, which matches their protocol. Following Gururangan et al. [24], we choose either
3 or 10 epochs based on average validation performance over seeds. Our fine-tuning implementation follows Gururangan et al. [24].
I Computing the KL reduction metric
To compute the reduction metric for a particular dataset, we took the first 100k examples from the dataset and computed the hashed n-gram counts. Normalizing these counts gives an MLE estimate of the hashed n- n-gram distribution for the dataset. We use the same procedure to compute the hashed n-gram distribution parameters for The Pile (from the Pile validation set).

---

For manual curation (DAPT), we attempted to download the datasets used in the paper (RealNews [91 l], S2ORC [49], and Amazon reviews [25]). However, Gururangan et [24] uses an internal version of S2ORC that cannot be released. We approximate S2ORC for CS papers and Biomed by using the first 100k documents in the public version of S2ORC that contain 'Computer Science' and 'Medicine" as metadata field, respectively.
For RoBERTa, we approximate the pretraining distribution by computing the hashed n-gram distribution from Wikipedia and books data in the Pile validation set.
J
Quality filter
For heuristic classification and IS methods, we devise a few hand-crafted ways to filter out low quality data preprocessing step, according
Word length: between 40 and 500
Repeat ratio, defined as max
Poceureecees word example,
between 0.02 and 0.2
example word length
Informativeness ratio, defined as
#o non-stepwerds and non-punctuation in example. example word length
between 0.3 and 0.7
Numeric ratio, defined
o numbers in example, example word longth
less than 0.2
The words are based on the NLTK word tokenizer [7]. These are difficult for simple n-gram based importance weight estimator or classifier to use as features because it requires global context. We decide to keep vs. discard examples using some simple thresholds on the above values, decided using inspection the Pile validation set. Below, we detail some statistics of the quality filtering procedure and provide some data examples.
Statistics of quality filtering.
With the above thresholds, find that:
The length filter is the most selective after applying the length filter, only 55% of the examples are left.
The repeat ratio filter keeps 78% of the data.
The informativeness filter keeps 72% of the data.
The numeric filter keeps 91% of the data.
Overall, when applying all the filters at the same time. 52% of the examples are kept. Thus, we are mainly filtering by length, which seems like good proxy for quality.
Kept vs. discarded examples according quality filter. First, show the beginning characters from some randomly selected kept vs. discarded examples.
XEPT:
all rights, , the girl should be hanged for coining and thievery, and you, sir, millennia of ancient carvings, magical awords and glittering jewels and textilea. Xmax, mean asphericity Q) on corneal tomography were evaluated [M] other selects therapist who requires co-pay in
inf informations about call once you are done and you don't need info anymore
DISCARDED;
(31)
SUCH DAMAGE, And
Am 1993--1997 48 months var value formattedTime + ' type At + name + >\ eventTxt +
J,An An
"setup": An
[nn
NA (\)55 yr age)
FILEDAn
110.88 (108.42 to 113.34) 107.89 (105.28 to 110.50) 1.25 (-2.18 to 4.67)
Wye Mun no podia evitar recordar lo que su padre siempre decia: <<luuca olvides 2.18
25

---

Extreme length examples.
Very short examples tend to be data snippets or otherwise nonsensical:
278713.30 303 3771574. 556 by are centered at 280828. 793 3766437.
94a to 279188. 184 3771745. 314 by are centered 280945 177 3766474. to 280325.499 3771995. 774 by are centered
281478.555 3766560. 741 to
Very long examples tend to be dense code, repetitive, or very technical:
y 1 Aeap. <A Vin Azatheal (d) (y) Asetazinus \ (C.0 ) 8 the cube couplex E 0_0"- \eap 2 is called vertebra
rigurea [ifig: pantagesa))
(vertabra)) (-..87 --3.17 restangls (6.57,5.42) (0,0)-- (0.1))
(0,0)- (1,0);; (1,1) (1,0) ) (1,1) (1,1.56) (00.77.11771
(0.85,1.77 i plotl [
variabier)) ) (C1) 0.711 + cos (x) +01 0.771 + sin (x) 1.) [00 0.722 e cos (I) +11 0.711 ain ( 11)
plath [ (6:5iin4477:555. variable-) ((1) 0.755 a coa (x) +0) 0.71% A ain (+) (0)() [0) 0.711 coa (x) (x)+1)44 0.771
(x)) ploc) (dosain=-0.79) (x)
variables) ((1) 0.75%
(2) +0) 0.71) aai (x)) [00 0.711
0.71% + sin (x) : plot) Ldsmiin
variable- J (T x 0.15) 4 coa (x) +0% x 0.155 aai {))
[ov 0.115 + cos (x) +1) + 0.155 + sin (F))() ()) plots [ (.c0sinr3.88:4.71 variables) ] [[-1+0077
(x)
[0% (r)
(+)
(+))(): plati (drrain4.71:5.55
variable-) ([-V) 0.71%) cos (x) +0) 0.71) sin (x) (0. + cos (x) +1 (l)
plots [ [cmsian-0779:0 variable=) ] ( [-1) 0.72) ces (z)+00) 0..71 sin ()) 0.77 coa -1) -11
0.71) xin (r)] ; [ (.006i0881122:4.77
veriable-)] (( ([-1) 0.115) c (r) 4..15) (r)
[oy 0.15)
(x)
(x))
(2.0) (1,1)
1.229.1.71) (1.16,1.77) ploc) [400011=3.98:4.75 vartablee)
([- 0.71 (x) +0) (x)
[d) 0.71) * ena (r)
+1) 0.711 sin (r) (}) plott [=0mmin44 variable J
[11 0.71 cos (+) +01
0.71) + sin (x)]()
( 0.71% ces (x) +1 0.71% sin (=)
variable-) ] (111 ()) ploi) (
(2) +0% 00.71) (z))
00.11 (x)
(x)) cox (x) +.115
1o2::n.3.142::.71. veriable ([- 0.15) cos (1) (0V
* * (x) (}) ploc) dossaim-3.9::4.777 variablee) (z) C..71 ain ala (x))()
[0) 0..71 (x))(). , 4
(r) 0.71) (r) plot) [d00min vartable)]] 0.77) coa (+)
0.77
( p(t a (x) plot)
(braiis--00 variable-)
([1) 0.711 coa (x) +0) 0.71) (1.
[ (42:41 varisblee)) (I (r) (r)
can
(3,1)- + con (+)+
It 0.155 + sis (x) ()); (4.0)-(4.1) (4.0) (3.1)
(3.29.1.71)
sariablee)] 1 [[ ([-1) 0.71) cas (r) +0) 0.711 (2.7.711)) (2.85.1.11 plot)
(8.15.1.71)
(2.0))((8))) (3,0) (2,1)
(r)](),
6.77 con (r 0.71% ain (+) ());
plany [ [cmmin-4471:555
veriable- ([-1) 6.77 con +61 4.777 aia (1) 0.71) cm +1)+
0.71V + (r) ()) plott [onntt variable= ] (C 0.71 cor 00.711 (r)
[ov 0.71% cas (r) 0.711
(11())
varisllee)
cou (x)+ +0) 0.15) A sin (+) (o) 0.15) ona (1)+)) 0.15; sin (z) plock [dmmine3.93:4477
varsable (X 0.71V cas (r) +01 0.713 sin (r)l(), [ + 0o7 (
0.711 ain sin ain
(r)1())
plati [ (éomiin44
variable-) 1 (1 1 0.71% con (+) +0) 0.711
()) 0.770
ona (r) -1)-1)
0.71% (x) ()))))))) ploot [dominn
varisble-) ([1)
(x) +0) (x))
[d) 0.71) con (T) +1) 0.711 sin ( plot) doonttr3.
cos (T)+
d 0.155) a ain (r) (). [.) + 0.15) + coa (+) +1
cas (T) 100-00++0000 0.711 ean (r)]() (r)++)+ 0.15) 0.115 ain
(F) ()) plas) [drrain33.9::4771
vartable=) ] ( ([I+ veriable-)
I0 cos (T) 0.711 o.7 (1)
plat) Leomstn44.7:55..
0.711 con (r)
0.711 sin (r)(),, 0..71 (r) 0.711
ain (+)]()) plett) 7.00 variable=) ( [1) 0.71) ena (r) +01 0.71% xin (r) ( [ 0.71%
cos + ein ()) plot) [ (0002123.116224.7 : plat) [d 95:4..11 variabler)]
variable= ( 0.16% cos 40
sin (T)]().. 000 + 0.151 con (r) 0.115 sin (r)]
variable=) [ ([-1) ()) ())
([-I)+ 0.711, plot) I comstm44.72:5.5
(r) 0.711
(1)
(r)]
coe
0.77) coe (t)+
IV + 0.71% + ain (r)(()) plat) [ (dorais=-0)79:00 variable=) [[-I)+ . cna (r) +0) +0) 0.711 xin (})
[ol 0.711
(r) +1)
()) plot: (000si033.14::..11 vartable=) ([--) 0..1) 505 (x)
10 0.15%
(r)1() [ 0.156 coe (x) 41) o.1 (r) ()) (0,0) (
(7,0)
(T,1) (01)((7.0) (7,0) : (7,1) (7,1.58 (7.20...1) -
(6,0) (8.1 ( (6,0)- - (7,0) (7,1)-
(T,155) ; (6.71.1.11
(6.06,1.71 (4,0) (6,0);
(5,0) ; (6,1) (5,1.66 ; (4.71,1771-
(4.8.1171 (6,00 (6,0); (6.1 (5.1.66 : (5.28..1171 (6.15.1.71 plot) 63341133.93:4.7
variable=]] (I-1)
(r) 0.7 (r)1
0.711
(2)
Extreme repeat ratio example. Examples with a high repeat ratio are mostly examples without much content except for one repeated token, sometimes within code:
$ d (x, y+leett)) )
$
s Aeet ( (-iinfty, \frac { 3'delta } { }
+ (Neps)deltaariin x__0, y) $
* look forward to one day becoming mother 870- 130 -0.312 0.066 28.
871
Extreme informativeness ratio examples. (sometimes extremely) short:
Very low informative ratio examples also tend to be
26

---

Extremely high informativeness examples often foreign language, since they don' 't contain English stop words:
maillmankaikteuden sinnikkäimmällä valittajahahmolle umayya abu-hannalla on taas sanottavaa niin tiedätte kyllå mistä suomalaisten ra-sis-mis-ta abu-hannaa haastatellaan päivän helsingin sanomissa ehkä ziljoonannen kerran tästä yhdestä ja samasta aiheesta en ymmärrä miksi abu-hanna on ollut tällä viikolla suomessa noutamassa global family award -palkintoa avarakatseisuudesta ja monikulttuurissuude merkittävästä edistämisestä suomalaisessa yhteiskunnassa " avarakatseiisudeetaa
tiedă ketään jonka katsantokanta kuin abu-hannan hänen toistuvat suuri ова suomalaisista on rasisteja "
julkisen kuvan perusteella niin kapea
puheenvuoronsa eivät myöskään synnytä minkäänlaista positiivista dialogia tähän yhteiskuntaan miistä tykkäävät ja somessa peukuttavat ainoastaan ihmiset jotka ovat itsekin sitä mieltă, ettå suomi raakalaismaisten rasistien maa muissa suomalaisissa ne herättävät kohtuuttomuudessan ja jankkauksessaan vain ärtymystä kuten kaikki tiedänme abu-hanna asuu nykyään hollannissa vielä vuosi sitten kyseinen maa näyttäytyi hänen haastatteluissaa paratiisina mutta nyt tämä ei varmasti yllätä ketään
Examples with informative ratio close 0.5
standard English:
the feature (2) mentioned above the assumption of $ p 1 might be unrealistic in usual ferromagnetic metals however if the exchange interaction between atoms is accomplished via $ \pi $ -bands of 60 $ discussed earlier polarization $ Api -electrons we can also consider the effect of magnetic polaron in magnetic semiconductors such as eu chalcogenides carrier makes surrounding magnetic moments be polarized via exchange interaction and forms magnetic polaron field magnetic polarons have move with flipping some magnetic moments which are more or less randomly oriented and their conduction is suppressed application of magnetic field aligns spin directions and carriers become mobile as result negative magnetoresistance occurs the negative magnetoresistance above t_c can be attributed this
expect large spin
kasuyai
27


Reviews:
[{"review_id": "sxK4CQ6Dxy", "reviewer": "Reviewer_LSWj", "summary": "The authors begin by highlighting an important issue related to data quality in (encoder only) LMs, motivating the need for an improved way to filter large datasets (e.g. The Pile) for samples which are in distribution to a held out target sample. The authors propose a metric, KL-reduction, which quantifies how well matching a target distribution signals downstream performance after fine-tuning on the data. To sample in-distribution points, they compute a importance weight for each sample, by learning a generative distribution for the raw data and target data. Finally they re-sample data using the weights without replacement. Beyond relying on the KL-reduction, the authors finetune RoBERTa on their filtered data (compared to other baseline filtering/sampling methods) and measure performance improvements on the downstream task, showing several percentage point improvements.", "questions": "* TF-IDF (or bm25), i.e. a simple retrieval system, could be used to retrieve documents relevant to the target set sufficient. Line 263 references experiments but the results should be included. Related to this, if a bm25 approach will yield many duplicate documents, why can’t they just be deduplicated? And why does the proposed approach not yield duplicates?\n\n* How are the distributions (q(x) and p(x)) on lines 91/92 computed? Elsewhere it is described as a generative distribution, but is it simply parameterized by counting n-grams in the respective datasets? \n", "limitations": "The discussion of limitations is adequate.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Paper is well written and easy to understand; motivation and set up are clear.\n* The described KL-reduction is a simple metric which appears to strongly correlate with downstream performance after fine-tuning, saving time and compute costs.\n* Paper shows strong performance improvements on existing datasets/baselines using the proposed strategy\n", "weaknesses": "* The authors only run experiments using Encoder only models, this is in line with a 2021 paper which their benchmarks (data + a model) are based on. However given the recent attention to decoder only models, and the shown impact data quality has on pretraining (see llama / red pajama), an obvious question is if decoder only models also benefit from this low cost strategy. Including decoder/generative experiments may increase the impact of the method.\n* The task shares similarities with active learning, that should include be included somewhere in the related work section.\n* The authors only consider n-gram based features, it’s mentioned in the limitations but it likely wouldn’t take much to run the experiments with other features. On Line 173, the authors mention the potential for BERT embeddings, but there are no related experiments. Clearly, the use of just n-grams is effective, but why are n-grams the best way to determine if something is ‘in distribution’ with a target text?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. LSWj notes that the paper is **“highlighting an important issue” and “shows strong performance improvements on existing datasets/baselines”**. We answer specific questions below:\n\n> “The authors only consider n-gram based features, it’s mentioned in the limitations but it likely wouldn’t take much to run the experiments with other features”\n\n**During the rebuttal period, we implemented a first-pass version of DSIR with embeddings from a pretrained language model.** We extracted features using a Sentence Transformer (miniLM-v6-2), learned raw and target distributions over the features parameterized as 1000 and 50-component Gaussian mixture models respectively, and used these within the DSIR framework to select data for general-domain pretraining. The results are shown below in a table. **On average, DSIR with neural features improves by 1-1.5%+ over random selection and heuristic classification** and is on par with top-k heuristic classification and top-k DSIR, but still underperforms DSIR with n-gram features. However, we believe that this is still a promising direction since some steps could be improved, such as the hyperparameters of the Gaussian mixture model. We thank the reviewer for the suggestion.\n\n| GLUE dev                       |  MNLI |  QNLI |   QQP |   RTE | SST-2 |  MRPC |  CoLA | STS-B |      Avg |\n|--------------------------------|------:|------:|------:|------:|------:|------:|------:|------:|---------:|\n| Random selection               | 82.63 |  86.9 | 89.57 | 67.37 | 90.05 |  87.4 | 49.41 | 88.63 |   80.25 |\n| Heuristic classification       | 82.69 | 85.95 | 89.77 | 68.59 | 88.94 | 86.03 | 48.17 | 88.62 |   79.85 |\n| Top-k heuristic classification | 83.34 | 88.62 | 89.89 | 70.04 | 91.15 | 86.37 | 53.02 |  89.3 | 81.47 |\n| Top-k DSIR                     | 83.39 | 88.63 | 89.94 | 72.49 | 91.01 | 86.18 |  49.9 | 89.52 |  81.38 |\n| DSIR + n-gram features                          | 83.07 | 89.11 |  89.8 | 75.09 | 90.48 |  87.7 |    54 | 89.17 |  82.30 |\n| DSIR + neural features         | 83.44 |  88.2 | 89.81 | 70.68 |  90.5 | 87.55 | 52.58 |  88.4 |   81.40 |\n\n> “Retrieval methods: Line 263 references experiments but the results should be included. “\n\n- Line 263 refers to small preliminary data selection tests using BM25 retrieval methods, where we found that when selecting for with AGNews as the target, **out of 6.1M documents retrieved by BM25, there were only 1.8M unique documents (70% were exact duplicates).** We will add these results in the final revision.\n- Since heuristic classification is the method that is used to filter large datasets such as the Pile and the data for GPT3 and PaLM and there is a close similarity between heuristic classification (which uses an inner product score between pretrained word embeddings and a learned vector) and retrieval, we mainly conducted full comparisons against heuristic classification.\n\n> “if a bm25 approach will yield many duplicate documents, why can’t they just be deduplicated? And why does the proposed approach not yield duplicates?”\n\n**Although BM25 retrieved data could be deduplicated, this results in less control over the number of selected examples.** In comparison, DSIR will return exactly the number of examples requested, avoids choosing the exact same document by sampling without replacement, and naturally handles deduplication of repeated documents in the raw data via importance resampling.\n\nIn detail, DSIR more gracefully handles duplicates through 2 mechanisms: \n- **1) DSIR avoids sampling the same document by sampling without replacement.** BM25 may select the same document multiple times from different query strings.\n- **2) DSIR naturally decreases the probability to sample an example proportionally to how much it is duplicated in the raw data.** For instance, an example x that is duplicated 1000 times in the raw data will have 1000 times higher probability p(x) under the raw distribution. This reduces the importance weight by a factor of 1000 (since p(x) is in the denominator). \n\n> “ Including decoder/generative experiments may increase the impact of the method”\n\nDue to compute limitations, we weren’t able to pretrain a decoder-only model that is large enough for meaningful few-shot/generative evaluations, but we would like to do so in the future. We thank the reviewer for the suggestion and will add it to the discussion in the final revision.\n\n> “The task shares similarities with active learning, that should include be included somewhere in the related work “\n\n\n**We agree and had already included some active learning works** in the related work (e.g., https://arxiv.org/abs/1901.01151, https://arxiv.org/abs/1708.00489), but we will add more and make the reference to active learning more explicit in the final revision. We thank the reviewer for the suggestion.\n\n> “How are the distributions (q(x) and p(x)) on lines 91/92 computed? … is it simply parameterized by counting n-grams in the respective datasets?”\n\nYes, they are computed by counting n-gram frequencies. This allows DSIR with n-gram features to be cheap to run on large raw datasets.\n"}, {"review_id": "rmJGMXunQY", "reviewer": "Reviewer_NB9k", "summary": "This paper proposes a simple method for selecting pretraining data for language modeling based on the downstream fine-tuning tasks. It uses ngrams as features for a corpus, and weighs the pretraining data based on importance sampling, which estimates how similar the pretraining data is to the task specific fine-tuning data.", "questions": "1. Have you considered comparing to learned data selection method such as DoReMi(https://arxiv.org/pdf/2305.10429.pdf)?\n", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method is simple and easy to understand. The presentation of the paper is generally easy to follow\n2. The paper tested the method on both domain adaptive continued pretraining and training from scratch\n", "weaknesses": "1. The paper uses a simple n-gram feature for importance weighting, but it’s not clear whether other dense features would bring better performance. Even if using dense features is more expensive or unrealistic, it would be helpful to have a comparison on the small scale, or analysis of the differences in computation required for using different features.\n2. The paper did not compare to other automatic data selection methods such as\n3. The method does seem like a simple/scalable approach, but there is not a good analysis on the resources(memory/training time) needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Reviewer NB9k notes that the method “seems like a simple/scalable approach” and the paper included comprehensive experimental settings, as it “tested the method on both domain adaptive continued pretraining and training from scratch”. We answer specific questions below:\n\n> “Even if using dense features is more expensive or unrealistic, it would be helpful to have a comparison on the small scale”\n\n**During the rebuttal period, we implemented a first-pass version of DSIR with embeddings from a pretrained language model.** We extracted features using a Sentence Transformer (miniLM-v6-2), learned raw and target distributions over the features parameterized as 1000 and 50-component Gaussian mixture models respectively, and used these within the DSIR framework to select data for general-domain pretraining. The results are shown below in a table. **On average, DSIR with neural features improves by 1-1.5%+ over random selection and heuristic classification** and is on par with top-k heuristic classification and top-k DSIR, but still underperforms DSIR with n-gram features. However, we believe that this is still a promising direction since some steps could be improved, such as the hyperparameters of the Gaussian mixture model. We thank the reviewer for the suggestion.\n\n\n| GLUE dev                       |  MNLI |  QNLI |   QQP |   RTE | SST-2 |  MRPC |  CoLA | STS-B |      Avg |\n|--------------------------------|------:|------:|------:|------:|------:|------:|------:|------:|---------:|\n| Random selection               | 82.63 |  86.9 | 89.57 | 67.37 | 90.05 |  87.4 | 49.41 | 88.63 |   80.25 |\n| Heuristic classification       | 82.69 | 85.95 | 89.77 | 68.59 | 88.94 | 86.03 | 48.17 | 88.62 |   79.85 |\n| Top-k heuristic classification | 83.34 | 88.62 | 89.89 | 70.04 | 91.15 | 86.37 | 53.02 |  89.3 | 81.47 |\n| Top-k DSIR                     | 83.39 | 88.63 | 89.94 | 72.49 | 91.01 | 86.18 |  49.9 | 89.52 |  81.38 |\n| DSIR + n-gram features                          | 83.07 | 89.11 |  89.8 | 75.09 | 90.48 |  87.7 |    54 | 89.17 |  82.30 |\n| DSIR + neural features         | 83.44 |  88.2 | 89.81 | 70.68 |  90.5 | 87.55 | 52.58 |  88.4 |   81.40 |\n\n\n> “there is not a good analysis on the resources(memory/training time) needed”, and it would be helpful to have an “analysis of the differences in computation required for using different features.”\n\n- **DSIR is compute and memory efficient due to the simple n-gram featurization.** The memory usage is constant (keep track of 10k n-gram frequency counts), and it takes about 38 minutes to compute n-gram counts on the Pile (2B examples) using 30 workers.\n- **Comparison between n-gram and neural features: using neural embeddings requires at least 138M times more FLOPs+integer ops than the n-gram featurization, if a small 23M parameter neural embedding model.** This is mainly due to the FLOPs needed to run the forward pass of the neural embedding model vs. counting n-grams. **Due to the compute costs, it takes about 16 hours to extract neural embeddings from the Pile with 30 workers with 1 GPU each (23M model), vs 38 minutes with n-gram features** \n- We thank the reviewer for the comment and will add more discussion of the compute resources needed in the final revision.\n\n> “Have you considered comparing to learned data selection method such as DoReMi(https://arxiv.org/pdf/2305.10429.pdf)?”\n\n- DSIR and DoReMi tackle different data curation problems. **DoReMi does not select data for a target distribution and can only reweight the data on the coarser level of domains** (instead of example level selection). Instead, DoReMi tries to find a reweighting that is robust to many target distributions via minimax optimization. Although DoReMi could be applicable to the general-domain experiments, scaling up DoReMi to example level selection (where domain = 1 example) would require some fundamental changes to DoReMi, since it currently depends on seeing all/most domains in the minibatch to keep the domain weights updated.\n- We also note that **DoReMi was also released on arXiv after the NeurIPS deadline**. "}, {"review_id": "UbJZNas6ch", "reviewer": "Reviewer_rytZ", "summary": "The authors present a novel framework for selecting examples from a large and diverse dataset which are most relevant to a specific target domain. They apply this towards data selection for 'continued pretraining' in which a pretrained LM is trained further on a domain-specific dataset, in order to improve its ultimate downstream performance on domain-relevant tasks. They also define a novel data-evaluation metric which is cheap to calculate and correlates well with downstream task performance. They evaluate the proposed method on continued training for RoBERTa against competing methods for 8 tasks, and demonstrate superior average performance. They present additional experiments ablating the target domain dataset, and demonstrate the utility of the method towards data selection for a 'general domain' LM (via heuristic of formality).", "questions": "In Table 3, the strong performance of the CS-dataset-target models on non-target domains, especially compared to the relatively poor performance of the Reviews datasets, shows asymmetry in the transfer between domains (line 225). This is an interesting finding and could bear slightly more explanation. \n\nIn the discussion, it is mentioned (line 319) that amplifying biases in the target data is a potential harm. On the other hand, the possibility of mitigating bias might confer a tremendous benefit; what do the authors think of the potential to use DSIR as a means of removing bias from a giant dataset?", "limitations": "The limitations of the work and potential negative societal impact are sufficiently addressed.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper was a pleasure to read. The overall problem addressed in the paper is important and especially relevant given the preponderance of general LLMs in contemporary research. The relative tractability of this the proposed method at scale makes it more pragmatic than some competing approaches. The paper is clearly written and well structured, each section flows nicely from the last. The results outperform extant methods, and are thoroughly and engagingly analyzed. The proposed method provides a tractable and approachable means of solving a quite general problem, as well as presents many compelling directions for future work in a similar direction.", "weaknesses": "DSIR is general to the kind of feature extractor in use, yet in this work only hashed n-gram and unigram feature are explored. It would be instructive as to demonstrate DSIR with different feature spaces. Naively I would assume this is a more interesting ablation than discriminative vs generative approach to learning the importance weight estimator, as this also might demonstrate a tradeoff of performance and efficiency. However, n-grams are simple and a very reasonable place to start. This does not significantly detract from the overall work, and this omission is rightly mentioned in the limitations as being left for future work. \n\nNits:\nFigure 3 and Table 3 cover the same data, ideally would be presented on the same page.\nTable 2 and its analysis (lines 199-212) are also on different pages.\nFigure 3 references the overall avg performance of the values reported in Table 3, please add that as a final column to the Table itself. Additionally, please add a separate row showing the performance of the baseline model with no continued training.\nTable 3 might be made more readable (depending on execution) by color-coding the relative performance each dataset. As presented, it is very difficult to parse that ChemProt is stronger on avg than Sci-ERC, that IMDB is weak on average, or that IMDB on HyperPartisan is such a negative outlier. Additionally, it might be useful to standardize the column widths and center the text so the main diagonal is more obviously a diagonal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Reviewer rytZ felt that the problem is **“important and especially relevant”** and the paper provides a **“tractable and approachable means of solving a quite general problem, as well as presents many compelling directions for future work”**. We answer specific questions below:\n\n> DSIR is general to the kind of feature extractor in use, yet in this work only hashed n-gram and unigram feature are explored. It would be instructive as to demonstrate DSIR with different feature spaces.\n\n**During the rebuttal period, we implemented a first-pass version of DSIR with embeddings from a pretrained language model.** We extracted features using a Sentence Transformer (miniLM-v6-2), learned raw and target distributions over the features parameterized as 1000 and 50-component Gaussian mixture models respectively, and used these within the DSIR framework to select data for general-domain pretraining. The results are shown below in a table. **On average, DSIR with neural features improves by 1-1.5%+ over random selection and heuristic classification** and is on par with top-k heuristic classification and top-k DSIR, but still underperforms DSIR with n-gram features. However, we believe that this is still a promising direction since some steps could be improved, such as the hyperparameters of the Gaussian mixture model. We thank the reviewer for the suggestion.\n\n\n| GLUE dev                       |  MNLI |  QNLI |   QQP |   RTE | SST-2 |  MRPC |  CoLA | STS-B |      Avg |\n|--------------------------------|------:|------:|------:|------:|------:|------:|------:|------:|---------:|\n| Random selection               | 82.63 |  86.9 | 89.57 | 67.37 | 90.05 |  87.4 | 49.41 | 88.63 |   80.25 |\n| Heuristic classification       | 82.69 | 85.95 | 89.77 | 68.59 | 88.94 | 86.03 | 48.17 | 88.62 |   79.85 |\n| Top-k heuristic classification | 83.34 | 88.62 | 89.89 | 70.04 | 91.15 | 86.37 | 53.02 |  89.3 | 81.47 |\n| Top-k DSIR                     | 83.39 | 88.63 | 89.94 | 72.49 | 91.01 | 86.18 |  49.9 | 89.52 |  81.38 |\n| DSIR + n-gram features                          | 83.07 | 89.11 |  89.8 | 75.09 | 90.48 |  87.7 |    54 | 89.17 |  82.30 |\n| DSIR + neural features         | 83.44 |  88.2 | 89.81 | 70.68 |  90.5 | 87.55 | 52.58 |  88.4 |   81.40 |\n\n\n> In Table 3, the strong performance of the CS-dataset-target models on non-target domains, especially compared to the relatively poor performance of the Reviews datasets, shows asymmetry in the transfer between domains (line 225). This is an interesting finding and could bear slightly more explanation.\n\nIn Appendix Figure 6, we show that the distribution of data sources selected by DSIR for CS-dataset targets is generally much more diverse, and we believe this could be a reason for its strong performance on many domains. This might make sense since ACL-ARC (a dataset of NLP papers) is likely to be on a more diverse set of topics than reviews. \n\n> the possibility of mitigating bias might confer a tremendous benefit; what do the authors think of the potential to use DSIR as a means of removing bias from a giant dataset?\n\n**DSIR has potential for removing dataset bias by collecting a target dataset with less bias** and using it to define the target distribution. For example, if the target dataset has roughly equal representation across groups, DSIR will upweight minority groups and downweight majority groups accordingly. However, since the user interface is simply to collect a target dataset (and no groups have to be explicitly defined), more intricate patterns about what makes the target dataset less biased can also be taken into account. We thank the reviewer for the great idea!\n\n> Nits / presentation of tables and figures\n\nWe will update these in the final revision - we thank the reviewer for the suggestions.\n"}, {"review_id": "q7U2Wb9WB8", "reviewer": "Reviewer_znTv", "summary": "The work presents a novel framework for effectively selecting a representative document subset during the pre-training of Large Language Models. Pre-training such models incurs significant costs, prompting efforts to minimize the subset size and associated expenses. The authors commence by highlighting the importance of the subject matter and its practical applications. Subsequently, they identify the limitations and characteristics of existing approaches. Building upon the recognized limitations and compelling evidence of Importance Resampling's applicability in the context of Large Models, the authors propose an innovative framework based on KL reduction.\n\nIn general, the paper has a good idea and a good novelty factor. The authors claim they improved the state-of-the-art on the text classification task, and, indeed, there is strong evidence by the results presented in the paper. In sum, my unique improvement suggestion is to include statistical treatment of the presented results. \n\nIn summary, this research exhibits a commendable goal and innovative ideas, demonstrating substantial potential. With minor changes addressed, I recommend accepting this paper. I would like to extend my congratulations to the authors for their extensive experimental work and the promising results they have achieved.\n", "questions": "Q1: Overall, the paper demonstrates a strong conceptual framework with a commendable novelty factor. However, there is room for improvement, particularly in addressing the previously mentioned weakness.\n\nRegarding this weakness, it is worth noting that the authors have already conducted a 5-Fold random validation procedure. As a result, incorporating a statistical method to strengthen the research claims would require minimal additional effort. In light of this, I highly recommend applying a t-test with Bonferroni correction to account for multiple tests. [1,2]\n\n[1] Dacrema, M. F., Cremonesi, P., & Jannach, D. (2019). Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM conference on recommender systems (pp. 101–109).\n\n[2] Cunha, Washington, et al. \"On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study.\" Information Processing & Management 58.3 (2021): 102481\n", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "S1: The paper is well written. The authors clearly define the evaluated objectives, motivation, and contributions.\n\nS2: The implementation details and method-specific hyperparameters were defined. Thus, the paper is (possibly) reproducible. Besides, the authors shared their code as supplementary material with the submission in the OpenReview.\n\nS3: The considered datasets are well-known and widely used in the literature. ACL, Sci-ERC, ChemProt, RCT, AGNews, HyperPartisan, Helpfulness, and IMDB. \n\nS4: The proposed method was fairly compared to strong baselines. Besides, in terms of text classification methods, the proposed method was applied to RoBERTa, a strong SOTA method in the LLM field. Moreover, the baselines were very well explained in a simple and straightforward way.\n\nS5: The authors adopted proper metrics to handle and properly measure the effectiveness on both balanced and unbalanced datasets domains (Accuracy and Macro-F1, respectively).\n\nS6: Repetition: It is worth noting that the authors adopted a 5-Fold validation procedure, which demonstrates their commitment to rigorously evaluating the proposed framework.\n", "weaknesses": "W1: There is no statistical treatment of the results (e.g., statistical significance tests), which does not allow to rule out the null hypothesis of equality of results. There are strong evidences of the superiority. However, without tests, any claim of superiority can be considered unsubstantiated. To strengthen the research claims and ensure robust conclusions, it is desirable to include appropriate statistical analyses to validate the significance of the reported results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Overall, znTv notes that we provide **“a strong conceptual framework with a commendable novelty factor”, along with “extensive experimental work and promising results”**. We respond to specific questions below:\n\n> “There are strong evidences of the superiority… To strengthen the research claims and ensure robust conclusions, it is desirable to include appropriate statistical analyses to validate the significance of the reported results… I highly recommend applying a t-test with Bonferroni correction to account for multiple tests.”\n\nThe statistical tests below show that **DSIR outperforms all other methods with significance (α=0.05) in general-domain pretraining and is comparable with manual expert curation / top-k heuristic classification in domain-specific continued pretraining while outperforming all other baselines with significance. This is in line with what is claimed in the paper.** We conducted Student-t tests (unequal variances) on the average performance of the methods across downstream datasets, with a Bonferroni correction for multiple comparisons between DSIR and other methods. We thank the reviewer for the suggestion and will include it in the final revision.\n\nFor general-domain pretraining, with p-value threshold 0.05 / 3 = 0.0125: \n\n|                             | t-score     | p-value         |\n|-----------------------------|-------------|-----------------|\n| DSIR vs random              | 9.271188391 | 0               |\n| DSIR vs heuristic cls       |  10.8322228 | 0               |\n| DSIR vs top-k heuristic cls | 3.716599729 | 0.0001 |\n\nFor domain-specific continued pretraining, with p-value threshold 0.05 / 5 = 0.01: \n\n|                             | t-score      | p-value            |\n|-----------------------------|--------------|--------------------|\n| DSIR vs RoBERTa             |  5.064870983 | 0.0000002|\n| DSIR vs manual curation     |  0.816252706 |       0.207 |\n| DSIR vs random              |  3.199222061 |    0.0006 |\n| DSIR vs Heuristic cls       |  2.987762708 |     0.0014 |\n| DSIR vs Top-k heuristic cls | 0.2570449689 |       0.398 |\n"}, {"review_id": "q0hvUvSzsH", "reviewer": "Reviewer_LQ7V", "summary": "This paper aims to improve the samples to pre-train language models. \nIt propose a novel method called \"Data Selection with Importance Resampling (DSIR)\" to select better pre-training samples and found superior fine-tuning performance compared to various other approaches, including random-sampling. \nThe authors show the effectiveness of this approach by reporting 2-2.5% better average performance on the GLUE dataset.", "questions": "- How do you see your approach connected to curriculum learning, where we gradually increase the difficulties during pre-training?\n- Do you think more recent learning paradigms like in-context learning or prompting can gain in the same way as fine-tuning does?", "limitations": "Limitations are addressed in the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper addresses a crucial issue \"how can we make pre-training more efficient?\" \nIt is well written, easy to follow, covers relevant related work, and considers various other methods. \nTheir approach effectively improves performance when fine-tuning models on downstream tasks. ", "weaknesses": "- While show performance improvement, this paper considers GLUE a rather old, general language benchmark. Therefore, this paper evaluates only a part of the possible tasks. Especially in these times when we see the big success of larger language models on such general tasks using in-context learning, it would be interesting to see how models perform on other more challenging tasks - like reasoning intense ones. \n- This paper only reports the GLUE performance on the dev set. While we see an improvement, it remains to be seen how this transfer to the test set. This is especially relevant since we do not know whether the authors use the dev set to find the best-performing epoch. From the paper, we know that RoBERTa default hyperparameters were used - which includes early stopping on the dev set - but in this paper nothing is stated regarding this issue. \n- The authors report that domain-specific pre-training heavily harms downstream tasks' performance in the worst case. This also reflects a crucial limitation of this approach; when pre-training a language model, we aim for an optimal model in general without having specific downstream tasks/domains in mind. But with this work, we introduce new dependence to the target domain, which only sometimes matches the full richness of existing and upcoming language properties/domains. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank LQ7v for the feedback. Overall, LQ7v felt that the paper was “well written” and “addresses a crucial issue”. We answer specific questions below:\n\n> “we do not know whether the authors use the (GLUE) dev set to find the best-performing epoch. From the paper, we know that RoBERTa default hyperparameters were used - which includes early stopping on the dev set - but in this paper nothing is stated regarding this issue.”\n\n**To clarify, we do not use the dev set for tuning hyperparameters and instead use a fixed set of hyperparameters** for each GLUE dataset. By the RoBERTa default hyperparameters, we mean that we copied and fixed the default hyperparameters for learning rate, batch size, and number of epochs provided in the RoBERTa official codebase, without using any of their additional tuning strategies. We apologize for any confusion and will clarify in the final revision.\n\n>  “this paper considers GLUE a rather old, general language benchmark…it would be interesting to see how models perform on other more challenging tasks - like reasoning intense ones.”\n\nDue to compute limitations, we weren’t able to pretrain a model that is large enough for reasoning capabilities from scratch for the general-domain experiments. However, we agree that challenging reasoning tasks are a great direction for future evaluation of data curation methods on large LMs and we would like to do this in the future. We will add this in the discussion for the final revision.\n\n> “when pre-training a language model, we aim for an optimal model in general without having specific downstream tasks/domains in mind. But with this work, we introduce new dependence to the target domain…”\n\n- In general, the target distribution is a hyperparameter that can be tuned to be as general as we desire. For example **in the general-domain experiments we show that even a heuristic/proxy target distribution (Wikipedia and books) can improve performance on general downstream benchmarks.** For general-domain models, the task then becomes to design a good target distribution for high-quality data, and then apply DSIR. \n- For **domain-specific models, we often have a target distribution of interest** (e.g., code, law, medical) and DSIR can be applied directly to gather more relevant data for this target distribution.\n\n> Do you think more recent learning paradigms like in-context learning or prompting can gain in the same way as fine-tuning does?\n\n- **Yes.** For example, the GLaM paper (https://arxiv.org/abs/2112.06905) shows that **even heuristic classification (filtering the pretraining data with a Wikipedia classifier) can significantly improve few-shot in-context learning** performance. We believe **DSIR is a more principled approach and would bring further gains** at these large scales.\n- There are also some works that show that in-context learning benefits from selecting the few-shot examples themselves intelligently (https://aclanthology.org/2022.emnlp-main.622/, https://arxiv.org/abs/2302.13539, https://arxiv.org/pdf/2307.07164.pdf, https://arxiv.org/abs/2101.06804, https://arxiv.org/abs/2301.11916).   \n- Another related area is selecting good examples for RLHF / instruction tuning of large models. The target distribution here could roughly be described as “helpful answers to a diverse set of prompts”. DSIR could be used to scale up the amount of good instruction tuning data for RLHF.\n\n> How do you see your approach connected to curriculum learning, where we gradually increase the difficulties during pre-training?\n\nCurriculum learning could be viewed within the DSIR framework as selecting data against a set of target distributions that changes over the course of training, starting with preferring easier examples and progressing to harder ones.\n"}]

Author rebuttal:

Meta-review: In sum, reviewers were positive on this paper, with two arguing strongly for acceptance and others closer to the borderline.  One primary concern brought up in the less positive reviews was that the authors only experimented with n-gram based features for data selection.  To address this, the author response included preliminary experiments with dense representations, a helpful data point (although the approach did not show improvements over the ngram approach in the paper).  Incorporating these in the final paper and addressing the other reviewer comments would improve the work.  One additional suggestion: while the related work notes potential shortcomings of Moore-Lewis selection, an experiment evaluating against this classical approach seems like it would strengthen the work.
Decision: Accept (poster)