SlowGuess/NeurIPS_2023_OCR huggingface dataset:
distribution of data sources for target distributions from the same domain are similar. When the target is a task from the CS domain, the distribution of data sources is the most diverse. Biomedical and news domains are particularly different; when the target is from the biomedical domain, most of the selected examples are from PubMed Abstracts and PubMed Central, and when the target is from the news domain, most of the selected examples are from web data (Pile-CC and OpenWebText2).
Cross-domain KL reduction. Figure 7 plots the KL reduction against average downstream F1 for all datasets selected by DSIR (one dataset for each target). KL reduction is still a strong indicator of downstream performance in this case. Data selected using CS papers tend to transfer the best to other domains, while data selected using reviews hurts performance. This also shows that transfer between domains is very asymmetric. In Figure 6, we show that the distribution of data sources selected by DSIR for CS targets is generally the most diverse, which could contribute to its strong performance on many domains. Intuitively, ACL-ARC (a dataset of NLP papers) is likely to contain a more diverse set of topics than reviews.
Table 5: Continued pretraining results on the GLUE dev set when the target distribution is formal text. DSIR improves average GLUE performance by  $0.4 - 0.7\%$  over all baselines. All fine-tuning results are averaged over 5 seeds. Following RoBERTa [48], for RTE, STS, and MRPC we fine-tune starting from the MNLI model instead of from scratch.
MNLI	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B	Avg
BERT-base (no continued pretrain)	84.290.41	91.260.16	90.230.06	76.393.80	92.340.34	86.422.49	56.361.49	90.110.23	83.43
Random selection	83.820.48	89.860.63	90.470.39	76.032.20	92.000.31	87.211.47	59.002.57	90.320.17	83.59
Heuristic classification	84.030.33	90.470.65	90.460.36	76.751.74	91.880.42	86.030.78	56.034.22	90.300.22	83.24
DSIR	84.210.47	90.780.42	90.450.39	78.341.75	92.090.59	87.160.77	58.415.86	90.490.19	83.99
E Continued pretraining results when target is formal text
We also consider using the same datasets for continued pretraining, starting from the public BERT-base checkpoint. Here, all data selection methods improve over BERT-base on the GLUE dev set. Similarly to training from scratch, we find that heuristic classification slightly decreases performance compared to random selection (by  $0.2\%$  on average). DSIR improves over random selection by  $0.4\%$  and over BERT-base by  $0.6\%$ , achieving almost  $84\%$  on the GLUE dev set.
F Data selection details
Data preprocessing. We select data from The Pile [21], which comes in 30 random chunks. We reserve chunk 0 for validation purposes and only consider the last 29 chunks. We first divided the documents in The Pile into chunks of 128 "words", according to whitespace tokenization. These chunks define the examples that we do data selection on, totaling 1.7B examples. For heuristic classification and DSIR, we first apply a manual quality filter (Appendix J) and only consider the examples that pass the filter. Random selection selects from the unfiltered Pile.
Heuristic classification. We use a bigram fasttext classification model [30], which first forms a list of unigrams and bigrams, hashes them into a predefined number of tokens (2M in this case), maps these tokens into learned feature vectors, and then learns a logistic regression model on top of averaged feature vectors across the model. We initialize the feature vectors from 300 dimensional pretrained subword fasttext vectors trained from Common Crawl. We use the fasttext hyperparameter autotuning functionality with a duration timeout of 30 minutes.
The classification model is trained on a balanced dataset of examples from The Pile validation set and examples from the target distribution (downstream unlabeled training inputs or Wikipedia/book text from The Pile validation set). We downsample the larger dataset of the two to create the balanced dataset. Each example is lowercased and stripped of newlines by first tokenizing using the NLTK word tokenizer and rejoining the words with spaces.
For noisy thresholding, we select a raw example with probability  $\rho_{i}$  predicted by the fasttext model if  $\rho_{i} > 1 - \beta_{i}$ , where  $\beta_{i}$  is sampled from a Pareto distribution with shape parameter 9. If the number of examples that do not cross the threshold is smaller than the desired number of examples  $k$ , then we repeat this process on the examples that were not chosen and continue to add to the dataset. After we have chosen at least  $k$  examples, we take  $k$  random samples without replacement from the chosen examples.
For top-  $k$  heuristic classification, we simply take the examples with the top-  $k$  predicted probabilities  $\rho_{i}$
Importance resampling. Our importance resampling-based methods use a bag-of-words generative model of text. We process each example by lowercasing and splitting into words using the WordPunct tokenizer from NLTK [7]. Following [30], we incorporate unigram and bigram information by hashing the unigrams and bigrams into 10k buckets, which defines a vocabulary of 10k "words" for the generative model. Both unigrams and bigrams are hashed into the same space of words. We learn two bag-of-words models, one for the target and one for The Pile, using target data (downstream unlabeled training inputs or Wikipedia/book text from The Pile validation set) and Pile validation data. The parameters of the models are learned by simply counting the word frequencies across the dataset.
For unigram-based DSIR, we use the RoBERTa tokenizer [17], which allows us to avoid hashing. With bigrams, this is more difficult since we must consider  $50000^2$  pairs of tokens in the RoBERTa vocabulary. Still, even in the unigram case we find that there are often tokens that are never seen in
Table 6: Hyperparameters for training general-domain LMs from scratch.
Architecture	BERT-base
Max token length	128
Batch size	4096
Learning rate	1e-3 or 8e-4
Learning rate schedule	Linear
Weight decay	0.01
Warmup steps	3000
Total steps	50000
Optimizer	AdamW
Adam β1	0.9
Adam β2	0.999
Adam ε	1e-8
GPUs	4 Titan RTX
Table 7: Hyperparameters for continued pretraining of general-domain LMs.
Architecture	BERT-base
Max token length	512
Batch size	2048
Learning rate	1e-4
Learning rate schedule	Linear
Weight decay	0.01
Warmup steps	1440
Total steps	25000
Optimizer	AdamW
Adam β1	0.9
Adam β2	0.999
Adam ε	1e-8
GPUs	4 Titan RTX
the target dataset, so we smooth the MLE parameters by mixing with the uniform distribution over tokens with a weight of 1e-5.
Implementation of importance resampling. We implement importance resampling with the Gumbel top- $k$  trick [81; 38; 87; 39], which produces  $k$  samples without replacement according to the softmax distribution of the given scores. In the Gumbel top- $k$  procedure, we add IID standard Gumbel noise  $g_{i}$  to each log-importance weight to produce a score  $s_{i} = \log \frac{\hat{p}_{\mathrm{feat}}(z_{i})}{\hat{q}_{\mathrm{feat}}(z_{i})} + g_{i}$  for each raw example. We select the examples corresponding to the top  $k$  scores. Note that producing the log-likelihood ratios and adding independent Gumbel noise to them can be trivially parallelized, and selecting top  $k$  can be done in linear time with the introselect algorithm [57], implemented by numpy .argpartition.
Sampling data for general-domain LMs. To select a dataset that is suitable for both pretraining from scratch at token length 128 and continued pretraining with token length 512, we choose to first select 102.4M examples then concatenate every two examples to create 51.2M examples. This ensures that the examples are long enough for a max token length of 512 without much padding. We train the importance weight estimator or fasttext classifier from The Pile validation set, where the target is Wikipedia + BookCorpus2 + Gutenberg + Books3 and the raw data come from the rest of the data sources in The Pile. We first select 98.4M examples from non-Wikipedia and book data, then randomly select 2M from Wikipedia and 0.66M each from BookCorpus2, Gutenberg, and Books3. We mix in some examples from Wikipedia and books to balance the distribution of sources and to reduce catastrophic forgetting in continued pretraining. After this, we concatenate every two examples.
Details for ablations. We ablate top- $k$  heuristic classification in Section 5 in two ways. First, we consider the original heuristic classification method, which takes classifier probabilities  $\rho_{i} = f(x_{i})$  for an example and selects the example if  $\rho_{i} > 1 - \beta_{i}$  where  $\beta_{i}$  is a Pareto random variable. Second,
Table 8: Dataset-specific hyperparameters for fine-tuning LMs on GLUE, following best hyperparameters from RoBERTa [48].
Epochs	Batch size	Learning rate	Continue from MNLI?
MNLI	10	32	1e-5	N
RTE	10	16	2e-5	Y
MRPC	10	16	1e-5	Y
STSB	10	16	2e-5	Y
COLA	10	16	1e-5	N
QQP	10	32	1e-5	N
SST2	10	32	1e-5	N
QNLI	10	32	1e-5	N
we consider heuristic classification with importance resampling by first calibrating the classifier's probabilities with Platt scaling [63] against a validation set, then using the calibrated probabilities  $\rho_{i}$  to compute the importance weight  $\log \frac{\rho_i}{1 - \rho_i}$ . Similarly to DSIR, we use the Gumbel top- $k$  trick to select a subset using these importance weights.
We ablate the DSIR approach by replacing the generative importance weight estimator with a discriminative one. We use the same hashing method and define the features as 10k-dimensional counts of the n-grams. We normalize each count vector to sum to 1. On top of these features, we train a logistic regression classifier using the same dataset used to train the fasttext classifier in heuristic classification. We tune an L2 regularization weight based on best held-out accuracy (we further split the validation set in half to create another held out set) in the binary classification task. Similarly as above, we calibrate the probabilities using Platt scaling and use the classifier probabilities to compute the importance weight.
G Training details for training general-domain LMs
Pretraining from scratch. Table 6 shows the hyperparameters for training general-domain LMs from scratch. For all models except DSIR, we use learning rate 1e-3. We use 8e-4 for DSIR since we found that 1e-3 leads to divergence in the training. We use 16 accumulation steps with 4 GPUs to achieve a large batch size of 4096, following Izsak et al. [28]. Our hyperparameters result in a compute budget of 26B tokens processed  $(128 \times 4096 \times 50000)$ . Each training run takes about 50 hours. Our pretraining implementation is adapted from Yao et al. [89].
Continued pretraining (Appendix E). Table 7 shows the hyperparameters for continued pretraining general-domain LMs. We continue pretraining from the BERT-base [17] checkpoint. During BERT training, they process 43B tokens. We process 26B tokens during training so that the total compute after continued pretraining is 69B tokens. Each continued pretraining run takes about 60 hours.
Fine-tuning on GLUE. We follow the hyperparameters used by RoBERTa [48] for fine-tuning on GLUE (Tables 8 and 9). While RoBERTa searches over a space of hyperparameters, we just use the hyperparameters set for each task from the RoBERTa code base. The fine-tuning for RTE, MRPC, and STSB continues from the fine-tuned model for MNLI, following Liu et al. [48]. We use the default HuggingFace code for GLUE fine-tuning.
H Training details for continued pretraining of domain-specific LMs
Pretraining. Table 10 shows the hyperparameters for continued pretraining domain-specific LMs. We choose the pretraining compute budget to equal the number of tokens processed in the DAPT models from Gururangan et al. [24]. For all models, we first try pretraining with learning rate 5e-4, and if training diverges, we use 1e-4.
Fine-tuning. Table 11 shows the hyperparameters for fine-tuning on domain-specific datasets. We use the fine-tuning code from Gururangan et al. [24] and follow their fine-tuning protocols. For datasets from CS/Biomed/News domains, we use a max token length of 256 to match the pretraining length. For Reviews (IMDB and Helpfulness) datasets, we use a max token length of 512 since this
Table 9: Shared hyperparameters for fine-tuning LMs on GLUE, following Liu et al. [48].
Architecture	BERT-base
Max length	128 (from scratch) or 512 (continued pretrain)
Weight decay	0.1
Optimizer	AdamW
Adam β1	0.9
Adam β2	0.98
Adam ε	1e-6
Warmup ratio	0.06
LR schedule	Polynomial
Precision	FP16
GPUs	1 Titan RTX
Table 10: Hyperparameters for continued pretraining on domain-specific data.
Architecture	RoBERTa-base
Max token length	256
Total steps	12500
Batch size	4096
Weight decay	0.01
Adam β1	0.9
Adam β2	0.999
Adam ε	1e-8
Warmup steps	720
LR schedule	Linear
Learning rate	5e-4 or 1e-4
GPUs	4 Titan RTX
Table 11: Hyperparameters for fine-tuning on domain-specific data.
Architecture	RoBERTa-base
Max token length	256 or 512
Epochs	3 or 10
Patience	3 epochs
Batch size	4096
Weight decay	0.1
Optimizer	AdamW
Adam β1	0.9
Adam β2	0.98
Adam ε	1e-6
Warmup ratio	0.06
LR schedule	Linear
GPUs	1 Titan RTX
seems to change performance significantly. For DAPT models [24], we use a max token length of 512 for all datasets, which matches their protocol. Following Gururangan et al. [24], we choose either 3 or 10 epochs based on average validation performance over 5 seeds. Our fine-tuning implementation follows Gururangan et al. [24].
I Computing the KL reduction metric
To compute the KL reduction metric for a particular dataset, we took the first 100k examples from the dataset and computed the hashed n-gram counts. Normalizing these counts gives an MLE estimate of the hashed n-gram distribution for the dataset. We use the same procedure to compute the hashed n-gram distribution parameters for The Pile (from the Pile validation set).
For manual curation (DAPT), we attempted to download the datasets used in the paper (RealNews [91], S2ORC [49], and Amazon reviews [25]). However, Gururangan et al. [24] uses an internal version of S2ORC that cannot be released. We approximate S2ORC for CS papers and Biomed by using the first 100k documents in the public version of S2ORC that contain 'Computer Science' and 'Medicine' as a metadata field, respectively.
For RoBERTa, we approximate the pretraining distribution by computing the hashed n-gram distribution from Wikipedia and books data in the Pile validation set.
J Quality filter
For heuristic classification and IS methods, we devise a few hand-crafted ways to filter out low quality data as a preprocessing step, according to
- Word length: between 40 and 500
- Repeat ratio, defined as  $\max_{\text{word}} \frac{\# \text{occurrences of word in example}}{\text{example word length}}$ : between 0.02 and 0.2
- Informativeness ratio, defined as  $\frac{\# \text{ of non-stopwords and non-punctuation in example}}{\text{example word length}}$ : between 0.3 and 0.7
- Numeric ratio, defined as  $\frac{\text{of numbers in example}}{\text{example word length}}$ : less than 0.2
The words are based on the NLTK word tokenizer [7]. These are difficult for a simple n-gram based importance weight estimator or classifier to use as features because it requires global context. We decide to keep vs. discard examples using some simple thresholds on the above values, decided using inspection on the Pile validation set. Below, we detail some statistics of the quality filtering procedure and provide some data examples.
Statistics of quality filtering. With the above thresholds, we find that:
- The length filter is the most selective — after applying the length filter, only  $55\%$  of the examples are left.
- The repeat ratio filter keeps  $78\%$  of the data.
- The informativeness filter keeps  $72\%$  of the data.
- The numeric filter keeps  $91\%$  of the data.
- Overall, when applying all the filters at the same time,  $52\%$  of the examples are kept. Thus, we are mainly filtering by length, which seems like a good proxy for quality.
Kept vs. discarded examples according to quality filter. First, we show the beginning characters from some randomly selected kept vs. discarded examples.
KEPT:
all rights, the girl should be hanged for coining and thievery, and you, sir, millennia of ancient carvings, magical swords and glittering jewels and textiles. Kmax, and mean asphericity ( Q) on corneal tomography were evaluated [M]other selects a therapist who requires co-pay in\n informations about call once you are done and you don't need info anymore
DISCARDED:
(31) \\\\n
SUCH DAMAGE.\n#######
"mpls"\n ],\n
"setup": [\\n
[\n
1993--1997, 48 months
NA（\<5 yr age）
var value = formattedTime + '\backslash t' + type + '\backslash t' + name + '\backslash t' + eventTxt +
FILED\n
110.88 (108.42 to 113.34) 107.89 (105.28 to 110.50) 1.25 (-2.18 to 4.67)
Wye Mun no podia evitar recordar lo que su padre siempre decia: <<Nunca olvides
bG9hdDpsZWZ0021hcmdpbjoycHggNXB4OyBOZXhOLWFsaWduOmNlbRlcjsiPjxhIGhyZWY9ImhO\n
Extreme length examples. Very short examples tend to be data snippets or otherwise nonsensical:
278713.303 3771574.556 by arc centered at 280828.793 3766437.062 94a to 279188.184 3771745.314 by arc centered at 280945.177 3766474.440 to 280325.491 3771995.774 by arc centered at 281478.555 3766560.741 to
Very long examples tend to be dense code, repetitive, or very technical:
$ y ' = \cap_ {h \in \mathfrak{m}athcal \{d\}} (y) \setminus \operatorname*{setminus} \mathtt{g.} \setminus \{h_0 \} \} h^+ = $ . the cube complex $ v = h_0^r - \cap y ^ r $ is called a * verteb
see figures \[ \text{[fig: pentagons]} \] and \[ \text{[vertebrae]} \]. (-4.37, -3.17) rectangle (6.57, 5.42); (0,0) - (0,1);
(0,0) - (1,0); (1,1) - (1,0); (1,1) - (1,1.56); (0.71,1.71) - (0.85,1.71); plot[domain=3.93:4.71,
variable=] \] ([1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +1\*0.71\*sin(r)] { }); 
plot[domain=4.71:5.5, variable=] ([1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +0\*0.71\*sin(r)] { }]; 
plot[domain=-0.79:0, variable=] ([1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +0\*0.71\*sin(r)] { }]; 
plot[domain=3.142:4.71, variable=] ([-1\*0.71\*cos(r) +1\*0.71\*sin(r)] { }, [2,0] - (2,1); (2,0) - (1,0); (1,1) - (1,1.56);
(1.29,1.71) - (1.15,1.71); plot[domain=3.93:4.71, variable=] ([-1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +0\*0.71\*sin(r)] { }]; 
plot[domain=4.71:5.5, variable=] ([-1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +0\*0.71\*sin(r)] { }]; 
plot[domain=3.42:4.71, variable=] ([-1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +1\*0.71\*sin(r)] { }]; 
plot[domain=3.93:4.71, variable=] ([-1\*0.71\*cos(r) +0\*0.71\*sin(r)] { }, [0\*0.71\*cos(r) +0\*0.71\*sin(r)] { }]; 
plot[domain=3.42:4.71, variable=] ([-1\*0.71\*cos(r) +0\*0\.52; (2,0) - (3,0); (2,1) - (2,85,1\.71);
(3,3), - (3,56); (3,3) - (3,56); (3,29,1\.71)
(3,3), - (3,56); (3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(3,29,1\.71)
(4,0) - (5,0); (5,1) - (5,56); (4,8), - (8,1); (8,0) - (8,7);
(7,1) - (7,0); (7,2), - (7,56); (6,0) - (6,6); (6,0) - (6,6); (6,8), - (6,8);
(6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8),
(6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,8), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (6,9), - (5,5), - (5,5).
(5,2) - (5,5).
Extreme repeat ratio example. Examples with a high repeat ratio are mostly examples without much content except for one repeated token, sometimes within code:
$ d_h(x, y+ \delta delta) $$
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
.
...
...
...
...
...
 ...
.
...
.
...
.
...
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
#
....
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
;
-
?
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
* * i look forward to one day becoming a mother * * * * 0 * * . 
* * 871 * * * * 0 * * . * * 870 * * -0.130 -0.312 0.066 -0.286
Extreme informativeness ratio examples. Very low informative ratio examples also tend to be (sometimes extremely) short:
Extremely high informativeness examples are often foreign language, since they don't contain English stop words:
maailmankaikkeuden sinnikkäimmällä valittajahahmolla umayya abu-hannalla on taassanottavaa . niin , tiedätte kyllä mistä : suomalaisien ra-sis-mis-ta .abu-hannaa haastatellaan paivän helsingin sanomissa ehkä ziljoonannen kerrantäyhdestä ja samasta aiheesta . en ymmärrä miksi . abu-hanna on olluttällä viikolla suomessa noutamassa global family award -palkintoa ”avarakatseisuudesta ja monikulttuurisuuden merkittävästä edistämisetäsuomalaisessa yhteiskunnassa”. avarakatseisuudesta??? en tiedä ketään , jonka katsantokanta on ( julkisen kuvan perusteella ) niin kapeakuin abu-hannan . hänen toistuvat ” suuri osa suomalaisista on rasisteja”-puheenvuoronsa eivāt myöskän synnytä minkäänlaista positivista dialogia tähänyhteiskuntaan . niistä tykkävät ja somessa peukuttavat ainoastaan ne ihmiset ,jotka ovat itsekin sitä mieltä , etta suomi on raakalaismaisten rasistien maa .muissa suomalaisissa ne herättävät kohtuuttomuudessaan ja jankkauksessaan vain arytymasta . kuten kaikkietiedämme , abu-hanna asuu nykyään hollannissa . vielävuosi sitten kyseinen maa näytätyi hänen haastatteluissaan paratiisina . muttanyt - ja tämä ei varmasti yllätä ketään -
Examples with informative ratio close to 0.5 are more standard English:
the feature (2) mentioned above . the assumption of $ p $ = 1 might be unrealistic in usual ferromagnetic metals . however, if the exchange interaction between eu atoms is accomplished via $ \pi pi $ -bands of c $ _ {60} $ as discussed earlier, we can expect a large spin polarization of $ \pi pi $ -electrons . we can also consider the effect of magnetic polaron . in magnetic semiconductors such as eu chalcogenides , a carrier makes surrounding magnetic moments be polarized via exchange interaction and forms a magnetic polaron [ @ kasuya1 ] . at zero field , magnetic polarons have to move with flipping some magnetic moments which are more or less randomly oriented , and their conduction is suppressed . application of magnetic field aligns spin directions and carriers become mobile . as a result , negative magnetoresistance occurs . the negative magnetoresistance above $ t_c $ can be attributed to this