SlowGuess/ACL_2020_OCR huggingface dataset:
Table 11 shows which of the strategies for continued pretraining have already been explored in the prior work from the Related Work (§6). As evident from the table, our work compares various strategies as well as their interplay using a pretrained language model trained on a much more heterogeneous pretraining corpus.
A.1 ROBERTA's Pretraining Corpus
ROBERTA was trained on data from BOOKCORPUS (Zhu et al., 2015),<sup>6</sup> WIKIPEDIA,<sup>7</sup> a portion of the CCNEWS dataset (Nagel, 2016),<sup>8</sup> OPENWEB-TEXT corpus of Web content extracted from URLs shared on Reddit (Gokaslan and Cohen, 2019),<sup>9</sup> and a subset of CommonCrawl that it is said to resemble the "story-like" style of WINOGRAD schemas (STORIES; Trinh and Le, 2018).<sup>10</sup>
A.2 State of the Art
In this section, we specify the models achieving state of the art on our tasks. See the caption of
Table 5 for the reported performance of these models. For ACL-ARC, that is SCIBERT (Beltagy et al., 2019), a BERT-base model for trained from scratch on scientific text. For CHEMPROT and SCIERC, that is S2ORC-BERT (Lo et al., 2020), a similar model to SCIBERT. For AGNEWS and IMDB, XLNet-large, a much larger model. For RCT, Cohan et al. (2019). For HYPERPARTISAN, LONGFORMER, a modified Transformer language model for long documents (Beltagy et al., 2020). Thongtan and Phienthrakul (2019) report a higher number (97.42) on IMDB, but they train their word vectors on the test set. Our baseline establishes the first benchmark for the HELPFULNESS dataset.
Preprocessing for DAPT The unlabeled corpus in each domain was pre-processed prior to language model training. Abstracts and body paragraphs from biomedical and computer science articles were used after sentence splitting using scispaCy (Neumann et al., 2019). We used summaries and full text of each news article, and the entire body of review from Amazon reviews. For both news and reviews, we perform sentence splitting using spaCy (Honnibal and Montani, 2017).
Training details for DAPT We train ROBERTA on each domain for 12.5K steps. We focused on matching all the domain dataset sizes (see Table 1) such that each domain is exposed to the same amount of data as for 12.5K steps it is trained for. AMAZON reviews contain more documents, but each is shorter. We used an effective batch size of 2048 through gradient accumulation, as recommended in Liu et al. (2019). See Table 13 for more hyperparameter details.
Training details for TAPT We use the same pretraining hyperparameters as DAPT, but we artificially augmented each dataset for TAPT by randomly masking different tokens across epochs, using the masking probability of 0.15. Each dataset was trained for 100 epochs. For tasks with less than 5K examples, we used a batch size of 256 through gradient accumulation. See Table 13 for more hyperparameter details.
Optimization We used the Adam optimizer (Kingma and Ba, 2015), a linear learning rate scheduler with  $6\%$  warm-up, a maximum learning rate of 0.0005. When we used a batch size of 256, we
$^{6}$ https://github.com/soskek/bookcorpus  
 $^{7}$ https://github.com/google-research/bert  
 $^{8}$ https://github.com/fhamborg/news-please  
 $^{9}$ https://github.com/jcpeterson/openwebtext  
 $^{10}$ https://github.com/tensorflow/models/tree/master/research/lm_commonsense
8354
DAPT Domains (if applicable)	Tasks	Model	DAPT	TAPT	DAPT + TAPT	kNN-TAPT	Curated-TAPT
This Paper	biomedical & computer science papers, news, reviews	8 classification tasks	ROBERTA	✓	✓	✓	✓	✓
Aharoni and Goldberg (2020)	-	NMT	DISTILBERT + Transformer NMT	-	-	-	similar	-
Absentzer et al. (2019)	clinical text	NER, NLI, de-identification	(BIO)BERT	✓	-	-	-	-
Chakrabarty et al. (2019)	opinionated claims from Reddit	claim detection	ULMF1T	✓	✓	-	-	-
Chronopoulou et al. (2019)	-	5 classification tasks	ULMF1T†	-	similar	-	-	-
Han and Eisenstein (2019)	-	NER in historical texts	ELMo, BERT	-	✓	-	-	-
Howard and Ruder (2018)	-	6 classification tasks	ULMF1T	-	✓	-	-	-
Khandelwal et al. (2020)	-	language modeling	Transformer LM	-	-	-	similar	-
Lee et al. (2019)	biomedical papers	NER, QA, relation extraction	BERT	✓	-	-	-	-
Logeswaran et al. (2019)	-	zero-shot entity linking in Wikipedia	BERT	-	✓	-	-	-
Mitra et al. (2020)	-	commonsense QA	BERT	-	✓	-	-	-
Phang et al. (2018)	-	GLUE tasks	ELMo, BERT, GPT	-	✓	-	-	-
Radford et al. (2018)	-	NLI, QA, similarity, classification	GPT	-	similar	-	-	-
Sun et al. (2019)	sentiment, question, topic	7 classification tasks	BERT	✓	✓	-	-	-
Swayamdipta et al. (2019)	-	NER, parsing, classification	ELMo	-	similar	-	-	-
Xu et al. (2019a)	reviews	RC, aspect extract., sentiment classification	BERT	✓	✓	✓	-	-
Xu et al. (2019b)	restaurant reviews, laptop reviews	conversational RC	BERT	✓	✓	-	-	-
Table 11: Overview of prior work across strategies for continued pre-training summarized in Table 10. ULMFiT is pretrained on English Wikipedia; ULMFiT† on English tweets; ELMO on the 1BWORDBENCHMARK (newswire; Chelba et al., 2014); GPT on BOOKCORPUS; BERT on English Wikipedia and BOOKCORPUS. In comparison to these pretraining corpora, ROBERTA's pretraining corpus is substantially more diverse (see Appendix §A.1).
used a maximum learning rate of 0.0001, as recommended in Liu et al. (2019). We observe a high variance in performance between random seeds when fine-tuning ROBERTA to HYPERPARTISAN, because the dataset is extremely small. To produce final results on this task, we discard and resample degenerate seeds. We display the full hyperparameter settings in Table 13.
Implementation Our LM implementation uses the HuggingFace transformers library (Wolf et al., 2019)<sup>11</sup> and PyTorch XLA for TPU compatibility.<sup>12</sup> Each adaptive pretraining experi
iment was performed on a single v3-8 TPU from Google Cloud. $^{13}$  For the text classification tasks, we used AllenNLP (Gardner et al., 2018). Following standard practice (Devlin et al., 2019) we pass the final layer [CLS] token representation to a task-specific feedforward layer for prediction.
C Development Set Results
Adhering to the standards suggested by Dodge et al. (2019) for replication, we report our development set results in Tables 15, 17, and 18.
11https://github.com/huggingface/ transformers
12https://github.com/pytorch/xla
13http://github.com/allenai/ tpu-pretrain
8355
D Analysis of Domain Overlap
In Table 20 we display additional examples that highlight the overlap between IMDB reviews and REALNEWS articles, relevant for analysis in §3.1.
E Analysis of Cross-Domain Masked LM Loss
In Section §3.2, we provide ROBERTA's masked LM loss before and after DAPT. We display cross-domain masked-LM loss in Table 12, where we evaluate masked LM loss on text samples in other domains after performing DAPT.
We observe that the cross-domain masked-LM loss mostly follows our intuition and insights from the paper, i.e. ROBERTA's pretraining corpus and NEWS are closer, and BIOMED to CS (relative to other domains). However, our analysis in §3.1 illustrates that REVIEWS and NEWS also have some similarities. This is supported with the loss of ROBERTA that is adapted to NEWS, calculated on a sample of REVIEWS. However, ROBERTA that is adapted to REVIEWS results in the highest loss for a NEWS sample. This is the case for all domains. One of the properties that distinguishes REVIEWS from all other domains is that its documents are significantly shorter. In general, we find that cross-DAPT masked-LM loss can in some cases be a noisy predictor of domain similarity.
F  $k$ -Nearest Neighbors Data Selection
In Table 21, we display nearest neighbor documents in the BIOMED domain identified by our selection method, on the RCT dataset.
8356
Data Sample Unseen During DAPT
PT	BIOMED	CS	NEWS	REVEWS
DAPT	ROBERTA	1.19	1.32	1.63	1.08	2.10
BIOMED	1.63	0.99	1.63	1.69	2.59
CS	1.82	1.43	1.34	1.92	2.78
NEWS	1.33	1.50	1.82	1.16	2.16
REVEWS	2.07	2.23	2.44	2.27	1.93
Table 12: ROBERTA's (row 1) and domain-adapted ROBERTA's (rows 2-5) masked LM loss on randomly sampled held-out documents from each domain (lower implies a better fit). PT denotes a sample from sources similar to ROBERTA's pretraining corpus. The lowest masked LM for each domain sample is boldfaced.
Computing Infrastructure	Google Cloud v3-8 TPU
Model implementations	https://github.com/allenai/tpu_pretrain
Hyperparameter	Assignment
number of steps	100 epochs (TAPT) or 12.5K steps (DAPT)
batch size	256 or 2058
maximum learning rate	0.0001 or 0.0005
learning rate optimizer	Adam
Adam epsilon	1e-6
Adam beta weights	0.9, 0.98
learning rate scheduler	None or warmup linear
Weight decay	0.01
Warmup proportion	0.06
learning rate decay	linear
Table 13: Hyperparameters for domain- and task- adaptive pretraining.
Computing Infrastructure	Quadro RTX 8000 GPU
Model implementation	https://github.com/allenai/dont-stop-pretraining
Hyperparameter	Assignment
number of epochs	3 or 10
patience	3
batch size	16
learning rate	2e-5
dropout	0.1
feedforward layer	1
feedforward nonlinearity	tanh
classification layer	1
Table 14: Hyperparameters for ROBERTA text classifier.
8357
Domain	Task	ROBERTA	Additional Pretraining Phases
DAPT	TAPT	DAPT + TAPT
BIOMED	CHEMPROT	83.21.4	84.10.5	83.00.6	84.10.5
†RCT	88.10.05	88.50.1	88.30.1	88.50.1
CS	ACL-ARC	71.32.8	73.21.5	73.23.6	78.62.9
SCIERC	83.81.1	88.41.7	85.90.8	88.01.3
NEWS	HYPERPARTISAN	84.01.5	79.13.5	82.73.3	80.82.3
†AGNEWS	94.30.1	94.30.1	94.70.1	94.90.1
REVIEWS	†HELPFULNESS	65.53.4	66.51.4	69.22.4	69.42.1
†IMDB	94.80.1	95.30.1	95.40.1	95.70.2
Table 15: Results on different phases of adaptive pretraining compared to the baseline ROBERTA (col. 1). Our approaches are DAPT (col. 2, §3), TAPT (col. 3, §4), and a combination of both (col. 4). Reported results are development macro- $F_{1}$ , except for CHEMPROT and RCT, for which we report micro- $F_{1}$ , following Beltagy et al. (2019). We report averages across five random seeds, with standard deviations as subscripts.  $\dagger$  indicates high-resource settings. Best task performance is boldfaced. State-of-the-art results we can compare to: CHEMPROT (84.6), RCT (92.9), ACL-ARC (71.0), SCIERC (81.8), HYPERPARTISAN (94.8), AGNEWS (95.5), IMDB (96.2); references in §A.2.
Dom.	Task	ROB.	DAPT	\( \neg \)DAPT
BM	CHEMPROT	83.21.4	84.10.5	80.90.5
\( \dagger \)RCT	88.10.0	88.50.1	87.90.1
CS	ACL-ARC	71.32.8	73.21.5	68.15.4
SCIERC	83.81.1	88.41.7	83.90.9
NEWS	HYP.	84.01.5	79.13.5	71.64.6
\( \dagger \)AGNEWS	94.30.1	94.30.1	94.00.1
REV.	\( \dagger \)HELPFUL.	65.53.4	66.51.4	65.53.0
\( \dagger \)IMDB	94.80.1	95.30.1	93.80.2
Table 16: Development comparison of ROBERTA (ROBA.) and DAPT to adaptation to an irrelevant domain ( $\neg$  DAPT). See §3.3 for our choice of irrelevant domains. Reported results follow the same format as Table 5.
BIOMED	RCT	CHEMPROT
TAPT	88.30.1	83.00.6
Transfer-TAPT	88.00.1 (↓ 0.3)	81.10.5 (↓ 1.9)
NEWS	HYPERPARTISAN	AGNEWS
TAPT	82.73.3	94.70.1
Transfer-TAPT	77.63.6 (↓ 5.1)	94.40.1 (↓ 0.4)
CS	ACL-ARC	SCIERC
TAPT	73.23.6	85.90.8
Transfer-TAPT	74.04.5 (↑1.2)	85.51.1 (↓0.4)
AMAZON reviews	HELPFULNESS	IMDB
TAPT	69.22.4	95.40.1
Transfer-TAPT	65.42.7 (↓3.8)	94.90.1 (↓0.5)
Table 17: Development results for TAPT transferability.
Pretraining	BIOMED RCT-500	NEWS HYPERPARTISAN	REVEWS †IMDB
TAPT	80.51.3	82.73.3	95.40.1
DAPT + TAPT	83.90.3	80.82.3	95.70.2
Curated-TAPT	84.40.3	84.91.9	95.80.1
DAPT + Curated-TAPT	84.50.3	83.13.7	96.00.1
Table 18: Mean development set macro-  ${F}_{1}$  (for HYPERPARTISAN and IMDB) and micro-  ${F}_{1}$  (for RCT-500),with Curated-TAPT across five random seeds, with standard deviations as subscripts. † indicates high-resource settings.
8358
Pretraining	BIOMED	CS
CHEMPROT	RCT-500	ACL-ARC
ROBERTA	83.21.4	80.30.5	71.32.8
TAPT	83.00.6	80.51.3	73.23.6
RAND-TAPT	83.30.5	81.60.6	78.74.0
50NN-TAPT	83.30.8	81.70.5	70.13.5
150NN-TAPT	83.30.9	81.90.8	78.52.2
500NN-TAPT	84.50.4	82.60.4	77.42.3
DAPT	84.10.5	83.50.8	73.21.5
Table 20: Additional examples that highlight the overlap between IMDB reviews and REALNEWS articles.
8359
Source	During median follow-up of 905 days ( IQR 773-1050 ) , 49 people died and 987 unplanned admissions were recorded ( totalling 5530 days in hospital ) .
Neighbor 0	Of this group, 26% died after discharge from hospital, and the median time to death was 11 days (interquartile range, 4.0-15.0 days) after discharge.
Neighbor 1	The median hospital stay was 17 days (range 8-26 days), and all the patients were discharged within 1 month.
Neighbor 2	The median hospital stay was 17 days (range 8-26 days).
Neighbor 3	The median time between discharge and death was 25 days (mean, 59.1 days) and no patient was alive after 193 days.
Neighbor 4	The length of hospital stay after colostomy formation ranged from 3 days to 14 days with a median duration of 6 days (+IQR of 4 to 8 days).
Source	Randomized , controlled , parallel clinical trial .
Neighbor 0	Design: Unblinded, randomised clinical controlled trial.
Neighbor 1	These studies and others led to the phase III randomized trial RTOG 0617/NCCTG 0628/ CALGB 30609.
Neighbor 2	-Definitive randomized controlled clinical trial (RCT):
Neighbor 3	RCT 1/4 randomized controlled trial.
Neighbor 4	randomized controlled trial [ Fig. 3(A)].
Source	Forty primary molar teeth in 40 healthy children aged 5-9 years were treated by direct pulp capping .
Neighbor 0	In our study, we specifically determined the usefulness of the Er:YAG laser in caries removal and cavity preparation of primary and young permanent teeth in children ages 4 to 18 years.
Neighbor 1	Males watched more TV than females, although it was only in primary school-aged children and on weekdays.
Neighbor 2	Assent was obtained from children and adolescents aged 7-17 years.
Neighbor 3	Cardiopulmonary resuscitation was not applied to children aged ;5 years (Table 2).
Neighbor 4	It measures HRQoL in children and adolescents aged 2 to 25 years.
Table 21: 5 nearest neighbors of sentences from the RCT dataset (Source) in the BIOMED domain (Neighbors 0-4).
8360