SlowGuess/ACL_2020_OCR huggingface dataset:
Suchin Gururangan† Ana Marasovic† Swabha Swayamdipta†  
Kyle Lo† Iz Beltagy† Doug Downey† Noah A. Smith†
†Allen Institute for Artificial Intelligence, Seattle, WA, USA
$\diamond$ Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA {suching, anam, swabhas, kylel, beltagy, dougd, noah}@allenai.org
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multiphase adaptive pretraining offers large gains in task performance.
Today's pretrained language models are trained on massive, heterogeneous corpora (Raffel et al., 2019; Yang et al., 2019). For instance, ROBERTA (Liu et al., 2019) was trained on over 160GB of uncompressed text, with sources ranging from English-language encyclopedic and news articles, to literary works and web content. Representations learned by such models achieve strong performance across many tasks with datasets of varying sizes drawn from a variety of sources (e.g., Wang et al., 2018, 2019). This leads us to ask whether a task's textual domain—a term typically used to denote a distribution over language characterizing a given topic or genre (such as "science" or "mystery novels")—is still relevant. Do the latest large pretrained models work universally or is it still helpful to build
Figure 1: An illustration of data distributions. Task data is comprised of an observable task distribution, usually non-randomly sampled from a wider distribution (light grey ellipsis) within an even larger target domain, which is not necessarily one of the domains included in the original LM pretraining domain – though overlap is possible. We explore the benefits of continued pretraining on data from the task distribution and the domain distribution.
separate pretrained models for specific domains?
While some studies have shown the benefit of continued pretraining on domain-specific unlabeled data (e.g., Lee et al., 2019), these studies only consider a single domain at a time and use a language model that is pretrained on a smaller and less diverse corpus than the most recent language models. Moreover, it is not known how the benefit of continued pretraining may vary with factors like the amount of available labeled task data, or the proximity of the target domain to the original pretraining corpus (see Figure 1).
We address this question for one such high-performing model, ROBERTA (Liu et al., 2019)  $\S 2$ . We consider four domains (biomedical and computer science publications, news, and reviews;  $\S 3$ ) and eight classification tasks (two in each domain). For targets that are not already in-domain for ROBERTA, our experiments show that contin
8342
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ued pretraining on the domain (which we refer to as domain-adaptive pretraining or DAPT) consistently improves performance on tasks from the target domain, in both high- and low-resource settings.
Above, we consider domains defined around genres and forums, but it is also possible to induce a domain from a given corpus used for a task, such as the one used in supervised training of a model. This raises the question of whether pretraining on a corpus more directly tied to the task can further improve performance. We study how domain-adaptive pretraining compares to task-adaptive pretraining, or TAPT, on a smaller but directly task-relevant corpus: the unlabeled task dataset (§4), drawn from the task distribution. Task-adaptive pretraining has been shown effective (Howard and Ruder, 2018), but is not typically used with the most recent models. We find that TAPT provides a large performance boost for ROBERTA, with or without domain-adaptive pretraining.
Finally, we show that the benefits from task-adaptive pretraining increase when we have additional unlabeled data from the task distribution that has been manually curated by task designers or annotators. Inspired by this success, we propose ways to automatically select additional task-relevant unlabeled text, and show how this improves performance in certain low-resource cases (§5). On all tasks, our results using adaptive pretraining techniques are competitive with the state of the art.
- a thorough analysis of domain- and task-adaptive pretraining across four domains and eight tasks, spanning low- and high-resource settings;
- an investigation into the transferability of adapted LMs across domains and tasks; and
- a study highlighting the importance of pretraining on human-curated datasets, and a simple data selection strategy to automatically approach this performance.
Our code as well as pretrained models for multiple domains and tasks are publicly available.
2 Background: Pretraining
Learning for most NLP research systems since 2018 consists of training in two stages. First, a neural language model (LM), often with millions of parameters, is trained on large unlabeled cor
pora. The word (or wordpiece; Wu et al. 2016) representations learned in the pretrained model are then reused in supervised training for a downstream task, with optional updates (fine-tuning) of the representations and network from the first stage.
One such pretrained LM is ROBERTA (Liu et al., 2019), which uses the same transformer-based architecture (Vaswani et al., 2017) as its predecessor, BERT (Devlin et al., 2019). It is trained with a masked language modeling objective (i.e., cross-entropy loss on predicting randomly masked tokens). The unlabeled pretraining corpus for ROBERTA contains over 160 GB of uncompressed raw text from different English-language corpora (see Appendix §A.1). ROBERTA attains better performance on an assortment of tasks than its predecessors, making it our baseline of choice.
Although ROBERTA's pretraining corpus is derived from multiple sources, it has not yet been established if these sources are diverse enough to generalize to most of the variation in the English language. In other words, we would like to understand what is out of ROBERTA's domain. Towards this end, we explore further adaptation by continued pretraining of this large LM into two categories of unlabeled data: (i) large corpora of domain-specific text (§3), and (ii) available unlabeled data associated with a given task (§4).
3 Domain-Adaptive Pretraining
Our approach to domain-adaptive pretraining (DAPT) is straightforward—we continue pretraining ROBERTA on a large corpus of unlabeled domain-specific text. The four domains we focus on are biomedical (BIOMED) papers, computer science (CS) papers, newstext from REALNEWS, and AMAZON reviews. We choose these domains because they have been popular in previous work, and datasets for text classification are available in each. Table 1 lists the specifics of the unlabeled datasets in all four domains, as well as ROBERTA's training corpus. $^{1}$
3.1 Analyzing Domain Similarity
Before performing DAPT, we attempt to quantify the similarity of the target domain to ROBERTA's pretraining domain. We consider domain vocabularies containing the top 10K most frequent unigrams (excluding stopwords) in comparably sized
<sup>1</sup>For BIOMED and CS, we used an internal version of S2ORC that contains papers that cannot be released due to copyright restrictions.
'https://github.com/allenai/dont-stop-pretraining
8343
Domain	Pretraining Corpus	# Tokens	Size	\( {\mathcal{L}}_{\text{ROB. }} \)	\( {\mathcal{L}}_{\text{DAPT }} \)
BIOMED	2.68M full-text papers from S2ORC (Lo et al., 2020)	7.55B	47GB	1.32	0.99
CS	2.22M full-text papers from S2ORC (Lo et al., 2020)	8.10B	48GB	1.63	1.34
NEWS	11.90M articles from REALNEWS (Zellers et al., 2019)	6.66B	39GB	1.08	1.16
REIEWS	24.75M AMAZON reviews (He and McAuley, 2016)	2.11B	11GB	2.10	1.93
ROBERTA (baseline)	see Appendix §A.1	N/A	160GB	$1.19	-
Table 1: List of the domain-specific unlabeled datasets. In columns 5 and 6, we report ROBERTA's masked LM loss on 50K randomly sampled held-out documents from each domain before  $(\mathcal{L}_{\mathrm{ROB}})$  and after  $(\mathcal{L}_{\mathrm{DAPT}})$  DAPT (lower implies a better fit on the sample).  $\ddagger$  indicates that the masked LM loss is estimated on data sampled from sources similar to ROBERTA's pretraining corpus.
Figure 2: Vocabulary overlap  $(\%)$  between domains. PT denotes a sample from sources similar to ROBERTA's pretraining corpus. Vocabularies for each domain are created by considering the top 10K most frequent words (excluding stopwords) in documents sampled from each domain.
random samples of held-out documents in each domain's corpus. We use 50K held-out documents for each domain other than REVIEWS, and 150K held-out documents in REVIEWS, since they are much shorter. We also sample 50K documents from sources similar to ROBERTA's pretraining corpus (i.e., BOOKCORPUS, STORIES, WIKIPEDIA, and REALNEWS) to construct the pretraining domain vocabulary, since the original pretraining corpus is not released. Figure 2 shows the vocabulary overlap across these samples. We observe that ROBERTA's pretraining domain has strong vocabulary overlap with NEWS and REVIEWS, while CS and BIOMED are far more dissimilar to the other domains. This simple analysis suggests the degree of benefit to be expected by adaptation of ROBERTA to different domains—the more dissimilar the domain, the higher the potential for DAPT.
Our LM adaptation follows the settings prescribed for training ROBERTA. We train ROBERTA on each domain for 12.5K steps, which amounts to single pass on each domain dataset, on a v3-8 TPU; see other details in Appendix B. This second phase of pretraining results in four domain-adapted LMs, one for each domain. We present the masked LM loss of ROBERTA on each domain before and after DAPT in Table 1. We observe that masked LM loss decreases in all domains except NEWS after DAPT, where we observe a marginal increase. We discuss cross-domain masked LM loss in Appendix §E.
Under each domain, we consider two text classification tasks, as shown in Table 2. Our tasks represent both high- and low-resource  $(\leq 5\mathrm{K}$  labeled training examples, and no additional unlabeled data) settings. For HYPERPARTISAN, we use the data splits from Beltagy et al. (2020). For RCT, we represent all sentences in one long sequence for simultaneous prediction.
Baseline As our baseline, we use an off-the-shelf ROBERTA-base model and perform supervised fine-tuning of its parameters for each classification task. On average, ROBERTA is not drastically behind the state of the art (details in Appendix §A.2), and serves as a good baseline since it provides a single LM to adapt to different domains.
Classification Architecture Following standard practice (Devlin et al., 2019) we pass the final layer [CLS] token representation to a task-specific feedforward layer for prediction (see Table 14 in Appendix for more hyperparameter details).
Results Test results are shown under the DAPT column of Table 3 (see Appendix §C for validation results). We observe that DAPT improves over ROBERTA in all domains. For BIOMED, CS, and REVIES, we see consistent improve
8344
Domain	Task	Label Type	Train (Lab.)	Train (Unl.)	Dev.	Test	Classes
BIOMED	CHEMPROT	relation classification	4169	-	2427	3469	13
†RCT	abstract sent. roles	18040	-	30212	30135	5
CS	ACL-ARC	citation intent	1688	-	114	139	6
SCIERC	relation classification	3219	-	455	974	7
NEWS	HYPERPARTISAN	partisanship	515	5000	65	65	2
†AGNEWS	topic	115000	-	5000	7600	4
REVIEWS	†HELPFULNESS	review helpfulness	115251	-	5000	25000	2
†IMDB	review sentiment	20000	50000	5000	25000	2
Table 2: Specifications of the various target task datasets.  $\dagger$  indicates high-resource settings. Sources: CHEMPRO (Kringelum et al., 2016), RCT (Dernoncourt and Lee, 2017), ACL-ARC (Jurgens et al., 2018), SCIERC (Luan et al., 2018), HYPERPARTISAN (Kiesel et al., 2019), AGNEWS (Zhang et al., 2015), HELPFULNESS (McAuley et al., 2015), IMDB (Maas et al., 2011).
Dom.	Task	ROBA.	DAPT	\( \neg \)DAPT
BM	CHEMPROT	81.91.0	84.20.2	79.41.3
\( \dagger \)RCT	87.20.1	87.60.1	86.90.1
CS	ACL-ARC	63.05.8	75.42.5	66.44.1
SCIERC	77.31.9	80.81.5	79.20.9
NEWS	HYP.	86.60.9	88.25.9	76.44.9
\( \dagger \)AGNEWS	93.90.2	93.90.2	93.50.2
REV.	\( \dagger \)HELPFUL.	65.13.4	66.51.4	65.12.8
\( \dagger \)IMDB	95.00.2	95.40.2	94.10.4
Table 3: Comparison of ROBERTA (ROBA.) and DAPT to adaptation to an irrelevant domain ( $\neg$  DAPT). Reported results are test macro- $F_1$ , except for CHEMPROT and RCT, for which we report micro- $F_1$ , following Beltagy et al. (2019). We report averages across five random seeds, with standard deviations as subscripts.  $\dagger$  indicates high-resource settings. Best task performance is boldfaced. See §3.3 for our choice of irrelevant domains.
ments over ROBERTA, demonstrating the benefit of DAPT when the target domain is more distant from ROBERTA's source domain. The pattern is consistent across high- and low- resource settings. Although DAPT does not increase performance on AGNEWS, the benefit we observe in HYPERPARTISAN suggests that DAPT may be useful even for tasks that align more closely with ROBERTA's source domain.
3.3 Domain Relevance for DAPT
Additionally, we compare DAPT against a setting where for each task, we adapt the LM to a domain outside the domain of interest. This controls for the case in which the improvements over ROBERTA might be attributed simply to exposure to more data,
regardless of the domain. In this setting, for NEWS, we use a CS LM; for REVIEWS, a BIOMED LM; for CS, a NEWS LM; for BIOMED, a REVIEWS LM. We use the vocabulary overlap statistics in Figure 2 to guide these choices.
Our results are shown in Table 3, where the last column ( $\neg \text{DAPT}$ ) corresponds to this setting. For each task, DAPT significantly outperforms adapting to an irrelevant domain, suggesting the importance of pretraining on domain-relevant data. Furthermore, we generally observe that  $\neg \text{DAPT}$  results in worse performance than even ROBERTA on end-tasks. Taken together, these results indicate that in most settings, exposure to more data without considering domain relevance is detrimental to end-task performance. However, there are two tasks (SCIERC and ACL-ARC) in which  $\neg \text{DAPT}$  marginally improves performance over ROBERTA. This may suggest that in some cases, continued pretraining on any additional data is useful, as noted in Baevski et al. (2019).
3.4 Domain Overlap
Our analysis of DAPT is based on prior intuitions about how task data is assigned to specific domains. For instance, to perform DAPT for HELPFULNESS, we only adapt to AMAZON reviews, but not to any REALNEWS articles. However, the gradations in Figure 2 suggest that the boundaries between domains are in some sense fuzzy; for example,  $40\%$  of unigrams are shared between REVIEWS and NEWS. As further indication of this overlap, we also qualitatively identify documents that overlap cross-domain: in Table 4, we showcase reviews and REALNEWS articles that are similar to these reviews (other examples can be found in Appendix  $\S D$ ). In fact, we find that adapting ROBERTA to
8345
IMDB review	REALNEWS article
“The Shop Around the Corner” is one of the great films from director Ernst Lubitsch. In addition to the talents of James Stewart and Margaret Sullavan, it's filled with a terrific cast of top character actors such as Frank Morgan and Felix Bressart. [...] The makers of “You've Got Mail” claim their film to be a remake, but that's just nothing but a lot of inflated self praise. Anyway, if you have an affection for romantic comedies of the 1940's, you'll find “The Shop Around the Corner” to be nothing short of wonderful. Just as good with repeat viewings.	[...] Three great festive films... The Shop Around the Corner (1940) Delightful Comedy by Ernst Lubitsch stars James Stewart and Margaret Sulla van falling in love at Christmas. Remade as You've Got Mail. [...]
HELPFULNESS review	REALNEWS article
Simply the Best! I've owned countless Droids and iPhones, but this one destroys them all. Samsung really nailed it with this one, extremely fast, very pocketable, gorgeous display, exceptional battery life, good audio quality, perfect GPS & WiFi performance, transparent status bar, battery percentage, ability to turn off soft key lights, superb camera for a smartphone and more! [...]	We're living in a world with a new Samsung. [...] more on battery life later [...] Exposure is usually spot on and focusing is very fast. [...] The design, display, camera and performance are all best in class, and the phone feels smaller than it looks. [...]
Table 4: Examples that illustrate how some domains might have overlaps with others, leading to unexpected positive transfer. We highlight expressions in the reviews that are also found in the REALNEWS articles.
NEWS not as harmful to its performance on REVIEWS tasks (DAPT on NEWS achieves  $65.5_{2.3}$  on HELPFULNESS and  $95.0_{0.1}$  on IMDB).
Although this analysis is by no means comprehensive, it indicates that the factors that give rise to observable domain differences are likely not mutually exclusive. It is possible that pretraining beyond conventional domain boundaries could result in more effective DAPT; we leave this investigation to future work. In general, the provenance of data, including the processes by which corpora are curated, must be kept in mind when designing pretraining procedures and creating new benchmarks that test out-of-domain generalization abilities.
4 Task-Adaptive Pretraining
Datasets curated to capture specific tasks of interest tend to cover only a subset of the text available within the broader domain. For example, the CHEMPROT dataset for extracting relations between chemicals and proteins focuses on abstracts of recently-published, high-impact articles from hand-selected PubMed categories (Krallinger et al., 2017, 2015). We hypothesize that such cases where the task data is a narrowly-defined subset of the broader domain, pretraining on the task dataset itself or data relevant to the task may be helpful.
Task-adaptive pretraining (TAPT) refers to pretraining on the unlabeled training set for a given task; prior work has shown its effectiveness (e.g. Howard and Ruder, 2018). Compared to domain-adaptive pretraining (DAPT; §3), the task-adaptive approach strikes a different trade-off: it uses a far smaller pretraining corpus, but one that is much
more task-relevant (under the assumption that the training set represents aspects of the task well). This makes TAPT much less expensive to run than DAPT, and as we show in our experiments, the performance of TAPT is often competitive with that of DAPT.
Similar to DAPT, task-adaptive pretraining consists of a second phase of pretraining ROBERTA, but only on the available task-specific training data. In contrast to DAPT, which we train for 12.5K steps, we perform TAPT for 100 epochs. We artificially augment each dataset by randomly masking different words (using the masking probability of 0.15) across epochs. As in our DAPT experiments, we pass the final layer [CLS] token representation to a task-specific feedforward layer for classification (see Table 14 in Appendix for more hyperparameter details).
Our results are shown in the TAPT column of Table 5. TAPT consistently improves the ROBERTA baseline for all tasks across domains. Even on the news domain, which was part of ROBERTA pretraining corpus, TAPT improves over ROBERTA, showcasing the advantage of task adaptation. Particularly remarkable are the relative differences between TAPT and DAPT. DAPT is more resource intensive (see Table 9 in §5.3), but TAPT manages to match its performance in some of the tasks, such as SCIERC. In RCT, HYPERPARTISAN, AGNEWS, HELPFULNESS, and IMDB, the results even exceed those of DAPT, highlighting the efficacy of this cheaper adaptation technique.
8346
Domain	Task	ROBERTA	Additional Pretraining Phases
DAPT	TAPT	DAPT + TAPT
BIOMED	CHEMPROT	81.91.0	84.20.2	82.60.4	84.40.4
†RCT	87.20.1	87.60.1	87.70.1	87.80.1
CS	ACL-ARC	63.05.8	75.42.5	67.41.8	75.63.8
SCIERC	77.31.9	80.81.5	79.31.5	81.31.8
NEWS	HYPERPARTISAN	86.60.9	88.25.9	90.45.2	90.06.6
†AGNEWS	93.90.2	93.90.2	94.50.1	94.60.1
REVEWS	†HELPFULNESS	65.13.4	66.51.4	68.51.9	68.71.8
†IMDB	95.00.2	95.40.1	95.50.1	95.60.1
Table 5: Results on different phases of adaptive pretraining compared to the baseline ROBERTA (col. 1). Our approaches are DAPT (col. 2, §3), TAPT (col. 3, §4), and a combination of both (col. 4). Reported results follow the same format as Table 3. State-of-the-art results we can compare to: CHEMPROT (84.6), RCT (92.9), ACL-ARC (71.0), SCIERC (81.8), HYPERPARTISAN (94.8), AGNEWS (95.5), IMDB (96.2); references in §A.2.
BIOMED	RCT	CHEMPROT
TAPT	87.70.1	82.60.5
Transfer-TAPT	87.10.4 (↓0.6)	80.40.6 (↓2.2)
NEWS	HYPERPARTISAN	AGNEWS
TAPT	89.99.5	94.50.1
Transfer-TAPT	82.27.7 (↓7.7)	93.90.2 (↓0.6)
CS	ACL-ARC	SciERC
TAPT	67.41.8	79.31.5
Transfer-TAPT	64.12.7 (↓3.3)	79.12.5 (↓0.2)
REIEWS	HELPFULNESS	IMDB
TAPT	68.51.9	95.70.1
Transfer-TAPT	65.02.6 (↓3.5)	95.00.1 (↓0.7)
Table 6: Though TAPT is effective (Table 5), it is harmful when applied across tasks. These findings illustrate differences in task distributions within a domain.
Combined DAPT and TAPT We investigate the effect of using both adaptation techniques together. We begin with ROBERTA and apply DAPT then TAPT under this setting. The three phases of pretraining add up to make this the most computationally expensive of all our settings (see Table 9). As expected, combined domain- and task-adaptive pretraining achieves the best performance on all tasks (Table 5).<sup>2</sup>
Overall, our results show that DAPT followed by TAPT achieves the best of both worlds of domain and task awareness, yielding the best performance. While we speculate that TAPT followed by DAPT would be susceptible to catastrophic forgetting of the task-relevant corpus (Yogatama et al., 2019), alternate methods of combining the procedures may result in better downstream performance. Future work may explore pretraining with a more sophisticated curriculum of domain and task distributions.
Cross-Task Transfer We complete the comparison between DAPT and TAPT by exploring whether adapting to one task transfers to other tasks in the same domain. For instance, we further pretrain the LM using the RCT unlabeled data, fine-tune it with the CHEMPROT labeled data, and observe the effect. We refer to this setting as Transfer-TAPT. Our results for tasks in all four domains are shown in Table 6. We see that TAPT optimizes for single task performance, to the detriment of cross-task transfer. These results demonstrate that data distributions of tasks within a given domain might differ. Further, this could also explain why adapting only to a broad domain is not sufficient, and why TAPT after DAPT is effective.
5 Augmenting Training Data for Task-Adaptive Pretraining
In §4, we continued pretraining the LM for task adaptation using only the training data for a supervised task. Inspired by the success of TAPT, we next investigate another setting where a larger pool of unlabeled data from the task distribution exists,
2Results on HYPERPARTISAN match those of TAPT, within a standard deviation arising from the five seeds.
8347
Pretraining	BIOMED RCT-500	NEWS HYP.	REVEWS IMDB†
TAPT	79.81.4	90.45.2	95.50.1
DAPT + TAPT	83.00.3	90.06.6	95.60.1
Curated-TAPT	83.40.3	89.99.5	95.70.1
DAPT + Curated-TAPT	83.80.5	92.13.6	95.80.1
typically curated by humans.