SlowGuess/ACL_2020_OCR huggingface dataset:
Roee Aharoni<sup>1</sup> & Yoav Goldberg<sup>1,2</sup>
<sup>1</sup> Computer Science Department, Bar Ilan University
2 Allen Institute for Artificial Intelligence
first.last@gmail.com
The notion of "in-domain data" in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addition, domain labels are many times unavailable, making it challenging to build domain-specific systems. We show that massive pretrained language models implicitly learn sentence representations that cluster by domains without supervision – suggesting a simple data-driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and by precision and recall of sentence selection with respect to an oracle.
It is common knowledge in modern NLP that using large amounts of high-quality training data is a key aspect in building successful machine-learning based systems. For this reason, a major challenge when building such systems is obtaining data in the domain of interest. But what defines a domain? Natural language varies greatly across topics, styles, levels of formality, genres and many other linguistic nuances (van der Wees et al., 2015; van der Wees, 2017; Niu et al., 2017). This overwhelming diversity of language makes it hard to find the right data for the task, as it is nearly impossible to well-define the exact requirements from such data with respect to all the aforementioned aspects. On top of that, domain labels are usually unavailable – e.g. in large-scale web-crawled data like Common Crawl<sup>1</sup> which was recently used to
Figure 1: A 2D visualization of average-pooled BERT hidden-state sentence representations using PCA. The colors represent the domain for each sentence.
train state-of-the-art pretrained language models for various tasks (Raffel et al., 2019).
Domain data selection is the task of selecting the most appropriate data for a domain from a large corpus given a smaller set of in-domain data (Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Silva et al., 2018). In this work, we propose to use the recent, highly successful self-supervised pre-trained language models, e.g. Devlin et al. (2019); Liu et al. (2019) for domain data selection. As pretrained LMs demonstrate state-of-the-art performance across many NLP tasks after being trained on massive amounts of data, we hypothesize that the robust representations they learn can be useful for mapping sentences to domains in an unsupervised, data-driven approach. We show that these models indeed learn to cluster sentence representations to domains without further supervision (e.g. Figure 1), and quantify this phenomenon by fitting Gaussian Mixture Models (GMMs) to the learned representations and measuring the purity of the resulting unsupervised clustering. We then pro
1https://commoncrawl.org/
7747
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
pose methods to leverage these emergent domain clusters for domain data selection in two ways:
- Via distance-based retrieval in the sentence embedding space induced by the pretrained language model.
- By fine-tuning the pretrained language model for binary classification, where positive examples are from the domain of interest.
Our methods enable to select relevant data for the task while requiring only a small set of monolingual in-domain data. As they are based solely on the representations learned by self-supervised LMs, they do not require additional domain labels which are usually vague and over-simplify the notion of domain in textual data. We evaluate our method on data selection for neural machine translation (NMT) using the multi-domain German-English parallel corpus composed by Koehn and Knowles (2017). Our data selection methods enable to train NMT models that outperform those trained using the well-established cross-entropy difference method of Moore and Lewis (2010) across five diverse domains, achieving a recall of more than  $95\%$  in all cases with respect to an oracle that selects the "true" in-domain data.
Our contributions in this work are as follows. First, we show that pre-trained language models are highly capable of clustering textual data to domains with high accuracy in a purely unsupervised manner. Second, we propose methods to select in-domain data based on this property using vector-space retrieval and positive-unlabeled fine-tuning of pretrained language models for binary classification. Third, we show the applicability of our proposed data selection methods on a popular benchmark for domain adaptation in machine translation. An additional contribution is a new, improved data split we create for this benchmark, as we point on issues with previous splits used in the literature. The code and data for this work is publicly available. We hope this work will encourage more research on understanding the data landscape in NLP, enabling to "find the right data for the task" in the age of massive models and diverse data sources.
2 Emerging Domain Clusters in Pretrained Language Models
The proliferation of massive pretrained neural language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) has enabled great progress on many NLP benchmarks (Wang et al., 2018, 2019a). Larger and larger models trained on billions of tokens of raw text are released in an ever-increasing pace (Raffel et al., 2019), enabling the NLP community to fine-tune them for the task of interest. While many works tried to "probe" those models for the morphological, syntactic and semantic information they capture (Tenney et al., 2019; Goldberg, 2019; Clark et al., 2019), an important aspect of language remained overlooked in this context – the domain the data comes from, often referred to as the "data distribution".
The definition of domain is many times vague and over-simplistic (e.g. "medical text" may be used for biomedical research papers and for clinical conversations between doctors and patients, although the two vary greatly in topic, formality etc.). A common definition treats a domain as a data source: "a domain is defined by a corpus from a specific source, and may differ from other domains in topic, genre, style, level of formality, etc." (Koehn and Knowles, 2017). We claim that a more data-driven definition should take place, as different data sources may have sentences with similar traits and vice versa - a single massive web-crawled corpus contains texts in numerous styles, topics and registers. Our analysis in Section 2 shows examples for such cases, e.g. a sentence discussing "Viruses and virus-like organisms" in a legal corpus.
We hypothesize that massive pretrained LMs can learn representations that cluster to domains, as texts from similar domains will appear in similar contexts. We test this hypothesis across several large, publicly-available pretrained LMs; we explore both masked-language-models (MLMs) and auto-regressive LMs.
2.2 Method
We encode multi-domain data at the sentence level into vector representations. We then cluster these vector representations for each model using a Gaussian Mixture Model (GMM) with  $k$  pre-defined clusters. We chose GMM as our clustering approach as it allows soft assignments (vs. hard as
$^{2}$ https://github.com/roeeaharoni/unsupervised-domain-clusters
7748
k=5	k=10	k=15
Random	15.08 (±0.0)	16.77 (±0.0)	17.78 (±0.0)
LDA	24.31 (±0.99)	26.73 (±2.19)	30.79 (±2.97)
with PCA  $(n = 50)$
without PCA
k=5	k=10	k=15	k=5	k=10	k=15
word2vec	53.65 (±0.79)	68.14 (±2.58)	73.44 (±0.68)	45.93	65.80	76.26
BERT-base	87.66 (±0.24)	88.02 (±1.10)	88.37 (±0.66)	85.74	85.08	86.37
BERT-large	85.64 (±6.13)	87.61 (±0.26)	89.07 (±0.53)	68.56	86.53	86.99
DistillBERT	83.68 (±7.14)	86.31 (±0.86)	87.53 (±0.85)	79.00	86.42	88.14
RoBERTa-base	79.05 (±0.10)	86.39 (±0.90)	86.51 (±0.28)	70.21	80.35	81.49
RoBERTa-large	80.61 (±0.33)	89.04 (±0.15)	89.94 (±0.23)	69.88	81.07	85.91
GPT-2	70.30 (±0.05)	84.76 (±0.30)	82.56 (±1.29)	37.82	39.02	41.45
XLNet	55.72 (±0.69)	68.17 (±3.93)	72.65 (±1.92)	30.36	32.96	48.55
Table 1: Unsupervised domain clustering as measured by purity for the different models. Best results are marked in bold for each setting.
signments as in e.g. K-means) which we think fits the task better (as a sentence can be seen as drawn from a mixture of several domain).<sup>3</sup> In all cases, to create a sentence representation we perform average pooling of the last hidden state (before the softmax layer) for each token in the sentence.<sup>4</sup> To accelerate the clustering process and enable visualization we also experiment with performing dimensionality reduction with PCA over the sentence vectors before clustering them. We experiment with k in 5, 10 and 15 to test how adding flexibility would improve the domain clustering accuracy.
2.3 Models and Baselines
For MLM-based models we use BERT (Devlin et al., 2019), DistilBERT (Sanh et al., 2019) and RoBERTa (Liu et al., 2019) (in both the base and large versions). For autoregressive models we use GPT-2 (Radford et al., 2018) and XLNet (Yang et al., 2019). In all cases we use the implementations from the HuggingFace Transformers toolkit (Wolf et al., 2019). We also evaluated three additional, simpler baselines. The first is using representations from word2vec (Mikolov et al., 2013), where we average-pooled the word vectors for the tokens that were present in the model vocabulary. The second is using Latent Dirichlet Allocation (LDA, Blei et al., 2003), which is a classic approach to unsupervised clustering of text. We also
report results for a baseline which assigns sentences by sampling randomly from a uniform distribution over the clusters.
To evaluate the unsupervised domain clustering we used the multi-domain corpus proposed by Koehn and Knowles (2017) which includes textual data in five diverse domains: subtitles $^{6}$ , medical text (PDF documents from the European Medicines Agency), legal text (legislative text of the European Union), translations of the Koran, and IT-related text (manuals and localization files of open-source software). This dataset includes parallel sentences in English and German; for this experiment we used the English portion of the data. See more details on the dataset in Section 3.1. We used 2000 distinct sentences from each domain. To evaluate whether the resulting clusters indeed capture the domains the data was drawn from we measure the clustering purity, which is a well-known metric for evaluating clustering (Manning et al., 2008). To measure the clustering purity, we assign each unsupervised cluster with the most common "true" domain in the sentences assigned to that cluster, and then compute the accuracy according to this majority-based cluster-domain assignment (note that in this case several unsupervised clusters can be assigned to the same domain). In cases where randomness is involved we run each experiment five times with different initializations and report the mean and variance of the purity metric for each model.
3See further discussion comparing GMMs and K-means in Daume (2009).
4 Using the penultimate layer or others may result in better performance; we leave this for future work.
<sup>5</sup>We used the LDA implementation provided in the Gensim toolkit: https://radimrehurek.com/gensim/
<sup>6</sup>From http://www.opensubtitles.org/
Figure 2: A confusion matrix for clustering with  $k = 5$  using BERT-base.
2.5 Results and Discussion
As can be seen in Table 1, pre-trained language models are indeed highly capable of generating sentence representations that cluster by domains, resulting in up to  $87.66\%$ ,  $89.04\%$  and  $89.94\%$  accuracy when using  $k = 5$ ,  $k = 10$  and  $k = 15$  clusters, respectively, across 10,000 sentences in 5 domains. We find these scores remarkably high given our straight-forward average-pooling strategy and that no domain-supervision was involved in the process of learning the pre-trained representations. Figure 3 also demonstrates the quality of the obtained clusters in 2D using the BERT-base model, where the ellipses describe the mean and variance parameters learned for each cluster by the GMM with  $k = 5$ .
We note that some classes of models did better than others: while all vector-based models did far better than the random and LDA baselines, the MLM-based models dominated in all cases over word2vec and the auto-regressive models. This may be explained by the fact that the MLM-based models use the entire sentence context when generating the representations for each token, while the auto-regressive models only use the past context, and word2vec uses a limited window context. Using PCA improved performance in most cases and especially for the auto-regressive models, although the results for the MLMs remain high in
both cases - suggesting that these models encode the information very differently.
2.6 Analysis
As can be seen in Figure 3, in some areas the domains are somewhat overlapping in the embedding space, which may lead to outlier cases where examples from one domain are assigned to a cluster of another domain. We plot a confusion matrix (Figure 2) to analyze this further based on the clustering with BERT-base and  $k = 5$ . We first note that the outlier sentences are much shorter than the average sentence length in the corpus (11.62 tokens on average for outliers vs. 20.5 tokens on average in general). This makes sense as shorter sentences contain less information, making it harder to assign them to an appropriate cluster. Table 2 shows examples of outlier sentences, assigned to clusters of domains different from their originating domain. We can see that in many cases the assignments are sensible – for example for sentences originating from the subtitles corpus, a sentence that mentions "great priest" is assigned to the Koran cluster, a sentence that mentions "The International Criminal Court in The Hague" is assigned to the Law cluster, a sentence that mentions "the virus" is assigned to the Medical cluster and so on. This strengthens our claim that defining domains based on the corpus they originated from is over-simplistic, and using a data-driven approach may enable to find better domain assignments across different corpora.
The domain that attracted the largest number of outliers is the IT domain cluster, with 597 sentences assigned to it from other domains. Looking
Figure 3: A 2D visualization of the unsupervised GMM clustering for the same sentences as in Figure 1.
7Similar visualizations for additional models are available in the supplementary material.
Note that the LDA models were trained using the multidomain data alone, and did not utilize additional pretraining as in the other, more successful models. This may explain their relatively weak performance.
Subtitles assigned to Koran	Subtitles assigned to Medical
I am Spa'am, high priest of the boars.	Oxygen supply at 50%.
Joseph, go in peace, and the Lord be with you.	Or it can help her walk again if the virus is kept in check with this.
Subtitles assigned to IT	Subtitles assigned to Law
Push it up to the front of the screen.	Statutes, transcripts, redacted immunity agreements.
Polyalloy requires programming to take permanent form.	The Security Council therefore must press for his immediate referral to the International Criminal Court in The Hague.
Law assigned to Medical	Law assigned to IT
- Viruses and virus-like organisms	"INFORMATION SOCIETY STATISTICS
where the glucose content is equal to or less than the fructose content.	This document must be attached to the certificate and field with it, except where there is a computerised checking system.
Medical assigned to Law	Medical assigned to IT
This will be introduced by a Regulation adopted by the European Commission.	An updated and improved version of the CD-ROM was issued to all subscribers during the first half of the year.
The marketing authorisation was renewed on 22 May 2002 and 22 May 2007.	- All tables will be based on generic and not product-specific data.
IT assigned to Medical	IT assigned to Subtitles
R65: Harmful: may cause lung damage if swallowed	At the end we say good bye.
Automatic Red-Eye Removal	What would you like to do for your next shot?
Table 2: Sentences from one domain which were assigned to another domain by the BERT-based clustering,  $\mathrm{k} = 5$  .
more closely we find that more than half of these sentences (340 out of 597) included numbers (e.g. "34% 25% 34%" (from medical), " (b) reference number 20 is deleted;" (from law), "(Command of Prostration # 1)" (from Koran) or "The message, R2." (from subtitles)). As numbers appear in many different contexts, they may be harder to assign to a specific domain by the context-aware language models in such short sentences. The second largest attractor of outliers is the Subtitles cluster, with 372 sentences assigned to it from other domains. We find that most of these sentences contain personal pronouns or question marks (228 out of 372,  $61.2\%$  ) while the ratio of such sentences in the entire corpus is only  $40\%$ . Examples include "Why did you choose the name & amarok;?" (from IT), or "What is Avonex?" (from Medical). This may be expected as the subtitles corpus mainly includes transcriptions of spoken, conversational language, and "conversation tends to have more verbs, more personal pronouns, and more questions" (Conrad and Biber, 2005). Another possible reason for the subtitles domain to attract outliers is the fact that this is the least-topical cluster: movies and TV series may discuss diverse topics, unlike medical, religious, legal and technical texts that may have a more cohesive topic.
3 Neural Machine Translation in a Multi-Domain Scenario
As we showed that pre-trained language models are indeed very useful in clustering sentence representations by domains in an unsupervised manner, we now seek to harness this property for a down
stream task - domain data selection for machine translation. Domain data selection is the task of selecting examples from a large corpus which are as close as possible to the domain of interest, given a smaller set of in-domain examples. The selected examples can be used to either (1) train a domain-specific model from scratch (Axelrod et al., 2011), (2) fine-tune a pre-trained general-domain model (Sajjad et al., 2017; Silva et al., 2018), or (3) prioritize data for annotation as in an Active-Learning framework, if only monolingual data is available (Haffari et al., 2009). To demonstrate the need for domain data selection and set the stage for our data selection experiments, we perform preliminary experiments with NMT in a multi-domain scenario.
3.1 Multi-Domain Dataset
To simulate a diverse multi-domain setting we use the dataset proposed in Koehn and Knowles (2017), as it was recently adopted for domain adaptation research in NMT (Hu et al., 2019; Müller et al., 2019; Dou et al., 2019a,b). The dataset includes parallel text in German and English from five diverse domains (Medical, Law, Koran, IT, Subtitles; as discussed in Section 2), available via OPUS (Tiedemann, 2012; Aulamo and Tiedemann, 2019).
In a preliminary analysis of the data we found that in both the original train/dev/test split by Koehn and Knowles (2017) and in the more recent split by Müller et al. (2019) there was overlap between the training data and the dev/test data. Fixing these issues is important, as it may affect the conclusions one draws from experiments with
9 More details are available in the supplementary material.
7751
Original	New Split
Medical	1,104,752	248,099
Law	715,372	467,309
IT	378,477	222,927
Koran	533,128	17,982
Subtitles	22,508,639	14,458,058
this dataset. For example, as overlapping development sets favor memorization of the training set, one may choose checkpoints and report results on over-fitting models. This is especially relevant with neural sequence-to-sequence models, as they are highly susceptible to memorization (Aharoni and Goldberg, 2018) and hallucination (Lee et al., 2018), as confirmed by Müller et al. (2019).
To create a better experimental setting to test generalization within and across domains, we create a new data split where we ensure that no such overlap between the training, development and test sets occur. We started from the split of Müller et al. (2019) as it included newer versions of some of the datasets.[10] Furthermore, we did not allow more than one translation of a given source or target sentence, as such cases were very frequent in the dataset and usually stand for duplicate sentence pairs (See Table 3). For example, applying this filtering reduced the size of the Koran corpus from 533,128 sentence pairs to only 17,982. Finally, following Müller et al. (2019) we cap the subti- tles corpus to 500,000 sentence pairs as it is much larger than the rest. We make the new split publicly available and hope it will enable better future experimentation on this important subject.[11]
3.2 Cross-Domain Experiments
Experimental Setup We follow Hu et al. (2019) and train domain-specific models for all domains. We then evaluate each model across the different domain test sets, enabling us to understand the effect of different domains on the downstream MT performance and to set up strong baselines for data selection experiments. We also train a general-domain model using the available data from all domains, as it is also a common approach in multidomain scenarios (Müller et al., 2019). In all experiments we use a similar Transformer (Vaswani et al., 2017) model, and only control for the train-
Table 3: Number of training examples for each domain in the original split (Muller et al., 2019) and in our split.
Medical	Law	Koran	IT	Subtitles
Medical	56.5	18.3	1.9	11.4	4.3
Law	21.7	59	2.7	13.1	5.4
Koran	0.1	0.2	15.9	0.2	0.5
IT	14.9	9.6	2.8	43	8.6
Subtitles	7.9	5.5	6.4	8.5	27.3
All	53.3	57.2	20.9	42.1	27.6
Table 4: SacreBLEU (Post, 2018) scores of our baseline systems on the test sets of the new data split. Each row represents the results from one model on each test set. The best result in each column is marked in bold.
ing data. More details on the exact training and hyperparameter settings for the NMT models are available in the supplementary material.
Results The results for the cross-domain evaluation are available in Table 4. In most cases, the best results for each domain are obtained by training on the in-domain data. Training on all the available data helped mostly for the Koran test set. This is expected as the training data for this domain is considerably smaller than the training data for rest of the domains (Table 3). We can also see that more data is not necessarily better (Gascó et al., 2012): while the subtitles corpus is the largest of all 5 and includes 500,000 sentence pairs, it is second to last in performance as measured by the average BLEU across all test sets.