Question: Are pre-trained language models highly capable of clustering textual data to domains with high accuracy in a purely unsupervised manner?
Evidence: It is common knowledge in modern NLP that using large amounts of high-quality training data is a key aspect in building successful machine-learning based systems. For this reason, a major challenge when building such systems is obtaining data in the domain of interest. But what defines a domain? Natural language varies greatly across topics, styles, levels of formality, genres and many other linguistic nuances BIBREF0, BIBREF1, BIBREF2. This overwhelming diversity of language makes it hard to find the right data for the task, as it is nearly impossible to well-define the exact requirements from such data with respect to all the aforementioned aspects. On top of that, domain labels are usually unavailable – e.g. in large-scale web-crawled data like Common Crawl which was recently used to train state-of-the-art pretrained language models for various tasks BIBREF3.
Domain data selection is the task of selecting the most appropriate data for a domain from a large corpus given a smaller set of in-domain data BIBREF4, BIBREF5, BIBREF6, BIBREF7. In this work, we propose to use the recent, highly successful self-supervised pre-trained language models, e.g. devlin-etal-2019-bert,liu2019roberta for domain data selection. As pretrained LMs demonstrate state-of-the-art performance across many NLP tasks after being trained on massive amounts of data, we hypothesize that the robust representations they learn can be useful for mapping sentences to domains in an unsupervised, data-driven approach. We show that these models indeed learn to cluster sentence representations to domains without further supervision (e.g. Figure FIGREF2), and quantify this phenomenon by fitting Gaussian Mixture Models (GMMs) to the learned representations and measuring the purity of the resulting unsupervised clustering. We then propose methods to leverage these emergent domain clusters for domain data selection in two ways:
Via distance-based retrieval in the sentence embedding space induced by the pretrained language model.
By fine-tuning the pretrained language model for binary classification, where positive examples are from the domain of interest.
Our methods enable to select relevant data for the task while requiring only a small set of monolingual in-domain data. As they are based solely on the representations learned by self-supervised LMs, they do not require additional domain labels which are usually vague and over-simplify the notion of domain in textual data. We evaluate our method on data selection for neural machine translation (NMT) using the multi-domain German-English parallel corpus composed by BIBREF8. Our data selection methods enable to train NMT models that outperform those trained using the well-established cross-entropy difference method of BIBREF4 across five diverse domains, achieving a recall of more than 95% in all cases with respect to an oracle that selects the “true” in-domain data.
Our contributions in this work are as follows. First, we show that pre-trained language models are highly capable of clustering textual data to domains with high accuracy in a purely unsupervised manner. Second, we propose methods to select in-domain data based on this property using vector-space retrieval and positive-unlabeled fine-tuning of pretrained language models for binary classification. Third, we show the applicability of our proposed data selection methods on a popular benchmark for domain adaptation in machine translation. An additional contribution is a new, improved data split we create for this benchmark, as we point on issues with previous splits used in the literature. The code and data for this work is publicly available. We hope this work will encourage more research on understanding the data landscape in NLP, enabling to “find the right data for the task” in the age of massive models and diverse data sources.
Our contributions in this work are as follows. First, we show that pre-trained language models are highly capable of clustering textual data to domains with high accuracy in a purely unsupervised manner.
Answer: Yes.