Introduction
Text classification is a fundamental task in Natural Language processing which has been found useful in a wide spectrum of applications ranging from search engines enabling users to identify content on websites, sentiment and social media analysis, customer relationship management systems, and spam detection. Over the past several years, text classification has been predominantly modeled as a supervised learning problem (e.g., BIBREF0 , BIBREF1 , BIBREF2 ) for which appropriately labeled data must be collected. Such data is often domain-dependent (i.e., covering specific topics such as those relating to “Business” or “Medicine”) and a classifier trained using data from one domain is likely to perform poorly on another. For example, the phrase “the mouse died quickly” may indicate negative sentiment in a customer review describing the hand-held pointing device or positive sentiment when describing a laboratory experiment performed on a rodent. The ability to handle a wide variety of domains has become more pertinent with the rise of data-hungry machine learning techniques like neural networks and their application to a plethora of textual media ranging from news articles to twitter, blog posts, medical journals, Reddit comments, and parliamentary debates BIBREF0 , BIBREF3 , BIBREF4 , BIBREF5 .
The question of how to best deal with multiple domains when training data is available for one or few of them has met with much interest in the literature. The field of domain adaptation BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 aims at improving the learning of a predictive function in a target domain where there is little or no labeled data, using knowledge transferred from a source domain where sufficient labeled data is available. Another line of work BIBREF11 , BIBREF12 , BIBREF13 assumes that labeled data may exist for multiple domains, but in insufficient amounts to train classifiers for one or more of them. The aim of multi-domain text classification is to leverage all the available resources in order to improve system performance across domains simultaneously.
In this paper we investigate the question of how domain-specific data might be obtained in order to enable the development of text classification tools as well as more domain aware applications such as summarization, question answering, and information extraction. We refer to this task as domain detection and assume a fairly common setting where the domains of a corpus collection are known and the aim is to identify textual segments which are domain-heavy, i.e., documents, sentences, or phrases providing evidence for a given domain.
Domain detection can be formulated as a multilabel classification problem, where a model is trained to recognize domain evidence at the sentence-, phrase-, or word-level. By definition then, domain detection would require training data with fine-grained domain labels, thereby increasing the annotation burden; we must provide labels for training domain detectors and for modeling the task we care about in the first place. In this paper we consider the problem of fine-grained domain detection from the perspective of Multiple Instance Learning (MIL; BIBREF14 ) and develop domain models with very little human involvement. Instead of learning from individually labeled segments, our model only requires document-level supervision and optionally prior domain knowledge and learns to introspectively judge the domain of constituent segments. Importantly, we do not require document-level domain annotations either since we obtain these via distant supervision by leveraging information drawn from Wikipedia.
Our domain detection framework comprises two neural network modules; an encoder learns representations for words and sentences together with prior domain information if the latter is available (e.g., domain definitions), while a detector generates domain-specific scores for words, sentences, and documents. We obtain a segment-level domain predictor which is trained end-to-end on document-level labels using a hierarchical, attention-based neural architecture BIBREF15 . We conduct domain detection experiments on English and Chinese and measure system performance using both automatic and human-based evaluation. Experimental results show that our model outperforms several strong baselines and is robust across languages and text genres, despite learning from weak supervision. We also showcase our model's application potential for text summarization.
Our contributions in this work are threefold; we propose domain detection, as a new fine-grained multilabel learning problem which we argue would benefit the development of domain aware NLP tools; we introduce a weakly supervised encoder-detector model within the context of multiple instance learning; and demonstrate that it can be applied across languages and text genres without modification.
Related Work
Our work lies at the intersection of multiple research areas, including domain adaptation, representation learning, multiple instance learning, and topic modeling. We review related work below.
Problem Formulation
We formulate domain detection as a multilabel learning problem. Our model is trained on samples of document-label pairs. Each document consists of INLINEFORM0 sentences INLINEFORM1 and is associated with discrete labels INLINEFORM2 . In this work, domain labels are not annotated manually but extrapolated from Wikipedia (see Section SECREF6 for details). In a non-MIL framework, a model typically learns to predict document labels by directly conditioning on its sentence representations INLINEFORM3 or their aggregate. In contrast, INLINEFORM4 under MIL is a learned function INLINEFORM5 of latent instance-level labels, i.e., INLINEFORM6 . A MIL classifier will therefore first produce domain scores for all instances (aka sentences), and then learn to integrate instance scores into a bag (aka document) prediction.
In this paper we further assume that the instance-bag relation applies to sentences and documents but also to words and sentences. In addition, we incorporate prior domain information to facilitate learning in a weakly supervised setting: each domain is associated with a definition INLINEFORM0 , i.e., a few sentences providing a high-level description of the domain at hand. For example, the definition of the “Lifestyle” domain is “the interests, opinions, behaviors, and behavioral orientations of an individual, group, or culture”.
Figure FIGREF5 provides an overview of our Domain Detection Network, which we call DetNet. The model comprises two modules; an encoder learns representations for words and sentences whilst incorporating prior domain information; a detector generates domain scores for words, sentences, and documents by selectively attending to previously encoded information. We describe the two modules in more detail below.
The Encoder Module
We learn representations for words and sentences using identical encoders with separate learning parameters. Given a document, the two encoders implement the following steps: INLINEFORM0
For each sentence INLINEFORM0 , the word-level encoder yields contextualized word representations INLINEFORM1 and their attention weights INLINEFORM2 . Sentence embeddings INLINEFORM3 are obtained via weighted averaging and then provided as input to the sentence-level encoder which outputs contextualized representations INLINEFORM4 and their attention weights INLINEFORM5 .
In this work we aim to model fairly long documents (e.g., Wikipedia articles; see Section SECREF6 for details). For this reason, our encoder builds on the Transformer architecture BIBREF15 , a recently proposed highly efficient model which has achieved state-of-the-art performance in machine translation BIBREF15 and question answering BIBREF35 . The Transformer aims at reducing the fundamental constraint of sequential computation which underlies most architectures based on recurrent neural networks. It eliminates recurrence in favor of applying a self-attention mechanism which directly models relationships between all words in a sentence, regardless of their position.
The Detector Module
DetNet adopts three detectors corresponding to words, sentences, and documents: INLINEFORM0
WordDet first produces word domain scores using both lexical semantic information INLINEFORM0 and prior (domain) knowledge INLINEFORM1 ; SentDet yields domain scores for sentences while integrating downstream instance signals INLINEFORM2 and sentence semantics INLINEFORM3 ; finally, DocDet makes the final document-level predictions based on sentence scores.
Experimental Setup
[t] Document Generation Input: INLINEFORM0 : Label combinations
INLINEFORM0 : Sentence subcorpora
INLINEFORM0 : Maximum document length
Output: A synthetic document [0] Generate INLINEFORM0 Generate a document domain set INLINEFORM1 INLINEFORM2
INLINEFORM0 Number of domain labels Generate a noisy domain INLINEFORM1 INLINEFORM2
INLINEFORM0 ; A set of candidate domain sets
INLINEFORM0 INLINEFORM1 INLINEFORM2
INLINEFORM0 Number of unused labels INLINEFORM1 Number of sentence blocks INLINEFORM2 For generated sentences
INLINEFORM0 INLINEFORM1 Generate INLINEFORM2 Generate INLINEFORM3 sentences INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8 Shuffle INLINEFORM9 INLINEFORM10
Automatic Evaluation
In this section we present the results of our automatic evaluation for sentence and document predictions. Problematically, for sentence predictions we do not have gold-standard domain labels (we have only extrapolated these from Wikipedia for documents). We therefore developed an automatic approach for creating silver standard domain labels which we describe below.
Human Evaluation
Aside from automatic evaluation, we also assessed model performance against human elicited domain labels for sentences and words. The purpose of this experiment was threefold: (a) to validate the results obtained from automatic evaluation; (b) to evaluate finer-grained model performance at the word level; and (c) to examine whether our model generalizes to non-Wikipedia articles. For this, we created a third test set from the New York Times, in addition to our Wikipedia-based English and Chinese datasets. For all three corpora, we randomly sampled two documents for each domain, and then from each document, we sampled one long paragraph or a few consecutive short paragraphs containing 8–12 sentences. Amazon Mechanical Turkers were asked to read these sentences and assign a domain based on the seven labels used in this paper (multiple labels were allowed). Participants were provided with domain definitions. We obtained five annotations per sentence and adopted the majority label as the sentence's domain label. We obtained two annotated datasets for English (Wiki-en and NYT-en) and one for Chinese (Wiki-zh), consisting of 122/14, 111/11, and 117/12 sentences/documents each.
Word-level domain evaluation is more challenging; taken out-of-context, individual words might be uninformative or carry meanings compatible with multiple domains. Expecting crowdworkers to annotate domain labels word-by-word with high confidence, might be therefore problematic. In order to reduce annotation complexity, we opted for a retrieval-style task for word evaluation. Specifically, AMT workers were given a sentence and its domain label (obtained from the sentence-level elicitation study described above), and asked to highlight which words they considered consistent with the domain of the sentence. We used the same corpora/sentences as in our first AMT study. Analogously, words in each sentence were annotated by five participants and their labels were determined by majority agreement.
Fully hierarchical variants of our model (i.e., DetNet INLINEFORM0 , DetNet INLINEFORM1 ) and L-LDA are able to produce word-level predictions; we thus retrieved the words within a sentence whose domain score was above the threshold of 0 and compared them against the labels provided by crowdworkers. MilNet and DetNet INLINEFORM2 can only make sentence-level predictions. In this case, we assume that the sentence domain applies to all words therein. HierNet can only produce document-level predictions based on which we generate sentence labels and further assume that these apply to sentence words too. Again, we report INLINEFORM3 2prp+r INLINEFORM4 p INLINEFORM5 r INLINEFORM6
We show model performance against AMT domain labels in Table TABREF42 . Consistent with the automatic evaluation results, DetNet variants are the best performing models on the sentence-level task. On the Wikipedia datasets, DetNet INLINEFORM0 or DetNet INLINEFORM1 outperform all baselines and DetNet INLINEFORM2 by a large margin, showing that word-level signals can indeed help detect sentence domains. Although statistical models are typically less accurate when they are applied to data that has a different distribution from the training data, DetNet INLINEFORM3 works surprisingly well on NYT, substantially outperforming all other systems. We also notice that prior information is useful in making domain predictions for NYT sentences: since our models are trained on Wikipedia, prior domain definitions largely alleviate the genre shift to non-Wikipedia sentences. Table TABREF43 provides a breakdown of the performance of DetNet INLINEFORM4 across domains. Overall, the model performs worst on LIF and GEN domains (which are very broad) and best on BUS and MIL (which are very narrow).
With regard to word-level evaluation, DetNet INLINEFORM0 and DetNet INLINEFORM1 are the best systems and are significantly better against all comparison models by a wide margin, except L-LDA. The latter is a strong domain detection system at the word-level since it is able to directly associate words with domain labels (see Equation ( EQREF34 )) without resorting to document- or sentence-level predictions. However, our two-level hierarchical model is superior considering all-around performance across sentences and documents. The results here accord with our intuition from previous experiments: hierarchical models outperform simpler variants (including MilNet) since they are able to capture and exploit fine-grained domain signals relatively accurately. Interestingly, prior information does not seem to have an effect on the Wikipedia datasets, but is useful when transferring to NYT. We also observe that models trained on the Chinese datasets perform consistently better than English. Analysis of the annotations provided by crowdworkers revealed that the ratio of domain words in Chinese is higher compared to English (27.47 INLINEFORM2 vs. 13.86 INLINEFORM3 in Wikipedia and 16.42 INLINEFORM4 in NYT), possibly rendering word retrieval in Chinese an easier task.
Table TABREF44 shows the 15 most representative domain words identified by our model (DetNet INLINEFORM0 ) on Wiki-en for our seven domains. We obtained this list by weighting word domain scores INLINEFORM1 with their attention scores: DISPLAYFORM0
and ranking all words in the development set according to INLINEFORM0 , separately for each domain. Since words appearing in different contexts are usually associated with multiple domains, we determine a word's ranking for a given domain based on the highest score. As shown in Table TABREF44 , biosecurity and authoritarianism are prevalent in both GOV and LAW domains. Interestingly, with contextualized word representations, fairly general English words are recognized as domain heavy. For example, technique is a strong domain word in HEA and 420 in GOV (the latter is slang for the consumption of cannabis and highly associated with government regulations).
For comparison, we also show the top domain words identified by L-LDA via matrix INLINEFORM0 (see Equation ( EQREF34 )). To produce meaningful output, we have removed stop words and punctuation tokens, which are given very high domain scores by L-LDA (this is not entirely surprising since INLINEFORM1 is based on simple co-occurrence). Notice that no such post-processing is necessary for our model. As shown in Table TABREF44 , the top domain words identified by L-LDA (on the right) are more general and less informative, compared to those from DetNet INLINEFORM2 (on the left).
Domain-Specific Summarization
In this section we illustrate how fine-grained domain scores can be used to produce domain summaries, following an extractive, unsupervised approach. We assume the user specifies the domains they are interested in a priori (e.g., LAW, HEA) and the system returns summaries targeting the semantics of these domains.
Specifically, we introduce DetRank, an extension of the well-known TextRank algorithm BIBREF42 , which incorporates domain signals acquired by DetNet INLINEFORM0 . For each document, TextRank builds a directed graph INLINEFORM1 with nodes INLINEFORM2 corresponding to sentences, and undirected edges INLINEFORM3 whose weights are computed based on sentence similarity. Specifically, edge weights are represented with matrix INLINEFORM4 where each element INLINEFORM5 corresponds to the transition probability from vertex INLINEFORM6 to vertex INLINEFORM7 . Following barrios2016variations, INLINEFORM8 is computed with the Okapi BM25 algorithm BIBREF43 , a probabilistic version of TF-IDF, and small weights ( INLINEFORM9 ) are set to zeros. Unreachable nodes are further pruned to acquire the final vertex set INLINEFORM10 .
To enhance TextRank with domain information, we first multiply sentence-level domain scores INLINEFORM0 with their corresponding attention scores: DISPLAYFORM0
and for a given domain INLINEFORM0 , we can extract a (domain) sentence score vector INLINEFORM1 . Then, from INLINEFORM2 , we produce vector INLINEFORM3 representing a distribution of domain signals over sentences: DISPLAYFORM0
In order to render domain signals in different sentences more discernible, we scale all elements in INLINEFORM0 to INLINEFORM1 before obtaining a legitimate distribution with the INLINEFORM2 function. Finally, we integrate the domain component into the original transition matrix as: DISPLAYFORM0
where INLINEFORM0 controls the extent to which domain-specific information influences sentence selection for the summarization task; higher INLINEFORM1 will lead to summaries which are more domain-relevant. Here, we empirically set INLINEFORM2 . The main difference between DetRank and TextRank is that TextRank treats INLINEFORM3 as a damping factor and a uniform probability distribution is applied to INLINEFORM4 .
In order to decide which sentence to include in the summary, a node’s centrality is measured using a graph-based ranking algorithm BIBREF42 . Specifically, we run a Markov chain with INLINEFORM0 on INLINEFORM1 until it converges to the stationary distribution INLINEFORM2 where each element denotes the salience of a sentence. In the proposed DetRank algorithm, INLINEFORM3 jointly expresses the importance of a sentence in the document and its relevance to the given domain (controlled by INLINEFORM4 ). We rank sentences according to INLINEFORM5 and select the top INLINEFORM6 ones, subject to a budget (e.g., 100 words).
We ran a judgment elicitation study on summaries produced by TextRank and DetRank. Participants were provided with domain definitions and asked to decide which summary was best according to the criteria of: Informativeness (does the summary contain more information about a specific domain, e.g., “Government and Politics”?), Succinctness (does the summary avoid unnecessary detail and redundant information?), and Coherence (does the summary make logical sense?). Amazon Mechanical Turk (AMT) workers were allowed to answer “Both” or “Neither” in cases where they could not discriminate between summaries. We sampled 50 summary pairs from the English Wikipedia development set. We collected three responses per summary pair and determined which system participants preferred based on majority agreement.
Table TABREF51 shows the proportion of times AMT workers preferred each system according to the criteria of Informativeness, Succinctness, Coherence, and overall. As can be seen, participants find DetRank summaries more informative and coherent. While it is perhaps not surprising for DetRank to produce summaries which are domain informative since it explicitly takes domain signals into account, it is interesting to note that focusing on a specific domain also helps discard irrelevant information and produce more coherent summaries. This, on the other hand, possibly renders DetRank's summaries more verbose (see the Succinctness ratings in Table TABREF51 ).
Figure FIGREF46 shows example summaries for the Wikipedia article Arms Industry for domains MIL and BUS. Both summaries begin with a sentence which introduces the arms industry to the reader. When MIL is the domain of interest, the summary focuses on military products such as guns and missiles. When the domain changes to BUS, the summary puts more emphasis on trade, e.g., market competition and companies doing military business, such as Boeing and Eurofighter.
Conclusions
In this work, we proposed an encoder-detector framework for domain detection. Leveraging only weak domain supervision, our model achieves results superior to competitive baselines across different languages, segment granularities, and text genres. Aside from identifying domain specific training data, we also show that our model holds promise for other natural language tasks, such as text summarization. Beyond domain detection, we hope that some of the work described here might be of relevance to other multilabel classification problems such as sentiment analysis BIBREF29 , relation extraction BIBREF44 , and named entity recognition BIBREF45 . More generally, our experiments show that the proposed framework can be applied to textual data using minimal supervision, significantly alleviating the annotation bottleneck for text classification problems.
A key feature in achieving performance superior to competitive baselines is the hierarchical nature of our model, where representations are encoded step-by-step, first for words, then for sentences, and finally for documents. The framework flexibly integrates prior information which can be used to enhance the otherwise weak supervision signal or to render the model more robust across genres. In the future, we would like to investigate semi-supervised instantiations of MIL, where aside from bag labels, small amounts of instance labels are also available BIBREF23 . It would also be interesting to examine how the label space influences model performance, especially since in our scenario the labels are extrapolated from Wikipedia and might be naturally noisy and/or ambiguous.
Acknowledgments
The authors would like to thank the anonymous reviewers and the action editor, Yusuke Miyao, for their valuable feedback. We acknowledge the financial support of the European Research Council (Lapata; award number 681760). This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract FA8650-17-C-9118. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation therein.

Question: What is the performance of their model?
Answer: Unanswerable

Question: Which text genres did they experiment with?
Answer: Unanswerable

Question: What domains are detected in this paper?
Answer: Answer with content missing: (Experimental setup not properly rendered) In our experiments we used seven target domains: “Business and Commerce” (BUS), “Government and Politics” (GOV), “Physical and Mental Health” (HEA), “Law and Order” (LAW),
“Lifestyle” (LIF), “Military” (MIL), and “General Purpose” (GEN). Exceptionally, GEN does
not have a natural root category.