NATCAT: Weakly Supervised Text Classification with Naturally Annotated Resources

1 . Introduction . Websites with community contributed content contain ample knowledge for natural language processing . In this paper , we seek to improve text classification by leveraging this knowledge in the form of texts paired with natural category annotations . In particular , we present NatCat , a large-scale resource constructed automatically from three data sources : Wikipedia , Stack Exchange,2 and Reddit.3 With rich knowledge about text and categories , we show that NatCat is a useful resource for text classification . To demonstrate the usefulness of NatCat , we use it to train models that compute a score for any document-category pair . As a result , we get weakly supervised ( i.e. , “ dataless ” ; Chang et al. , 2008 ) text classifiers that can be used off-the-shelf to produce interpretable and relevant topics for any document . They can also be effortlessly ported to a specific topic classification task by computing the score for only the labels in the task . Table 1 illustrates the use of a NatCat-trained model on a document from AGNews . To evaluate , we propose CatEval , a standardized benchmark for evaluating weakly supervised text classification with a choice of datasets , label descriptions for each dataset , and baseline results . CatEval comprises a diverse choice of 11 text classification tasks including both topic and sentiment labels , and contains both single and multi-label classification tasks . We show that NatCat is a valuable resource for weakly supervised text classification and study the impact of data domain and pretrained model choice on tasks in CatEval . We analyze the gap between weakly supervised and supervised models and show that the 1 . NatCat is available at https : //github.com/ZeweiChu/NatCat 2. https : //stackexchange.com/ 3. https : //www.reddit.com/ mistakes of our models are reasonable and humanlike , suggesting that the construction process of NatCat is a promising approach to mining data for building text classifiers . 2 . The NatCat Resource . In this section , we describe the creation of the NatCat resource . NatCat is constructed from three different data sources with natural category annotation : Wikipedia , Stack Exchange , and Reddit . Therefore , NatCat naturally contains a wide range of world knowledge that is useful for topical text classification . For each data source , we describe ways of constructing document-category pairs in which the document can be labeled with the category . Wikipedia . Wikipedia documents are annotated with categories by the community contributors . The categories of each Wikipedia document can be found at the bottom of the page . We obtained Wikipedia documents from Wikimedia Downloads . Wikipedia page-to-category mappings were generated from Wiki SQL dumps using the “ categorylinks ” and “ page ” tables . We removed hidden categories by SQL filtering , which are typically maintenance and tracking categories that are unrelated to the document content . We also removed disambiguation categories . After filtering , there are 5.75M documents with at least one category , and a total of 1.19M unique categories . We preprocessed the Wikipedia articles by removing irrelevant information such as the external links at the end of each article . We then removed Wikipedia documents with fewer than 100 non-stopwords . Some category names are lengthy and specific , e.g. , “ Properties of religious function on the National Register of Historic Places in the United States Virgin Islands ” . These categories are unlikely to be as useful for end users or downstream applications as shorter and more common categories . Therefore , we consider multiple ways of augmenting the given categories with additional categories . The first way is to use a heuristic method of breaking long category names into shorter ones . We first use stopwords as separators and keep each part of the non-stopword word sequence as a category name . For each category name of a document , we also run a named entity recognizer [ Honnibal and Montani , 2017 ] to find all named entities in that category name , and add them to the category set of the document . This way we expand the existing category names from Wikipedia . For the example category above , this procedure yields the following categories : “ religious function ” , “ the national register of historic places ” , “ properties ” , “ historic places ” , “ the united states virgin islands ” , “ properties of religious function on the national register of historic places in the united states virgin islands ” , “ united states virgin islands ” , and “ national register ” . Our second method of expansion is based on the fact that Wikipedia categories can have parent categories and therefore form a hierarchical structure . When expanding the category set by adding its ancestors , there is a trade-off between specificity/relevance and generality/utility of category names . Using only the categories provided for the article yields a small set of high-precision , specific categories . Adding categories that are one or two edges away in the graph increases the total number of training pairs and targets more general/common categories , but some of them will be less relevant to the article . In NatCat , we include all categories of documents that are up to two edges away . Stack Exchange . Stack Exchange is a question answering platform where users post and answer questions as a community . Questions on Stack Exchange fall into 308 subareas , each area having its own site . We create document-category pairs by pairing question titles or descriptions with their corresponding subareas . Question titles , descriptions , and subareas are available from Chu et al . [ 2020 ] . Many Stack Exchange subareas have their own corresponding “ meta ” sites . A meta site is meant to discuss the website itself regarding its policy , community , and bugs , etc . When creating this dataset , we merge the subareas with their corresponding meta areas . This gives us over 2 million documents with 156 categories . Reddit . Inspired by Puri and Catanzaro [ 2019 ] , we construct a category classification dataset from Reddit . In our dataset , we propose to classify Reddit post titles to their corresponding subreddit names . We use the OpenWebText4 toolkit to get Reddit posts with more than 3 karma and their subreddit names . We keep only the top 3000 most frequent subreddits as they better capture the common categories that we are interested in.5 This gives us over 7 million documents with 3000 categories . 4. https : //github.com/jcpeterson/openwebtext 5 . Subreddit names are generally more noisy than categories in the other two data sources , and many are abbreviations or shorthand that are meaningful only to particular groups . Wikipedia category names are more formal and understandable by most people . Hence we decide to only keep the most common categories from Reddit . Some example subreddit names ( with ranking by frequency in parentheses ) : Constructed by these three data sources , NatCat covers a wide range of topics and world knowledge . Table 2 summarizes statistics of training sets we sampled from NatCat . Note that all documents from Stack Exchange and Reddit have only one associated category , while a document from Wikipedia may have multiple categories describing it . 3 . CatEval Tasks . To evaluate NatCat , we will use it to build general purpose text classifiers and test them on a variety of text classification tasks . In this section , we introduce CatEval , which comprises a diverse choice of 11 text classification tasks including both topic-related and sentimentrelated labels , and contains both single and multi-label classification tasks . For single label topic classification , we have AGNews,6 DBpedia [ Lehmann et al. , 2015 ] , Yahoo [ Zhang et al. , 2015 ] , and 20 News Groups [ Lang , 1995 ] . For sentiment classification , we use Emotion [ Klinger et al. , 2018 ] , SST-2 [ Socher et al. , 2013 ] , Yelp-2 , and Amazon-2 [ Zhang et al. , 2015 ] . Emotion is a fine-grained sentiment classification tasks with labels expressing various emotions , while the other three are binary sentiment classification tasks differentiating positive and negative sentiments . As for multi-label topical classification , we have NYTimes [ Sandhaus , 2008 ] , Comment,7 and Situation [ Mayhew et al. , 2019 ] datasets . The NYTimes categories have hierarchical structure , but we merely use the category names from the lowest level . We removed newspaper-specific categories that are not topical in nature.8 Of the remaining 2295 categories , we only use the 100 most frequent categories in our experiments , and randomly sample 1 million documents for the training set , 10k for a dev set , and 10k as a test set.9 archeage ( 3001 ) , Deutsche Queers ( 10000 ) , HartfordWhalers ( 20000 ) , pyladies ( 30000 ) , Religitards ( 40000 ) , TheArnoldFanClub ( 50000 ) , HomeSeer ( 60000 ) , 001 ( 68627 ) . 6. https : //www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html 7. https : //dataturks.com/projects/zhiqiyubupt/comment 8. opinion , paid death notices , front page , and op-ed 9 . Training and dev sets are only used for the supervised baselines . Table 3 summarizes the key statistics of each dataset , including the average number of sentences , average number of words , and average sentence length . They cover a broad range of text classification tasks and can serve as a benchmark for text classifiers . Choice of label names . For a given document classification task , we need to specify the name of each label . As in prior work [ Chang et al. , 2008 , Song and Roth , 2014 ] , we manually choose words corresponding to labels in the downstream tasks . Our models and the baselines use the same label names which are provided in the appendix . In recent work , we showed empirically the variance of dataless classifiers due to label name variations , and proposed unsupervised methods to reduce this variance for particular datasets [ Chu et al. , 2021 ] . As NatCat is a large textual resource with ample categories , almost all labels in the CatEval datasets appear in NatCat except for some conjunction phrases , such as “ written work ” , “ manufacturing operations and logistics ” , and “ home and garden ” . However , there is no guarantee that the labels in NatCat have the same definition as the labels in the downstream tasks , and in fact we find such divergences to be causes of error , including when measuring human performance on the text classification tasks . Weakly supervised methods ( and humans ) are more susceptible to semantic imprecision in label names than supervised methods . Why is this weakly supervised ? The goal of weakly supervised text classification is to classify documents into any task-specific categories that are not necessarily seen during training . We build models with NatCat that are capable of scoring any candidate label for any document . To evaluate , we take the test sets of standard document classification tasks , use the model to score each label from the set of possible labels for that task , and return the label with the highest score . The reason we describe our method as “ weakly supervised ” is because it does not require annotated training data with the same labeling schema and from the same distribution as the test set , but rather uses freely-available , naturally-annotated document/category pairs as a training resource . Existing applications . Even though the category distributions of NatCat do not exactly match the target distribution in a downstream task , generalization is possible if they are sufficiently similar and cover a diverse set of knowledge . This setting is referred to using several different terms , including dataless classification [ Chang et al. , 2008 ] , transfer learning [ Pan and Yang , 2009 ] , distant supervision [ Mintz et al. , 2009 ] , and weakly-supervised learning [ Zhou , 2017 ] .

Summary: This paper presents a dataset and evaluation benchmark for weakly supervised text classification. The main resource, NatCat, contains a large number of diverse real-world documents and categories, making it appealing for general-purpose off-the-shelf text classification. The way of generating weakly supervised data is efficient, intuitive and easy to reproduce. The baseline models trained on this resource easily outperform previous weakly supervised approaches.