Venue: NAACL 2019

# Text Classification with Few Examples using Controlled Generalization

Abhijit Mahabal

Google, Pinterest

amahabbal@gmail.com &Jason Baldridge

Google

jridge@google.com &Burcu Karagol Ayan

Google

burcuka@google.com &Vincent Perot

Google

voperot@gaogle.com &Dan Roth

UPenn

danroth@daroth@seas.upenn.edu

###### Abstract

Training data for text classification is often limited in practice, especially for applications with many output classes or involving many related classification problems. This means classifiers must generalize from limited evidence, but the manner and extent of generalization is task dependent. Current practice primarily relies on pre-trained word embeddings to map words unseen in training to similar seen ones. Unfortunately, this squishes many components of meaning into highly restricted capacity. Our alternative begins with sparse pre-trained representations derived from unlabeled parsed corpora; based on the available training data, we select features that offers the relevant generalizations. This produces task-specific semantic vectors; here, we show that a feed-forward network over these vectors is especially effective in low-data scenarios, compared to existing state-of-the-art methods. By further pairing this network with a convolutional neural network, we keep this edge in low data scenarios and remain competitive when using full training sets.

## 1 Introduction

Modern neural networks are highly effective for text classification, with convolutional neural networks (CNNs) as the de facto standard for classifiers that represent both hierarchical and ordering information implicitly in a deep network Kim (2014). Deep models pre-trained on language model objectives and fine-tuned to available training data have recently smashed benchmark scores on a wide range of text classification problems Peters et al. (2018); Howard and Ruder (2018); Devlin et al. (2018).

Despite the strong performance of these approaches for large text classification datasets, challenges still arise with small datasets with few, possibly imbalanced, training examples per class. Labels can be obtained cheaply from crowd workers for some languages, but there are a nearly unlimited number of bespoke, challenging text classification problems that crop up in practical settings Yu et al. (2018). Obtaining representative labeled examples for classification problems with many labels, like taxonomies, is especially challenging.

_Text classification_ is a broad but useful term and covers classification based on topic, on sentiment, and even social status. As Systemic Functional Linguists such as Halliday (1985) point out, language carries many kinds of meanings. For example, words such as _amprosial_ and _delish_ inform us not just of the domain of the text (_food_) and sentiment, but perhaps also of the age of the speaker. Text classification problems differ on the dimensions they distinguish along and thus in the words that help in identifying the class.

As Sachan et al. (2018) show, classifiers mostly focus on sub-lexicons; they memorize patterns instead of extending more general knowledge about language to a particular task. When there is low lexical overlap between training and test data, accuracy drops as much as 23.7%. When training data is limited, most meaning-carrying terms are never seen in training, and the sub-lexicons correspondingly poorer. Classifiers must generalize from available training data, possibly exploiting external knowledge, including representations derived from raw texts. For small training sizes, this requires moving beyond sub-lexicons.

Existing strategies for low data scenarios include treating labels as informative Song and Roth (2014); Chang et al. (2008) and using label-specific lexicons Eisenstein (2017), but neither is competitive when labeled data is plentiful. Instead, we seek classifiers that adapt to both low and high data scenarios.

People exploit parallelism among examples for generalization Hofstadter (2001); Hofstadter and

Sander, 2013). Consider Table 1, which displays five examples from a single class for two tasks. Bolded terms for each task are clearly related, and to a person, suggest abstractions that help relate other terms to the task. This helps with disambiguation: that the word _Pluto_ is the planet and not Disney's character is inferred not just by within-example evidence (e.g. _mission_) but also by cross-example presence of _Mars_ and _astronauts_.

Cross-example analysis also reveals the amount of generalization warranted. For a word associated with a label, word embeddings give us neighbors, which often are associated with that label. What they do not tell us is the extent this associated-with-same-label phenomenon holds; that depends on the granularity of the classes. Cross-example analysis is required to determine how neighbors at various distances are distributed among labels in the training data. This should allow us to include _barley_ and _peaches_ as evidence for a class like _Agriculture_ but only _barley_ for _Grains_.

Most existing systems ignore cross-example parallelism and thus miss out on a strong classification signal. We introduce a flexible method for controlled generalization that selects syntactic-semantic features from sparse representations constructed by Category Builder Mahabal et al. (2018). Starting with sparse representations of words and their contexts, a tuning algorithm selects features with the relevant kinds and appropriate amounts of generalization, making use of parallelism among examples. This produces task-specific dense embeddings for new texts that can be easily incorporated into classifiers.

Our simplest model, CBC (Category Builder Classifier), is a feed-forward network that uses only CB embeddings to represent a document. For small amounts of training data, this simple model dramatically outperforms both CNNs and BERT Devlin et al. (2018). When more data is available, both CNNs and BERT exploit their greater capacity and broad pre-training to beat CBC. We thus create CBCNN, a simple combination of CBC and the CNN that concatenates their pre-prediction layers and adds an additional layer. By training this model with a scheduled block dropout Zhang et al. (2018) that gradually introduces the CBC sub-network, we obtain the benefits of CBC in low data scenarios while obtaining parity with CNNs when plentiful data is available. BERT still dominates when all data is available, suggesting that further combinations or ensembles are likely to improve matters further.

## 2 Evaluation Strategy

Our primary goal is to study classifier performance with limited data. To that end, we obtain learning curves on four standard text classification datasets (Table 2) based on evaluating predictions on the full test sets. At each sample size, we produce multiple samples and run several text classification methods multiple times, measuring the following:

* **Macro-F1 score**. Macro-F1 measures support for all classes better than accuracy, especially with imbalanced class distributions.
* **Recall for the rarest class**. Many measures like F1 and accuracy often mask performance on infrequent but high impact classes, such as detecting toxicity Waseem and Hovy (2016))
* **Degenerate solutions**. Complex classifiers with millions of parameters sometimes produce degenerate classifiers when provided very few training examples; as a result, they can skip some output classes entirely.

\begin{table}
\begin{tabular}{l|c|r l} dataset & \(k\) & train/test/dev & size range \\ \hline
20NG & 20 & 15076/1885/1885 & 513/810 \\ reuters & 8 & 6888/862/864 & 128/3128 \\ spam & 2 & 3344/1115/1115 & 436/2908 \\ attack & 2 & 10000/2000/2000 & 1126/8874 \\ \end{tabular}
\end{table}
Table 2: Data sizes, and the disparity between the smallest and the largest class in training data. The \(k\) column indicates the number of classes in the task.

 

The datasets we chose for evaluation, while all multi-class, form a diverse set in terms of the number of classes and kinds of cohesion among examples in a single class. The former clearly affects training data needs, while the latter informs appropriate generalization.

* **20 Newsgroups** 20Newsgroups (20NG) contains documents from 20 different newsgroups with about 1000 messages from each. We randomly split the documents into an 80-10-10 train-dev-test split. The classes are evenly balanced.
* **Reuters R8.** The Reuters21578 dataset contains Reuters Newswire articles. Following several authors Pinto and Rosso (2007); Zhao et al. (2018), for example), we use only the eight most frequent labels. We begin with a given 80/10/10 split. Given that we focused on single-label classification, we removed items associated with two or more of the top eight labels (about 3% of examples). Classes are highly imbalanced. Of the 6888 training examples, 3128 are labeled _earn_, while only 228 examples are of class _interest_ and only 128 are _ship_.
* **Wiki Comments Personal Attack.** The Wikipedia Detox project collected over 100k discussion comments from English Wikipedia and annotated them for presence of personal attack Wulczyn et al. (2017). We randomly select 10k, 2k, and 2k items as train/dev/test. 11% are attacks.
* **Spam** The SMS Spam Collection v.l has SMS labeled messages that were collected for mobile phone spam research Hidalgo et al. (2012). Each of the 5574 messages is labeled as _spam_ or _ham_.

## 3 Identifying Generalizing Features

In this section, we explicate the source of features, discuss the properties relevant to generalization by focusing on one feature in isolation, and present the overall feature selection method. The overview in Figure 1 displays the order of operations: identify _generalizing features_ based on the training data (done once), and for each document to be classified, convert it to a vector, where each entry corresponds to a generalizing feature.

### Category Builder

Our source of generalizing features is Category Builder CB Mahabal et al. (2018), which constructs a sparse vector space derived from parsed corpora Erk (2012). CB constructs features for _n-grams_ (not just unigrams) that are the union of syntactic context features \(F_{S}\) and co-occurrence features \(F_{C}\). Consider _milk_: an \(F_{S}\) feature is _gallon\(\xrightarrow{prep}\)of\(\xrightarrow{polyj}X\)_ and \(F_{C}\) features include _goal_, _cov_, _drink_, _spill_, etc. Table 3 provides other examples of features evoked by _milk_, along with other n-grams which evoke them. For present purposes, we can treat CB as a matrix with n-grams as rows and features in \(F_{C}\) and \(F_{S}\) as columns. The entries of CB are weights that give the association strength between an n-gram and a feature; these weights are an asymmetric variant of pointwise

Figure 1: **Shaded Region: We use Category Builder data Mahabal et al. (2018) and identify generalizing features in training data, producing a _vectorizer_. This is done once. Unshaded: Given a document, the vectorizer produces a dense vector usable in deep networks.**



mutual information Mahabal et al. (2018).

### Properties of Generalizing Features

Which features generalize well depends on the granularity of classes in a task. Useful features for generalization strike a balance between _breadth_ and _specificity_. A feature that is evoked by many words provides generalization potential because the feature's overall support is likely to be distributed across both the training data and test data. However, this risks over-generalization, so a feature should also be sufficiently specific to be a precise indicator of a particular class.

A key aspect of choosing good features based on a limited training set is to resolve referential ambiguity Quine (1960); Wittgenstein (1953) to the extent supported by the observed uses of the words. To illustrate, consider the _grains_ class in the Reuters Newswire dataset. The word _wheat_ can evoke the features at different levels of the taxonomical hierarchy: _triticum_ (the wheat genus), _poaceae_ (grass family), _spermatophyta_ (seeded plants), _plantae_ (plant kingdom), and _living thing_. The first among these has low breadth and is evoked only by _wheat_. The second is far more useful: specific and yet with a large support, including _maize_ and _sorghum_. The final feature is too broad. In general, the most useful features for generalization are the intermediate features, also known as Basic Level Categories Rosch et al. (1976).

Another important aspect of generalization comprises the _facets_ of meaning. For example, the word _milk_ has facets relating it to other liquids (e.g., _oil_, _kerosene_), foods (_cheese_, _pasta_), white things (_ivory_), animal products (_honey_, _eggs_), and allergens (_pollen_, _ragweed_). Along these axes, generalization can be more or less conservative; e.g., both _cheese_ and _tears of a phoenix_ are animal products, but the former is semantically closer to _milk_. Looking back at Table 3, the utility of individual features evoked by _milk_ for tasks involving related topics varies; e.g., does the classification problem pertain to _food_ or _animal husbandary_?

### Focus on a single feature

A single _generalizing feature_ is associated with many n-grams, each of which evokes it (with different strengths). Table 4 displays n-grams that evoke the feature _co-occurrence with Saturn V_, as discovered by unsupervised analysis of a large corpus of web pages. The table further displays the interaction of this unsupervised feature with supervised data, specifically, with the label _sci.space_ in 20NG, when using a size 320 training sample that contain only 18 _sci.space_ documents. Counts for some evoking terms are shown within and outside this class, for both training and test data.

**Notation.** We introduce some notation and explicate with Table 4. We have a labelled collection of training documents \(T\). \(T_{l}\) is the training examples with label \(l\). The _positive support set_\(\Psi_{l}(f,t)\) is the set of n-grams in \(T_{l}\) evoking feature \(f\) with weight greater than \(t\), here, {_apollo, rocket_, ..., _shattle_} for \(t{=}2.3\). The _positive support size_\(\Lambda_{l}(f,2.3){=}|\Psi_{l}(f,2.3){=}|{=}5\) and the _positive support weight_\(\lambda_{l}(f,2.3)\) is the sum of counts of supports of \(f\) in \(l\) with weight greater than \(2.3\), here \(1{+}3{+}2{+}1{+}2{=}9\). Analogously the _negative support weight_\(\lambda_{l}(f,2.3)\) is the sum of counts from outside \(T_{l}\); here, \(1{+}1{=}2\) since {_apollo, rocket_} were seen outside _sci.space_ once each.

What makes this feature (_words that have co-occurred with Saturn V_) well suited for _sci.space_ is that many evoking words here are associated with the label _sci.space_. What confirms the benefit is the limited amount of negative support. Crucially, the bolded terms do not occur in the training data, but do occur in the test data. (We stress that we include these counts here only for this exam

\begin{table}
\begin{tabular}{c|c|c c|c c}  & & \multicolumn{2}{c|}{Training} & Testing \\
**n-gram** & wt & \(C\) & \(\overline{C}\) & \(C\) & \(\overline{C}\) \\ \hline apollo & 8.93 & 1 & 1 & 5 & 1 \\
**launch pad** & 8.52 & 0 & 0 & 1 & 0 \\ rocket & 7.32 & 3 & 1 & 8 & 0 \\ rockets & 7.27 & 2 & 0 & 4 & 1 \\ liftoff & 6.92 & 1 & 0 & 1 & 0 \\
**space shuttle** & 6.27 & 0 & 0 & 4 & 0 \\
**space station** & 6.19 & 0 & 0 & 4 & 3 \\
**payload** & 4.23 & 0 & 0 & 5 & 0 \\ shuttle & 2.57 & 2 & 0 & 15 & 3 \\ kennedy & 2.30 & i & 0 & 1 & 4 \\ capacity & 1.95 & 0 & 1 & 0 & 4 \\ \end{tabular}
\end{table}
Table 4: Some evoking n-grams associated with the CB feature _co-occurrence with Saturn V_ and pivoting on the class _sci.space_. Counts for n-grams in training (sample size 320) and test data are shown, within _sci.space_ (\(C\)) and outside (\(\overline{C}\)). Bolded n-grams are not seen in training but occur in test, providing generalization. The dashed line represents a threshold; higher scoring n-grams are more cohesive, and thresholding can make a feature cleaner by decreasing semantic drift.

 

ple; our methods do not access the test data for feature selection in our experiments.)

That said, we must limit potential noise from such features, so we seek **thresholded features**\(\langle f,t\rangle\), as suggested by the dashed line in Table 4. Items below this line are prevented from evoking \(f\). We choose the highest threshold such that dropped negative support exceeds dropped positive support. This is determined simply by going through all the supports of a feature, sorted by ascending weight, and checking the positive and negative support of all features with smaller versus greater weight given the class. The weight of the feature at this cusp is used as the threshold of the feature for this particular class. This \(\langle f,t\rangle\) pair then forms one element of the CB-vector used as a feature for classification.

Given the labeled subsets of \(T\) and this feature thresholding algorithm, we produce a vectorizer that embeds documents. The values of a document's embedding _are not_ directly associated with any class. Such association happens during training. Although _sci.space_ accounts for just 6% of the documents, 75% of documents that contain an n-gram evoking the _Saturn V_ feature are in that class. A classifier trained with such an embedding should learn to associate this feature with that class, and an unseen document containing the unseen-in-training term _space shuttle_ stands a good chance to be classified as _sci.space_.

The feature displayed in Table 4 is useful for the 20NG problem because it contains a class related to space travel. This feature has no utility in spam classification or in sentiment classification, since, for those problems, seeing _rocket_ in one class does not make it more likely that a document containing _space station_ belongs to that same class. This example illustrates why a generalization strategy must incorporate both what we can learn from unsupervised data as well as (limited) labeled training data.

### Overall feature selection

We now describe how we use the training data \(T\) to produce a set of features-and-threshold pairs; each chosen feature-with-threshold \(\langle f,t\rangle\) will be one component in the CB-vectors provide to classifiers. Calculation of features for a single class is a three step process: (i) for each feature \(f\), choose a threshold \(t\) (as discussed above) (ii) score the resultant \(\langle f,t\rangle\) (iii) filter useless or redundant \(\langle f,t\rangle\).

Given a label \(l\) and a feature \(f\), we implicitly produce a table of supporting n-grams and their distribution within and outside \(l\) (e.g. as in Table 4). This involves computing the precision of a feature at a given threshold value, comparing it to the class probability and deciding whether to keep it.

Recall the positive support \(\lambda_{l}(f,t)\) and negative support \(\lambda_{\overline{j}}(f,t)\) defined previously. The **precision** of \(f\) at threshold \(t\) is \(\mu_{l}(f,t)=\frac{\lambda_{l}(f,t)}{\lambda_{l}(f,t)+\lambda_{\overline{l}}( f,t)}\), (this is \(\frac{9}{11}\) in the example of Table 4, with \(t{=}2.3\)). However, since we are often dealing with low counts, we smooth the precision toward the empirical class probability of \(l\), \(p(l)=\frac{|T_{l}|}{|T_{l}|+|T_{l}^{\prime}|}\).

\[\tilde{\mu}_{l}(f,t)=\frac{\lambda_{l}(f,t)+p(l)\alpha}{\lambda_{l}(f,t)+ \lambda_{\overline{l}}(f,t)+\alpha}\]

The score \(S_{l}(f,t)\) is reduction in error rate of the smoothed precision relative to the base rate:

\[S_{l}(f,t)=\frac{\tilde{\mu}_{l}(f,t)-p(l)}{1-p(l)}\]

We retain a thresholded feature if it is generalizing (\(\Lambda_{l}(\langle f,t\rangle)>1\)), has better-than-chance precision (we use \(S_{l}(\langle f,t\rangle)>0.01\)), and is not redundant (i.e., its positive support has one or more terms not present in positive supports of higher scoring features).

### Creating the CB-vector

Each vector dimension corresponds to some \(\langle f,t\rangle\). The evocation level of \(f\) is the sum of its evocation for the n-grams in the document \(d\), \(e_{d}(f)=\sum_{w\in d}CB(w,f)\). The vector entry is \(\frac{e_{d}(f)}{t}\) when \(e_{d}(f)>=t\), and is clipped to \(0\) otherwise.

## 4 Models

As benchmarks, we use a standard CNN with pretrained embeddings Kim (2014) and BERT Devlin et al. (2018).1 For CNN, we used 300 filters each of sizes 2, 3, 4, 5, and 6, fed to a hidden layer of 200 nodes after max pooling. Pretrained vectors provided by Google were used.2 For BERT, we used the run_classifier script from GitHub and used the BERT-large-uncased model.

Footnote 1: [https://github.com/google-research/bert](https://github.com/google-research/bert)

Footnote 2: [https://code.google.com/archive/p/wordZvec](https://code.google.com/archive/p/wordZvec)

We use the pre-computed vocab-to-context association matrix provided as part of the open 

source Categorial Builder repository.3 This contains 194,051 co-occurrence features (\(F_{C}\)) and 954,276 syntactic features (\(F_{S}\)).

Footnote 3: [https://github.com/google/categorybuilder](https://github.com/google/categorybuilder)

**CBC model**. The _CB-vector_ containing the derived features from the training dataset and CategoryBuilder can be exploited in various ways with existing techniques. The simplest of these is to use a feed-forward network over the _CB-vector_. This model does not encode the tokens or any word order information--information which is highly informative in many classification tasks.

**CBCNN model**. Inspired by the combination of standard features and deep networks in Wide-and-Deep models Cheng et al. (2016), we pair the CBC model with a standard CNN, concatenating their pre-prediction layers, and add an additional layer before the softmax prediction. In early experiments, this combined model performed worse than the CNN on larger data sizes, as the network above the CB-vector effectively stole useful signal from the CNN. To ensure that the more complex CNN side of the network had a chance to train, we employed a block dropout strategy Zhang et al. (2018) with a schedule. During training, with some probability, all weights in the CB-vector are set to 0.5. The probability of hiding decreases from 1 to 0 using a parameterized hyperbolic tangent function \(p_{k}\!=\!\frac{2}{e^{CC}+1}\). Lower values of \(C\) lead to slower convergence to zero. The effect is that the CBC sub-network is introduced gradually, allowing the CNN to train while eventually taking advantage of the additional information.

The natural strategy of replacing with 0s (instead of 0.5 as above) was tried and also works, but less well, since the network has no way to distinguish between genuine absence of feature and hiding. In CB-vector, non-zero values are at least 1, and thus 0.5 does not suffer from this problem.

## 5 Experiments

Our primary goal is to improve generalization for low-data scenarios, but we also want our methods to remain competitive on full data.

### Experimental setup

We compare different models across learning curves of increasing the training set sizes. We use training data sizes of \(40,80,\ldots,5120\) as well as the entire available training data. For each training size, we produce three independent samples by uniformly sampling the training data and training each model three times at each size. The final value reported is the average of all nine runs. All models are implemented in Tensorflow. Batch sizes are between 5 and 64 depending on training size. Training stops after there is no macro-F1 improvement on development data for 1000 steps.

For evaluation, we focus primarily on macro-F1 and recall of the rarest class. The recall on the rarest class is especially important for imbalanced classification problems. For such problems, a model can obtain high accuracy by strongly preferring the majority class, but we seek models that effectively identify minority class labels. (This is especially important for active learning scenarios, where we expect the CB-vectors to help with in future.)

### Results: low data scenarios

Figure 2 shows learning curves giving macro-F1 scores and rarest class recall for all four datasets. When very limited training data is available, the simple CBC model generally outperforms the CNN and BERT, except for the Spam dataset. The more powerful models eventually surpass CBC; however, the CBCNN model provides consistent strong performance at all dataset sizes by combining the generalization of CBC with the general efficacy of CNNs. Importantly, CBCNN provides massive error reductions with low data for 20NG and R8 (tasks with many labels).

Table 5's left half gives results for all models when using only 320 training examples. For 20NG, CNN's macro-F1 is just 43.9, whereas CBC and CBCNN achieve 61.7 and 62.4--the same as CNN performance with four times as much data. These models outperform CNN on R8 as well, reaching 83.7 vs CNN's 74.1, and also on the Wiki-attack dataset, achieving 80.6 vs CNNs 74.0. BERT fails to produce a solution for the two datasets with >2 labels, but does produce the best result for Spam--indicating an opportunity to more fully explore BERT's parameter settings for low data scenarios and to fruitfully combine CBC with BERT.

Rarest class recall is generally much better with less data when exploiting CB-features. For example, with 320 training examples for R8, CNNs reach 36.2 whereas CBCNN scores 76.2. Prediction quality with few training examples (especially getting good balance across all labels) also inter 

Figure 2: **Left:** F1 score by training size for 20NG, Reuters, SMS Spam, and Wiki-attack. Data is shown for non-degenerate models, and hence CNN and BERT start at higher sizes (see Table 6). **Right:** Recall for the rarest class for the same models.

 

acts with other strategies for dealing with limited resources, such as active learning. For example, Baldridge and Osborne (2008) obtained stronger data utilization ratios with better base models and uncertainty sampling for Reuters text classification: better models pick better examples for annotation and thus require fewer new labeled data points to achieve a given level of performance.

Importantly, the CBC and CBCNN models take far less data to produce non-degenerate models (defined as a model which produces all output classes as predictions). CNN and BERT have a large number of parameters, and using these powerful tools with small training sets produces unstable results. Table 6 gives the minimum training set sizes at which each model produces at least one non-degenerate model. While it might be possible to ameliorate the instability of CNN and BERT with a wider parameter search and other strategies, nothing special needs to be done for CBC. It is likely that an approach which adaptively selects CBC or CBCNN and BERT would obtain the strongest result across all training set sizes.

For each dataset, among the 100 best features chosen (for training size 640), the breakdown of domain features (\(F_{C}\)) versus type features (\(F_{S}\)) is revealing. As expected, domain features are more important in a topical task such as 20NG (71% are \(F_{C}\) features), while the opposite is true for Spam (19%) and a toxicity dataset like Wiki Attack (23%). Reuters shows a fairly even balance between the two types of features (41%): it is useful for R8 to be topically coherent and also to hone in on fairly narrow groups of words that collectively cover a Basic Level Category.

### Results: full data scenarios

Table 5 provides macro-F1 scores for all models when given all available training data. The CBC model performs well, but its (intentional) ignorance of the actual tokens in a document takes a toll when more labeled documents are available. The CNN benchmark, which exploits both word order and the tokens themselves, is a strong performer. The CBCNN model effectively keeps pace with the CNN--improving on 20NG and R8, though slipping on Wiki-Attack. BERT simply crushes all other models when there is sufficient training data, showing the impact of structured pre-training and consistent with performance across a wide range of tasks in Devlin et al. (2018).

## 6 Conclusion

We demonstrate an effective method for exploiting syntactically derived features from large external corpora and selecting the most useful of those given a small labeled text classification corpus. We show how to do this with the map provided by Category Builder n-grams to features, but other sources of well generalizing features have been exploited for text classification. These include topic models Blei et al. (2003), ontologies such as WordNet Bloehdorn and Hotho (2004) and Wikipedia Category structure Gabrilovich and Markovitch (2009). It may be possible to use these other sources exactly as we use CB. Some of these sources have been manually curated, which makes them high quality but limits the size and facets. We have not yet explored their use because CB features seem to cover many of these

\begin{table}
\begin{tabular}{c|c|c c c c} Model & k & CBC & CNN & CBCNN & BERT \\ \hline
20NG & 20 & 80 & 320 & 80 & 1280 \\ R8 & 8 & 40 & 160 & 40 & 640 \\ Spam & 2 & 40 & 40 & 40 & 40 \\ Attack & 2 & 40 & 40 & 40 & 40 \\ \end{tabular}
\end{table}
Table 6: Minimum training size at which a non-degenerate model was produced in any of 9 runs. With more classes, more data is needed by CNN and BERT to produce acceptable models. k is number of classes.

\begin{table}
\begin{tabular}{c|c c c c|c c c c}  & & \multicolumn{4}{c|}{320 Training Examples} & \multicolumn{4}{c}{Full Training Data} \\ Data & k & CBC & CNN & CBCNN & BERT & CBC & CNN & CBCNN & BERT \\ \hline
20NG & 20 & 61.7 & 43.9 & **62.4** & — & 82.9 & 89.5 & 90.2 & **92.0** \\ R8 & 8 & 82.3 & 74.2 & **83.6** & — & 89.2 & 93.1 & 93.4 & **94.4** \\ Spam & 2 & 80.3 & 87.1 & 90.8 & **95.7** & 93.7 & 96.1 & 96.3 & **97.8** \\ Attack & 2 & **80.7** & 74.0 & 79.6 & 70.5 & 83.8 & 86.2 & 84.5 & **88.2** \\ \end{tabular}
\end{table}
Table 5: Macro-F1 scores on all data sets when using 320 training examples (left) and when using all available training data (right). \(k\) is the number of classes. The CBCNN model provides the strongest overall performance across all data sizes. (Note that BERT produces degenerate solutions for the >2 class problems with 320 examples.) 

sources' strengths--for example, \(F_{C}\) features are like topics, and \(F_{S}\) features like nodes in ontologies. Nonetheless, a combination may add value.

Our focus is on data scarce scenarios. However, it would be ideal to derive utility at both the small and large labeled data sizes. This will likely require models that can generalize with contextual features while also exploiting implicit hierarchical organization and order of the texts, e.g. as done by CNNs and BERT. The CBCNN model is one effective way to do this and we expect there could be similar benefits from combining CBC with BERT. Furthermore, approaches like AutoML Zoph and Le (2017) would likely be effective for exploring the design space of network architectures for representing and exploiting the information inherent in both signals.

Finally, although we focus on multi-class problems here--each example belongs to a single class--the general approach of selecting features should work for multi-label problems. Our confidence in this (unevaluated) claim stems from the observation that we select features one class at a time, treating that class and its complement as a binary classification problem.

## Acknowledgements

We would like to thank our anonymous reviewers and the Google AI Language team, especially Rahul Gupta, Tania Bedrax-Weiss and Emily Pitler, for the insightful comments that contributed to this paper.

## References

* Baldridge and Osborne (2008) Jason Baldridge and Miles Osborne. 2008. Active learning and logarithmic opinion pools for HPSG parse selection. _Natural Language Engineering_, 14(2):191222.
* Blei et al. (2003) David M. Blei, Andrew Ng, and Michael Jordan. 2003. Latent Dirichlet Allocation. _JMLR_, 3:993-1022.
* Bloehdorn and Hotho (2004) Stephan Bloehdorn and Andreas Hotho. 2004. Boosting for text classification with semantic features. In _International Workshop on Knowledge Discovery on the Web_, pages 149-166. Springer.
* Chang et al. (2008) Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In _AAAI_.
* Cheng et al. (2015) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hirishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichnan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & deep learning for recommender systems. In _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems_, DLRS 2016, pages 7-10, New York, NY, USA. ACM.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Eisenstein (2017) Jacob Eisenstein. 2017. Unsupervised learning for lexicon-based classification. In _Proceedings of the National Conference on Artificial Intelligence (AAAI)_.
* Erk (2012) Katrin Erk. 2012. Vector space models of word meaning and phrase meaning: A survey. _Language and Linguistics Compass_, 6(10):635-653.
* Gabrilovich and Markovitch (2009) Evgeniy Gabrilovich and Shaul Markovitch. 2009. Wikipedia-based semantic interpretation for natural language processing. _Journal of Artificial Intelligence Research_, 34:443-498.
* Halliday (1985) M. A. K. Halliday. 1985. _An introduction to functional grammar_. Arnold.
* Hidalgo et al. (2012) Jose Maria Gomez Hidalgo, Tiago A Almeida, and Akebo Yamakami. 2012. On the validity of a new SMS spam collection. In _Proceedings of the 11st ICMLA_, volume 2, pages 240-245. IEEE.
* Hofstadter and Sander (2013) Douglas Hofstadter and Emmanuel Sander. 2013. _Surfaces and essences: Analogy as the fuel and fire of thinking_. Basic Books.
* Hofstadter (2001) Douglas R Hofstadter. 2001. Analogy as the core of cognition. _The Analogical Mind: Perspectives from Cognitive Science_, pages 499-538.
* Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 328-339, Melbourne, Australia.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751, Doha, Qatar.
* Mahabal et al. (2018) Abhijit Mahabal, Dan Roth, and Sid Mittal. 2018. Robust handling of polysemy via sparse representations. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_, pages 265-275.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana 

David Pinto and Paolo Rosso. 2007. On the relative hardness of clustering corpora. In _International Conference on Text, Speech and Dialogue_, pages 155-161. Springer.
* Van Orman Quine (1960) Willard Van Orman Quine. 1960. _Word and object_. MIT press.
* Rosch et al. (1976) Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem. 1976. Basic objects in natural categories. _Cognitive psychology_, 8(3):382-439.
* Sachan et al. (2018) Devendra Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. 2018. Investigating the working of text classifiers. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2120-2131, Santa Fe, New Mexico, USA.
* Song and Roth (2014) Yangqiu Song and Dan Roth. 2014. On dataless hierarchical text classification. In _AAAI_.
* Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In _Proceedings of the NAACL Student Research Workshop_, pages 88-93, San Diego, California.
* Wittgenstein (1953) Ludwig Wittgenstein. 1953. _Philosophical investigations_. John Wiley & Sons.
* Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In _Proceedings of the 26th International Conference on World Wide Web_, pages 1391-1399. International World Wide Web Conferences Steering Committee.
* Yu et al. (2018) Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1206-1215, New Orleans, Louisiana.
* Zhang et al. (2018) Yuan Zhang, Jason Riesa, Daniel Gillick, Anton Bakalov, Jason Baldridge, and David Weiss. 2018. A fast, compact, accurate model for language identification of codemixed text. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 328-337, Brussels, Belgium.
* Zhao et al. (2018) Taotao Zhao, Xiangfeng Luo, Wei Qin, Subin Huang, and Shaorong Xie. 2018. Topic detection model in a single-domain corpus inspired by the human memory cognitive process. _Concurrency and Computation: Practice and Experience_, 30(19):e4642.
* Zoph and Le (2017) Barret Zoph and Quoc V Le. 2017. Neural architecture search with reinforcement learning. In _Proceedings of the International Conference on Learning Representations (ICLR)_.

## Appendix A Appendix

### Changes to the Category Builder Matrix

Category Builder uses two matrices: one mapping items to features (\(M^{V\rightarrow\mathcal{F}}\)), the other mapping features to items (\(M^{\mathcal{F}\to Y}\)). Two are needed since the relationship is asymmetric: the feature _X is a star sign_ is more strongly associated with the term _cancer_ than vice versa, and the two matrices are thus not exact transposes of each other, although they almost are. For this current work, we just use one matrix, \(M^{\mathcal{V}\mapsto\mathcal{F}}\). For syntactic features \(F_{S}\), we directly use the Category Builder rows. For \(F_{C}\) features, however, Category Builder replaced the corresponding submatrix in \(M^{\mathcal{V}\mapsto\mathcal{F}}\) with an identity matrix as described in Mahabal et al. (2018). We obtain that part of the matrix by copying the corresponding rows from \((M^{\mathcal{F}\to Y})^{T}\).

This new matrix will be made available as part of the Category Builder project