SlowGuess/ACL_2020_OCR huggingface dataset:
Dheeraj Mekala $^{1}$  Jingbo Shang $^{1,2}$
$^{1}$  Department of Computer Science and Engineering, University of California San Diego, CA, USA
$^{2}$  Halicioglu Data Science Institute, University of California San Diego, CA, USA
{dmekala, jshang}@ucsd.edu
Weakly supervised text classification based on a few user-provided seed words has recently attracted much attention from researchers. Existing methods mainly generate pseudo-labels in a context-free manner (e.g., string matching), therefore, the ambiguous, context-dependent nature of human language has been long overlooked. In this paper, we propose a novel framework ConWea, providing contextualized weak supervision for text classification. Specifically, we leverage contextualized representations of word occurrences and seed word information to automatically differentiate multiple interpretations of the same word, and thus create a contextualized corpus. This contextualized corpus is further utilized to train the classifier and expand seed words in an iterative manner. This process not only adds new contextualized, highly label-indicative keywords but also disambiguates initial seed words, making our weak supervision fully contextualized. Extensive experiments and case studies on real-world datasets demonstrate the necessity and significant advantages of using contextualized weak supervision, especially when the class labels are fine-grained.
Weak supervision in text classification has recently attracted much attention from researchers, because it alleviates the burden of human experts on annotating massive documents, especially in specific domains. One of the popular forms of weak supervision is a small set of user-provided seed words for each class. Typical seed-driven methods follow an iterative framework — generate pseudolabels using some heuristics, learn the mapping between documents and classes, and expand the seed set (Agichtein and Gravano, 2000; Riloff et al., 2003; Kuipers et al., 2006; Tao et al., 2015; Meng et al., 2018).
Most of, if not all, existing methods generate pseudo-labels in a context-free manner, therefore, the ambiguous, context-dependent nature of human languages has been long overlooked. Suppose the user gives "penalty" as a seed word for the sports class, as shown in Figure 1. The word "penalty" has at least two different meanings: the penalty in sports-related documents and the fine or death penalty in law-related documents. If the pseudolabel of a document is decided based only on the frequency of seed words, some documents about law may be mislabelled as sports. More importantly, such errors will further introduce wrong seed words, thus being propagated and amplified over the iterations.
In this paper, we introduce contextualized weak supervision to train a text classifier based on user-provided seed words. The "contextualized" here is reflected in two places: the corpus and seed words. Every word occurrence in the corpus may be interpreted differently according to its context; Every seed word, if ambiguous, must be resolved according to its user-specified class. In this way, we aim to improve the accuracy of the final text classifier.
We propose a novel framework ConWea, as illustrated in Figure 1. It leverages contextualized representation learning techniques, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), together with user-provided seed information to first create a contextualized corpus. This contextualized corpus is further utilized to train the classifier and expand seed words in an iterative manner. During this process, contextualized seed words are introduced by expanding and disambiguating the initial seed words. Specifically, for each word, we develop an unsupervised method to adaptively decide its number of interpretations, and accordingly, group all its occurrences based on their contextualized representations. We design a principled comparative ranking method to select highly label-
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 323-333 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Our proposed contextualized weakly supervised method leverages BERT to create a contextualized corpus. This contextualized corpus is further utilized to resolve interpretations of seed words, generate pseudolabels, train a classifier and expand the seed set in an iterative fashion.
indicative keywords from the contextualized corpus, leading to contextualized seed words. We will repeat the iterative classification and seed word expansion process until the convergence.
To the best of our knowledge, this is the first work on contextualized weak supervision for text classification. It is also worth mentioning that our proposed framework is compatible with almost any contextualized representation learning models and text classification models. Our contributions are summarized as follows:
- We propose a novel framework enabling contextualized weak supervision for text classification.
- We develop an unsupervised method to automatically group word occurrences of the same word into an adaptive number of interpretations based on contextualized representations and user-provided seed information.
- We design a principled ranking mechanism to identify words that are discriminative and highly label-indicative.
- We have performed experiments on real-world datasets for both coarse- and fine-grained text classification tasks. The results demonstrate the superiority of using contextualized weak supervision, especially when the labels are fine-grained.
Our code is made publicly available at GitHub<sup>1</sup>.
Problem Formulation. The input of our problem contains (1) a collection of  $n$  text documents  $\mathcal{D} = \{\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_n\}$  and (2)  $m$  target classes  $\mathcal{C} = \{\mathcal{C}_1, \mathcal{C}_2, \dots, \mathcal{C}_m\}$  and their seed words  $S = \{S_1, S_2, \dots, S_m\}$ . We aim to build a high-quality
document classifier from these inputs, assigning class label  $\mathcal{C}_j\in \mathcal{C}$  to each document  $\mathcal{D}_i\in \mathcal{D}$ .
Note that, all these words could be upgraded to phrases if phrase mining techniques (Liu et al., 2015; Shang et al., 2018) were applied as preprocessing. In this paper, we stick to the words.
Framework Overview. We propose a framework, ConWea, enabling contextualized weak supervision. Here, "contextualized" is reflected in two places: the corpus and seed words. Therefore, we have developed two novel techniques accordingly to make both contextualizations happen.
First, we leverage contextualized representation learning techniques (Peters et al., 2018; Devlin et al., 2019) to create a contextualized corpus. We choose BERT (Devlin et al., 2019) as an example in our implementation to generate a contextualized vector of every word occurrence. We assume the user-provided seed words are of reasonable quality — the majority of the seed words are not ambiguous, and the majority of the occurrences of the seed words are about the semantics of the user-specified class. Based on these two assumptions, we are able to develop an unsupervised method to automatically group word occurrences of the same word into an adaptive number of interpretations, harvesting the contextualized corpus.
Second, we design a principled comparative ranking method to select highly label-indicative keywords from the contextualized corpus, leading to contextualized seed words. Specifically, we start with all possible interpretations of seed words and train a neural classifier. Based on the predictions, we compare and contrast the documents belonging to different classes, and rank contextualized words based on how label-indicative, frequent, and
https://github.com/dheeraj7596/ConWea
(a) Similarity Distribution: Windows
(b) Cluster Visualisation: Windows
Figure 2: Document contextualization examples using word "windows" and "penalty".  $\tau$  is decided based on the similarity distributions of all seed word occurrences. Two clusters are discovered for both words, respectively.
(c) Cluster Visualisation: Penalty
unusual these words are. During this process, we eliminate the wrong interpretations of initial seed words and also add more highly label-indicative contextualized words.
This entire process is visualized in Figure 1. We denote the number of iterations between classifier training and seed word expansion as  $T$ , which is the only hyper-parameter in our framework. We discuss these two novel techniques in detail in the following sections. To make our paper self-contained, we will also brief the pseudo-label generation and document classifiers.
3 Document Contextualization
We leverage contextualized representation techniques to create a contextualized corpus. The key objective of this contextualization is to disambiguate different occurrences of the same word into several interpretations. We treat every word separately, so in the rest of this section, we focus on a given word  $w$ . Specifically, given a word  $w$ , we denote all its occurrences as  $w_{1},\ldots ,w_{n}$ , where  $n$  is its total number of occurrences in the corpus.
Contextualized Representation. First, we obtain a contextualized vector representation  $\mathbf{b}_{w_i}$  for each  $w_i$ . Our proposed method is compatible with almost any contextualized representation learning model. We choose BERT (Devlin et al., 2019) as an example in our implementation to generate a contextualized vector for each word occurrence. In this contextualized vector space, we use the cosine similarity to measure the similarity between two vectors. Two word occurrences  $w_i$  and  $w_j$  of the same interpretation are expected to have a high cosine similarity between their vectors  $\mathbf{b}_{w_i}$  and  $\mathbf{b}_{w_j}$ . For the ease of computation, we normalize all contextualized representations into unit vectors.
Choice of Clustering Methods. We model the word occurrence disambiguation problem as a clustering problem. Specifically, we propose to use the  $K$ -Means algorithm (Jain and Dubes, 1988) to cluster all contextualized representations  $\mathbf{b}_{w_i}$  into  $K$  clusters, where  $K$  is the number of interpretations. We prefer  $K$ -Means because (1) the cosine similarity and Euclidean distance are equivalent for unit vectors and (2) it is fast and we are clustering a significant number of times.
Automated Parameter Setting. We choose the value of  $K$  purely based on a similarity threshold  $\tau$ .  $\tau$  is introduced to decide whether two clusters belong to the same interpretation by checking if the cosine similarity between two cluster center vectors is greater than  $\tau$ . Intuitively, we should keep increasing  $K$  until there exist no two clusters with the same interpretation. Therefore, we choose  $K$  to be the largest number such that the similarity between any two cluster centers is no more than  $\tau$ .
$$
K = \arg \max  _ {K} \left\{\cos \left(\mathbf {c} _ {i}, \mathbf {c} _ {j}\right) <   \tau \forall i, j \right\} \tag {1}
$$
where  $\mathbf{c}_i$  refers to the  $i$ -th cluster center vector after clustering all contextualized representations into  $K$  clusters. In practice,  $K$  is usually no more than 10. So we increase  $K$  gradually until the constraint is violated.
We pick  $\tau$  based on user-provided seed information instead of hand-tuning, As mentioned, we make two "majority" assumptions: (1) For any seed word, the majority of its occurrences follow the intended interpretation by the user; and (2) The majority of the seed words are not ambiguous — they only have one interpretation. Therefore, for each seed word  $s$ , we take the median of pairwise cosine similarities between its occurrences.
$$
\tau (s) = \operatorname {m e d i a n} \left(\left\{\operatorname {s i m} \left(\mathbf {b} _ {s _ {i}}, \mathbf {b} _ {s _ {j}}\right) | \forall i, j \right\}\right) \tag {2}
$$
Algorithm 1: Corpus Contextualization
Input: Word occurrences  $w_{1}, w_{2}, \ldots, w_{n}$  of the word  $w$ , Seed words  $s_{1}, s_{2}, \ldots, s_{m}$  and their occurrences  $s_{i,j}$ .
Output: Contextualized word occurrences  $\hat{w}_1, \hat{w}_2, \dots, \hat{w}_n$
Obtain  $\mathbf{b}_{w_i}$  and  $\mathbf{b}_{s_{i,j}}$  using BERT.
Compute  $\tau$  follow Equation 3.
K←1
while True do
Run K-Means on  $\{b_{w_i}\}$  for  $(\mathbf{K} + 1)$  clusters.
Obtain cluster centers  $\mathbf{c}_1,\mathbf{c}_2,\ldots ,\mathbf{c}_{K + 1}$
if  $\max_{i,j}\cos (\mathbf{c}_i,\mathbf{c}_j) > \tau$  then
K←K+1
Run K-Means on  $\{b_{w_i}\}$  for K clusters.
Obtain cluster centers  $\mathbf{c}_1, \mathbf{c}_2, \dots, \mathbf{c}_K$ .
for each occurrence  $w_{i}$  do
Compute  $\hat{w}_i$  following Equation 4.
Return  $\hat{w}_i$
Then, we take the median of these medians over all seed words as  $\tau$ . Mathematically,
$$
\tau = \operatorname {m e d i a n} (\{\tau (s) | \forall s \}) \tag {3}
$$
The nested median solution makes the choice of  $\tau$  safe and robust to outliers. For example, consider the word "windows" in the 20Newsgroup corpus. In fact, the word windows has two interpretations in the 20Newsgroup corpus — one represents an opening in the wall and the other is an operating system. We first compute the pairwise similarities between all its occurrences and plot the histogram as shown in Figure 2(a). From this plot, we can see that its median value is about 0.7. We apply the same for all seed words and obtain  $\tau$  following Equation 3.  $\tau$  is calculated to be 0.82. Based on this value, we gradually increase  $K$  for "windows" and it ends up with  $K = 2$ . We visualize its K-Means clustering results using t-SNE (Maaten and Hinton, 2008) in Figure 2(b). Similar results can be observed for the word penalty, as shown in Figure 2(c). These examples demonstrate how our document contextualization works for each word.
In practice, to make it more efficient, one can subsample the occurrences instead of enumerating all pairs in a brute-force manner.
Contextualized Corpus. The interpretation of each occurrence of  $w$  is decided by the cluster-ID to which its contextualized representation belongs.
Specifically, given each occurrence  $w_i$ , the word  $w$
Figure 3: The HAN classifier used in our ConWea framework. It is trained on our contextualized corpus with the generated pseudo-labels.
is replaced by  $\hat{w}_i$  in the corpus as follows:
$$
\hat {w} _ {i} = \left\{ \begin{array}{l l} w & \text {i f K = 1} \\ w \$ j ^ {*} & \text {o t h e r w i s e} \end{array} \right. \tag {4}
$$
$$
j ^ {*} = \arg \max  _ {j = 1} ^ {K} \cos (\mathbf {b} _ {w _ {i}}, \mathbf {c} _ {j})
$$
By applying this to all words and their occurrences, the corpus is contextualized. The pseudo-code for corpus contextualization is shown in Algorithm 1.
4 Pseudo-Label and Text Classifier
We generate pseudo-labels for unlabeled contextualized documents and train a classifier based on these pseudo-labels, similar to many other weakly supervised methods (Agichtein and Gravano, 2000; Riloff et al., 2003; Kuipers et al., 2006; Tao et al., 2015; Meng et al., 2018). These two parts are not the focus of this paper. We briefly introduce them to make the paper self-contained.
Pseudo-Label Generation. There are several ways to generate pseudo-labels from seed words. As proof-of-concept, we employ a simple but effective method based on counting. Each document is assigned a label whose aggregated term frequency of seed words is maximum. Let  $\operatorname{tf}(\hat{w}, d)$  denote term-frequency of a contextualized word  $w$  in the contextualized document  $d$  and  $S_{c}$  represents set of seed words of class  $c$ , the document  $d$  is assigned a label  $l(d)$  as follows:
$$
l (d) = \arg \max  _ {l} \left\{\sum_ {i} t f \left(s _ {i}, d\right) | \forall s _ {i} \in \mathcal {S} _ {l} \right\} \tag {5}
$$
Document Classifier. Our framework is compatible with any text classification model. We use Hierarchical Attention Networks (HAN) (Yang et al., 2016) as an example in our implementation. HAN considers the hierarchical structure of documents (document - sentences - words) and includes an attention mechanism that finds the most important words and sentences in a document while taking the context into consideration. There are two levels of attention: word-level attention identifies the important words in a sentence and sentence level attention identifies the important sentences in a document. The overall architecture of HAN is shown in Figure 3. We train a HAN model on contextualized corpus with the generated pseudo-labels. The predicted labels are used in seed expansion and disambiguation.
5 Seed Expansion and Disambiguation
Seed Expansion. Given contextualized documents and their predicted class labels, we propose to rank contextualized words and add the top few words into the seed word sets. The core element of this process is the ranking function. An ideal seed word  $s$  of label  $l$ , is an unusual word that appears only in the documents belonging to label  $l$  with significant frequency. Hence, for a given class  $\mathcal{C}_j$  and a word  $w$ , we measure its ranking score based on the following three aspects:
- Label-Indicative. Since our pseudo-label generation follows the presence of seed words in the document, ideally, the posterior probability of a document belonging to the class  $\mathcal{C}_j$  after observing the presence of word  $w$  (i.e.,  $P(\mathcal{C}_j|w)$ ) should be very close to  $100\%$ . Therefore, we use  $P(\mathcal{C}_j|w)$  as our label-indicative measure:
$$
\mathbf {L I} (\mathcal {C} _ {j}, w) = P (\mathcal {C} _ {j} | w) = \frac {f \mathcal {C} _ {j} , w}{f \mathcal {C} _ {j}}
$$
where  $f_{\mathcal{C}_j}$  refers to the total number of documents that are predicted as class  $\mathcal{C}_j$ , and among them,  $f_{\mathcal{C}_j,w}$  documents contain the word  $w$ . All these counts are based on the prediction results on the input unlabeled documents.
- Frequent. Ideally, a seed word  $s$  of label  $l$  appears in the documents belonging to label  $l$  with significant frequency. To measure the frequency score, we first compute the average frequency of seed word  $s$  in all the documents belonging to label  $l$ . Since average frequency is unbounded, we apply tanh function to scale it, resulting in
the frequency score,
$$
\mathbf {F} (\mathcal {C} _ {j}, w) = \tanh \bigl (\frac {f _ {\mathcal {C} _ {j}} (w)}{f _ {\mathcal {C} _ {j}}} \bigr)
$$
Here, different from  $f_{\mathcal{C}_j,w}$  defined earlier,  $f_{\mathcal{C}_j}(w)$  is the frequency of word  $w$  in documents that are predicted as class  $\mathcal{C}_j$ .
- Unusual: We want our highly label-indicative and frequent words to be unusual. To incorporate this, we consider inverse document frequency (IDF). Let  $n$  be the number of documents in the corpus  $\mathcal{D}$  and  $f_{\mathcal{D},w}$  represents the document frequency of word  $w$ , the IDF of a word  $w$  is computed as follows:
$$
\mathbf {I D F} (w) = \log \left(\frac {n}{f _ {\mathcal {D} , w}}\right)
$$
Similar to previous work (Tao et al., 2015), we combine these three measures using the geometric mean, resulting in the ranking score  $R(\mathcal{C}_j, w)$  of a word  $w$  for a class  $\mathcal{C}_j$ .
$$
R (\mathcal {C} _ {j}, w) = \left(\mathbf {L I} (\mathcal {C} _ {j}, w) \times \mathbf {F} (\mathcal {C} _ {j}, w) \times \mathbf {I D F} (w)\right) ^ {1 / 3}
$$
Based on this aggregated score, we add top words to expand the seed word set of the class  $\mathcal{C}_j$ .
Seed Disambiguation. While the majority of user-provided seed words are nice and clean, some of them may have multiple interpretations in the given corpus. We propose to disambiguate them based on the ranking. We first consider all possible interpretations of an initial seed word, generate the pseudo-labels, and train a classifier. Using the classified documents and the ranking function, we rank all possible interpretations of the same initial seed word. Because the majority occurrences of a seed word are assumed to belong to the user-specified class, the intended interpretation shall be ranked the highest. Therefore, we retain only the top-ranked interpretation of this seed word.
After this step, we have fully contextualized our weak supervision, including the initial user-provided seeds.
In this section, we evaluate our framework and many compared methods on coarse- and fine-grained text classification tasks under the weakly supervised setting.
Table 1: Dataset statistics.
Dataset	#Docs	# Coarse	#Fine	Avg Doc Len
NYT	13,081	5	25	778
20News	18,846	6	20	400
Following previous work (Tao et al., 2015), (Meng et al., 2018), we use two news datasets in our experiments. The dataset statistics are provided in Table 1. Here are some details.
- The New York Times (NYT): The NYT dataset contains news articles written and published by The New York Times. These articles are classified into 5 wide genres (e.g., arts, sports) and 25 fine-grained categories (e.g., dance, music, hockey, basketball).
- The 20 Newsgroups (20News): The 20News dataset<sup>2</sup> is a collection of newsgroup documents partitioned widely into 6 groups (e.g., recreation, computers) and 20 fine-grained classes (e.g., graphics, windows, baseball, hockey).
We perform coarse- and fine-grained classifications on the NYT and 20News datasets. NYT dataset is imbalanced in both fine-grained and coarse-grained classifications. 20News is nearly balanced in fine-grained classification but imbalanced in coarse-grained classification. Being aware of these facts, we adopt micro- and macro- $\mathrm{F}_1$  scores as evaluation metrics.
6.2 Compared Methods
We compare our framework with a wide range of methods described below:
- IR-TF-IDF treats the seed word set for each class as a query. The relevance of a document to a label is computed by aggregated TF-IDF values of its respective seed words. The label with the highest relevance is assigned to each document.
- Dataless (Chang et al., 2008) uses only label surface names as supervision and leverages Wikipedia to derive vector representations of labels and documents. Each document is labeled based on the document-label similarity.
- Word2Vec first learns word vector representations (Mikolov et al., 2013) for all terms in the corpus and derive label representations by aggregating the vectors of its respective seed words. Finally, each document is labeled with the most
similar label based on cosine similarity.
- Doc2Cube (Tao et al., 2015) considers label surface names as seed set and performs multidimensional document classification by learning dimension-aware embedding.
- WeSTClass (Meng et al., 2018) leverages seed information to generate pseudo documents and refines the model through a self-training module that bootstraps on real unlabeled documents.
We denote our framework as ConWea, which includes contextualizing corpus, disambiguating seed words, and iterative classification & key words expansion. Besides, we have three ablated versions. ConWea-NoCon refers to the variant of ConWea trained without the contextualization of corpus. ConWea-NoSeedExp is the variant of ConWea without the seed expansion module. ConWeaWSD refers to the variant of ConWea, with the contextualization module replaced by Lesk algorithm (Lesk, 1986), a classic Word-sense disambiguation algorithm (WSD).
We also present the results of HAN-Supervised under the supervised setting for reference. We use 80-10-10 for train-validation-test splitting and report the test set results for it. All weakly supervised methods are evaluated on the entire datasets.
6.3 Experiment Settings
We use pre-trained BERT-base-uncased<sup>3</sup> to obtain contextualized word representations. We follow Devlin et al. (2019) and concatenate the averaged word-piece vectors of the last four layers.
The seed words are obtained as follows: we asked 5 human experts to nominate 5 seed words per class, and then considered the majority words (i.e.,  $>3$  nominations) as our final set of seed words. For every class, we mainly use the label surface name as seed words. For some multi-word class labels (e.g., "international business"), we have multiple seed words, but never exceeds four per each class. The same seed words are utilized for all compared methods for fair comparisons.