SlowGuess/NAACL_2022_OCR huggingface dataset:
Ziqian Zeng $^{1,2}$ , Weimin Ni $^{1}$ , Tianqing Fang $^{2}$ , Xiang Li $^{3}$ , Xinran Zhao $^{4}$ , and Yangqiu Song $^{2}$
<sup>1</sup>Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, China
$^{2}$ Department of CSE, HKUST, Hong Kong, China
$^{3}$ Department of Computer Science, University of Illinois, Urbana Champaign, US
<sup>4</sup>Computer Science Department, Stanford University, US
zqzeng@scut.edu.cn, mewmn@mail.scut.edu.cn, xiang12@illinois.edu
xzhaoar@stanford.edu, {tfangaa, yqsong}@cse.ust.hk
Solving text classification in a weakly supervised manner is important for real-world applications where human annotations are scarce. In this paper, we propose to query a masked language model with cloze style prompts to obtain supervision signals. We design a prompt which combines the document itself and "this article is talking about [MASK]." A masked language model can generate words for the [MASK] token. The generated words which summarize the content of a document can be utilized as supervision signals. We propose a latent variable model to learn a word distribution learner which associates generated words to pre-defined categories and a document classifier simultaneously without using any annotated data. Evaluation on three datasets, AGNews, 20Newsgroups, and UCINews, shows that our method can outperform baselines by  $2\%$ ,  $4\%$ , and  $3\%$ .
Text classification is a fundamental task in Natural Language Processing (NLP) with diverse real-world applications such as identifying relevant documents of a case in legal proceedings (Roitblat et al., 2010), and classifying victim's requests (e.g., food, shelter, and medical aids) on social media platforms during earthquakes (Caragea et al., 2011). Current state-of-the-art text classification methods (Zhang et al., 2015; Zhou et al., 2016; Johnson and Zhang, 2017) still need a large number of annotated data. However, in the real world, naturally annotated data are rare and human annotations are expensive. Solving the text classification task without using annotated data but exploiting inexpensive supervision signals is worth investigation.
In the weakly supervised setting, any annotated document is not accessible, but inexpensive supervision signals such as label surface names or keywords can be used. Existing weakly supervised text classification methods (Meng et al., 2018, 2019;
Mekala and Shang, 2020; Meng et al., 2020) first used seed keywords to retrieve more keywords, and then created pseudo labels for documents and then train a model in a "standard" supervised learning manner. In previous work, supervision signals are restricted to a small set of keywords from documents contents.
Recent work shows that prompts can probe knowledge from PLMs (Devlin et al., 2019; Radford et al., 2019) and the knowledge can provide supervision signals to solve different NLP tasks including relation extraction (Shin et al., 2020), question answering (Petroni et al., 2020), and summarization (Radford et al., 2019). For example, (Petroni et al., 2019) solved the knowledge base completion task by querying an MLM with a prompt "Alan Turing was born in [MASK]." Using prompts to generate supervision signals for text classification is worth exploring.
We propose to query an MLM with a prompt which combines the document itself and "this article is talking about [MASK]"., and use generated words for the [MASK] token as supervision signals. For example, in Figure 1, given a prompt "The radio telescope at arecibo observatory will begin mapping the known galaxy on friday, scientists said. This article is talking about [MASK]", an MLM predicts "astronomy", "galaxies", "radio", "science", and "galaxy" for the [MASK] token. These words summarize the topic of the document. Hence, they can be used as supervision signals. Besides generating signal words, an intuitive approach to obtain supervision signals is by extracting important words from documents. We will compare two types of supervision signals.
After obtaining signal words, we need to associate these words to pre-defined categories. We propose a latent variable model (WDDC) to learn a Word Distribution learner and a Document Classifier simultaneously without using any annotated data. A word distribution learner aims to learn
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2295 - 2305
Figure 1: We combine a document and a cloze style sentence "This article is talking about [MASK]" to query a masked LM. It generates a set of words for the [MASK] token. These words are likely to summarize the topic of a document. After obtaining words such as "astronomy" and "galaxies", human beings can easily infer that this article is talking science rather than business because we know these words are frequently used in science topic. Word distributions given pre-defined categories bridge supervision signals (generated words) and our goal (the category of a document). The proposed model (WDDC) can learn word distributions given pre-defined categories and a document classifier simultaneously.
a probability of a generated word  $w$  given a category  $c$ , i.e.,  $p(w|c)$ . A document classifier aims to learn a probability of a category  $c$  given a document  $x$ , i.e.,  $p(c|x)$ . These two goals could be optimized simultaneously via maximizing the log-likelihood of generated words by introducing the category as a latent variable. In our latent variable model, a word distribution learner and a document classifier can be parameterized by any neural network.
Our contributions are summarized as follows,
- We propose to query an MLM using a prompt which combines the document and a cloze style sentence "this article is talking about [MASK]". We use generated words for the [MASK] token as supervision signals in the weakly supervised text classification task.
- We propose a latent variable model (WDDC) to learn a word distribution learner which associates generated words to pre-defined categories and a document classifier simultaneously without using any annotated data.
- The experimental results show that the proposed method WDDC can outperform other weakly supervised baselines in three datasets.
The code is available at https://github.com/ HKUST-KnowComp/WDDC.
In this section, we review the related work on querying an MLM with prompts, weakly supervised text
classification, zero-shot text classification, and variational methods.
Querying an MLM with Prompts. Querying an MLM with cloze style prompts provides a new direction to solve some NLP tasks in an unsupervised manner. (Petroni et al., 2019) queried an MLM using manually designed prompts to solve a knowledge base completion task. For example, in order to complete the missing entity X in (Alan Turing, born in, X), they designed a prompt "Alan Turing was born in [MASK]" to query an MLM. The generated word for the [MASK] token can be directly used to complete the missing fact. By querying language models, some NLP tasks such as relation extraction (Shin et al., 2020), question answering (Radford et al., 2019; Petroni et al., 2020), summarization (Radford et al., 2019) could be solved in an unsupervised manner. However, not all NLP tasks can directly use generated words from an MLM in downstream tasks. Some tasks such as sentiment analysis and textual entailment (Shin et al., 2020) need more steps for inference. For example, in the sentiment analysis task, (Shin et al., 2020) used annotated data to train a classifier that links generated words to pre-defined categories. Our work does not require annotated data for inference.
Weakly Supervised Text Classification. In the weakly supervised text classification task, any labeled documents are not allowed, but label sur
face names or limited word-level descriptions of each category can be used. Dataless (Chang et al., 2008; Song and Roth, 2014) used Explicit Semantic Analysis (ESA) vectors (Gabrilovich et al., 2007) to represent label name and documents. Predictions are based on the label-document similarity. Recently, (Meng et al., 2018, 2019; Mekala and Shang, 2020; Meng et al., 2020; Schick and Schütze, 2021; Schick and Schütze, 2021; Zhang et al., 2022) trained neural text classifiers in an weakly supervised manner. They generated pseudo labels for documents to pre-train a neural classifier and then performed self-training on unlabeled data for model refinement. LOTClass (Meng et al., 2020) is relevant to our work because they also used pre-trained language models. They used a LM to retrieve a set of semantically correlated words for each class, and then fine-tuned the LM to predict these words. Finally, they performed self-training on unlabeled data. Our work is different from LOTClass because we obtain supervision signals by querying an MLM with cloze style prompts and we propose a latent variable model to learn document classifier rather than using the self-training procedure. PRBOOST (Zhang et al., 2022) is also relevant to our work because they also use prompts to generate weak labels. PRBOOST first generated rules by using a small amount of labeled data, then asked human annotators to select high-quality rules to generate week labels. Finally, they trained a new model in a self-training manner. Our work is different from PRBOOST because our method associates predicted words with labels in an unsupervised manner while PRBOOST maps prompting based rules to labels by involving human feedback.
Zero-Shot Text Classification. In zero-shot learning settings, the classes covered by training instances and the classes we aim to classify are disjoint. Zero-shot learning text classification methods (Xia et al., 2018; Rios and Kavuluru, 2018; Zhang et al., 2019; Liu et al., 2019) generalized seen classes to unseen classes by learning semantic relationships between classes and documents via embeddings or semantic knowledge sources. However, zero-shot learning still requires annotated data for the seen classes training. We cannot apply zero-shot learning methods to weakly supervised settings where no annotated document is available.
Variational Methods. Variational autoencoders (Kingma and Welling, 2014; Rezende et al., 2014)
consists of an encoder and a decoder. The encoder estimates posterior probabilities and the decoder estimates the reconstruction likelihood given a latent variable. The objective function is to maximize the reconstruction likelihood of the observed variable. The latent variable in VAEs is continuous variable. Recently, many research works (Titov and Khoddam, 2015; Marcheggiani and Titov, 2016; Suster et al., 2016; Zhang et al., 2018; Chen et al., 2018; Zeng et al., 2019; Liang et al., 2019) use VAEs to solve different NLP tasks such as relation discovery, question answering, sentiment classification, etc. In above works, the latent variables are discrete variables. For example, (Marcheggiani and Titov, 2016) aimed to solve unsupervised open-domain relation discovery. The objective function is to reconstruct the likelihood of two entities. They introduced relation as the latent variable. The encoder is a relation classifier, which predicts a semantic relation between two entities. The decoder reconstructs entities given the predicted relation. Our method is also based on VAEs with a discrete latent variable but the estimated probabilities and the objective function are different.
In this section, we first introduce how to obtain supervision signals from an MLM and document itself, and then we introduce a latent variable model to learn a word distribution learner and a document classifier simultaneously.
3.1 Supervision Signals
3.1.1 Signal Words
Given a document, our goal is to obtain topic relevant words which are used as supervision signals. To achieve this, we append a cloze style sentence to the document at the end as a prompt. A prompt is designed as “[CLS] + document + This article is talking about [MASK]. + [SEP].” The [MASK] token serves as a placeholder for a topic relevant word which can summarize the document. It mimics the reading comprehension task which is using a word to summarize the content of a document. We select top  $k$  generated words as supervision signals.
Instead of generating signal words, a natural way to obtain supervision signals is by extracting words from the document. To achieve this, we extract all nouns and proper nouns in the document using part-of-speech tagger (Kristina et al., 2003). Since
Table 1: Signal words from an MLM and from the document(Doc).
Text	Label	Signal Words
The world ’s top two players roger federer and andy roddick reached the semifinals friday at the thailand open.	Sports	MLM: tennis, thailand, federer, seeds, wimbledonDoc: world, players, federer, andy, roddick, semifinals, friday, thailand, open
These circuits abound in most electronic project books. It has LED indicators also.	Science	MLM: circuits, computers, electronics, computing, graphsDoc: circuits, project, books, LED, indicators
Scientists discover a genetic indicator that could help prevent suicides.	Health	MLM: suicide, genetics, cancer, hiv, healthDoc: scientists, indicator, suicides
most of the generated words from an MLM are nouns and proper nouns, so we only extract words with two types of part-of-speech.
Table 1 shows top 5 predictions from an MLM (BERT (Devlin et al., 2019)) given prompts and extracted nouns and proper nouns from documents. For the first document, an MLM can infer that it is talking about a tennis match although "tennis" does not appear in the document. It also generates some relevant words such as "wimbledon." In this case, the MLM is better than extraction. For the second document, the first word from the MLM precisely summarizes the document. However, the MLM also generates a few words which are related to computer. Unfortunately computer is also a category in this dataset. Compared to the MLM, the extracting way is safer in this case. For the third document, an MLM generates "health" which is an exact match of the label surface name although "cancer" and "hiv" are not faithful to the original document. We will evaluate generation and extraction methods in the experiment.
3.1.2 Remove Non-discriminated Words
Words generated from an MLM are not always category discriminated. Non-discriminated words can harm the performance of inference. The intuition of removing non-discriminated words is that if some words appear in different categories with similar frequency, then it is possible that these words are not category-discriminated. Since we cannot access labels, the label in the following computation means the pseudo label. The pseudo label generation process is shown in section 3.1.3. Inspired by category-indicative measurement, (Mekala and Shang, 2020), we define category-indicative index:
$$
C I I (c _ {i}, w) = \frac {f (c _ {i} , w)}{f (c _ {i})}, \tag {1}
$$
where  $f(c_{i},w)$  is the number of occurrences of the signal word  $w$  in the documents which are labeled as  $c_{i}$ , and  $f(c_{i})$  is the total number of occurrences of all signal words in documents which are labeled as  $c_{i}$ .
We define category-indicative ratio as,
$$
C I R (w) = \frac {C I I (c _ {i} , w)}{C I I (c _ {j} , w)}, \tag {2}
$$
where  $CII(c_i, w)$  is the maximum value among all categories,  $CII(c_j, w)$  is the second maximum value all categories. Larger value of  $CIR(w)$  indicates  $w$  is more discriminated. If  $CII(c_j, w)$  is equal to 0, we will assign a large value to  $CIR(w)$ . If  $CIR(w) < t$ , we consider  $w$  is not discriminated and we remove  $w$  from signal words set.
3.1.3 Pseudo Label Generation
We assign pseudo labels to data based on label-word similarity. We represent a word using static representation which is introduced by (Mekala and Shang, 2020). Given a word  $w$ , static representation  $\mathcal{SR}(w)$  is computed by averaging the contextualized embeddings of all its occurrences in the corpus. The label-word similarity is the cosine similarity between the static representation of the label surface name and the static representation of signal words. If the label surface name or the supervision signal contains more than one word, we take the average of the static representations of all words. We assign a sample with the pseudo label which yields the maximum similarity value among all classes. And the similarity value should be greater
than a threshold  $\gamma$ . Setting a threshold can result in more accurate pseudo label assignments although the size of pseudo labeled data will shrink.
To summarize, there are three steps to obtain clean signal words: (1) Obtain signal words from an MLM or a document. (2) Generate pseudo labels. (3) Remove signal words which have low category-indicative ratio values.
After getting clean signal words, we then propose a latent variable model to learn a word distribution learner and a document classifier simultaneously.
Since there is no annotated data available, in order to best explain the observed data, i.e., signal words, the objective of our model is to maximize the log-likelihood of signal words. The ultimate goal is to identify the category of a document, hence, we introduce a latent variable  $C$  representing the category, into the objective function. Further, by applying Jensen's inequality (Jensen et al., 1906), we can derive an evidence lower bound (ELBO) of the log-likelihood. We define the objective function as follows,
$$
\begin{array}{l} \mathcal {L} _ {o} = \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \log p (w _ {r}) \\ = \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \log \sum_ {c} p (w _ {r}, c) \\ = \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \log \sum_ {c} q (c | x) \left[ \frac {p \left(w _ {r} , c\right)}{q (c | x)} \right] \\ \geq \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \sum_ {c} q (c | x) \left[ \log \frac {p \left(w _ {r} , c\right)}{q (c | x)} \right] \\ = \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \mathbb {E} _ {q (C | x)} \left[ \log p \left(w _ {r} | c\right) p (c) \right] \\ - \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \mathbb {E} _ {q (C | x)} [ \log q (c | x) ], \tag {3} \\ \end{array}
$$
where  $x$  is a document,  $X$  is a set of documents,  $\mathcal{R}_x$  is the set of signal words of document  $x$ ,  $w_r$  is a signal word,  $C$  is a discrete random variable representing the category of a document,  $c$  is a possible value of variable  $C$ . For example,  $c$  can be science or business.
There are three probabilities in the Eq. (3).  $q(c|x)$  is the document classifier which is our ultimate goal.  $p(w_r|c)$  is the word distribution learner which estimates the probability distribution of all signal words given a possible value  $c$ . We use neural networks to parameterize  $p(w_r|c)$  and  $q(c|x)$ .
$p(C)$  is a prior probability distribution. Since there are no annotated data available, we cannot estimate  $p(C)$ . Hence we assume it is a uniform distribution, and  $p(c)$  becomes a constant.
3.2.1 Word Distribution Learner
The word distribution learner aims to estimate the probability of a signal word  $w_{r}$  given a possible value of category  $c$ . It is defined as follows,
$$
p \left(w _ {r} \mid c\right) = \frac {\exp \left(\mathbf {v} _ {c} ^ {T} \mathbf {w} _ {r}\right)}{\sum_ {w _ {r ^ {\prime}}} \exp \left(\mathbf {v} _ {c} ^ {T} \mathbf {w} _ {r ^ {\prime}}\right)}, \tag {4}
$$
where  $\mathbf{v}_c$  is a trainable vector associated with  $c$  and  $\mathbf{w}_r$  is the trainable word embedding of signal word  $w_r$ . The intuition is that if a word (e.g., "scientist") appears frequently under the science category, the corresponding inner-product value is high, otherwise it is low.
Eq. (4) requires the summation over all signal words. Since the size of the word vocabulary can be large, we use the negative sampling technique (Mikolov et al., 2013) to approximate Eq. (4). Specifically, we approximates  $\log p(w_r|c)$  as follows,
$$
\log \sigma \left(\mathbf {v} _ {c} ^ {T} \mathbf {w} _ {r}\right) + \sum_ {w _ {r} ^ {\prime} \in \mathcal {N}} \log \left(1 - \sigma \left(\mathbf {v} _ {c} ^ {T} \mathbf {w} _ {r} ^ {\prime}\right)\right), \tag {5}
$$
where  $w_{r}^{\prime}$  is a negative sample in the vocabulary,  $\mathcal{N}$  is the set of negative samples and  $\sigma (\cdot)$  is the sigmoid function.
The objective function with an approximated word distribution learner is defined as follows,
$$
\begin{array}{l} \mathcal {L} = \sum_ {x \in X} \sum_ {w _ {r} \in \mathcal {R} _ {x}} \mathbb {E} _ {q (C | x)} [ \log \sigma (\mathbf {v} _ {c} ^ {T} \mathbf {w} _ {r}) \\ \left. + \sum_ {w _ {r} ^ {\prime} \in \mathcal {N}} \log \left(1 - \sigma \left(\mathbf {v} _ {c} ^ {T} \mathbf {w} _ {r} ^ {\prime}\right)\right) + \log p (c) \right] \\ - \mathbb {E} _ {q (C | x)} [ \log q (c | x) ]. \tag {6} \\ \end{array}
$$
3.2.2 Document Classifier
Most existing deep neural models (DNN) can be used to parameterize  $q(C|x)$ . As long as the input of DNNs is a document, and the output is a probability distribution of category  $C$ . Since models which involve latent variables are difficult to optimize, we give a good initialization of the document classifier. We pre-train the document classifier using pseudo labeled data to initialize it.
Table 2: Statistics and label surface names in AGNews, 20Newsgroup, and UCINews.
Datasets	# Train	# Dev	# Test	# Class	Label Surface Names
AGNews	108,000	12,000	7,600	4	politics, sports, business, technology
20Newsgroup	14,609	1,825	1,825	6	computer graphics, sports car, science electronics encryption health aerospace, politics gun homosexuality, religion atheist christianity, sale
UCINews	26,008	2,560	27,556	4	entertainment, technology, business, health
Table 3: Vocabulary size of signal words that are generated from an MLM and that are extracted from the document (Doc) after removing non-discriminated words.
Dataset	MLM	Doc
AGNews	724	584
20Newsgroup	1,037	413
UCINews	584	442
In this section, we show the empirical performance of our method on the text classification task.
We evaluate all methods on three datasets.
(1) AGNews consists of news articles. It is constructed by (Zhang et al., 2015), which has been gathered from more than 2000 news sources in more than one year of activity.
(2) 20Newsgroup comprises around 18,000 posts. It is originally collected by (Lang, 1995). We perform text classification on coarse-grained topics. It is an unbalanced dataset.
(3) UCINews consists of news pages collected from a web aggregator. It is maintained by (Dua and Graff, 2017).
Table 2 provides statistics and label surface names of three datasets. In 20Newgroups, we expand label surface names by combining fine-grained label surface names under the same coarse-grained category.
Table 3 shows the vocabulary size of signal words that are generated from an MLM and that extracted from the document (Doc) after removing non-discriminated words.
Dataless (Chang et al., 2008) is performed based on vector similarity between documents and label surface names using explicit semantic analysis representation. The prediction is the category that yields the maximum cosine similarity.
Label-Word Similarity is performed based on the vector similarity between words generated from an MLM and label surface names using the static representation. The prediction is the category that yields the maximum cosine similarity.
Pseudo-CNN assigns pseudo labels to documents in the training set based on label-word similarity. We train a CNN model using pseudo labeled samples in the training set. More details are provided in section 4.5.
Pseudo-BERT trains BERT (Devlin et al., 2019) BERT-base-uncased using the same pseudo labeled data as Pseudo-CNN. More details are provided in section 4.5.
WeSTClass (Meng et al., 2018) first generates pseudo labels for documents which contain user-provided keywords. It pre-trains a neural network using pseudo samples as the training set and then performs a self-training process.
LOTClass (Meng et al., 2020) constructs a category vocabulary for each class, using a pre-trained LM. The vocabulary contains words that are relevant to the label name. LOTClass fine-tunes an LM via word-level category prediction task, and then performs self-training on unlabeled data to generalize the model.