SlowGuess/NAACL_2022_OCR huggingface dataset:
ConWea (Mekala and Shang, 2020) leverages contextualized representations of word occurrences and seed word information to automatically distinguish multiple senses of the same word. The
Table 4: Micro F1 and macro F1 scores of all methods on AGNews, 20Newsgroup, and UCINews.
Methods\Datasets	AGNews	20Newsgroup	UCINews
Micro	Macro	Micro	Macro	Micro	Macro
Dataless (Chang et al., 2008)	0.6855	0.6844	0.5000	0.4700	0.6248	0.6253
Label-Word Similarity	0.7917	0.7884	0.7310	0.6390	0.6447	0.6390
Pseudo-CNN	0.8265	0.8237	0.7973	0.6825	0.7598	0.7632
Pseudo-BERT	0.8249	0.8219	0.8153	0.6896	0.7824	0.7820
WeSTClass (Meng et al., 2018)	0.8279	0.8268	0.5300	0.4300	0.6983	0.6999
LOTClass (Meng et al., 2020)	0.8659	0.8656	0.6121	0.5586	0.7320	0.7236
ConWea (Mekala and Shang, 2020)	0.7443	0.7401	0.6200	0.5700	0.3293	0.3269
X-Class (Wang et al., 2021)	0.8574	0.8566	0.6515	0.6316	0.6885	0.6962
WDDC-MLM	0.8826	0.8825	0.8121	0.6882	0.8150	0.8134
WDDC-Doc	0.8668	0.8657	0.8570	0.8250	0.7814	0.7772
CNN (Kim, 2014)	0.9025	0.9025	0.9397	0.9310	0.9002	0.8998
BERT (Devlin et al., 2019)	0.9305	0.9306	0.9660	0.9569	0.9313	0.9315
contextualized corpus is used to train the classifier and expand seed words iteratively.
X-Class (Wang et al., 2021) leverages BERT representations to generate class-oriented document presentations, then generates document-class pairs by clustering, and then fed pairs to a supervised model to train a text classifier.
CNN(Kim, 2014) trains a text CNN using annotated training data in a supervised manner. It is an upper bound of weakly supervised methods.
BERT fine-tunes BERT BERT-base-uncased (Devlin et al., 2019) using annotated training data. It is an upper bound of weakly supervised methods.
WDDC We use a text CNN(Kim, 2014) as the document classifier. Instead of randomly initializing CNN, we pre-train CNN using Pseudo-CNN.
WDDC-MLM uses the supervision signals from an MLM while WDDC-Doc uses the supervision signals from the document itself.
Table 4 shows that our method outperforms weakly supervised baselines by  $2\%$ ,  $4\%$ , and  $3\%$  in AGNews, 20Newsgroup, and UCINews, respectively. The gaps between the upper bound CNN and our method are  $2\%$ ,  $8\%$ , and  $8\%$  in AGNews, 20Newsgroup, and UCINews, respectively. There are still large performance gaps on 20Newsgroup and UCINews.
Label-Word Similarity and Dataless both use vector similarity for prediction. Label-Word Similarity consistently outperforms Dataless, which
shows that words generated from an MLM are useful compared with documents. The performance of Pseudo-BERT is comparable with WeSTClass in AGNews and better than any other baselines in 20Newsgroup and UCINews, which also shows the effectiveness of our pseudo label generation technique. In 20Newsgroup, Macro F1 scores are lower than Micro F1 scores in Pseudo-CNN, Pseudo-BERT, and WDDC-MLM methods. We found that the number of pseudo labeled data of sale category is much lower than other categories. So CNN does not have enough pseudo labeled data to learn the sale category. The F1 score of sale category is lower.
In AGNews and UCINews, WDDC-MLM outperforms WDDC-Doc by  $2\%$  and  $3\%$ , respectively, which shows that signal words from an MLM are more useful than extracted words from a document. But in 20Newsgroup, WDDC-Doc outperforms WDDC-MLM by  $4\%$ . The possible reason is that some categories in 20Newsgroup are not completely disjoint. According to general knowledge, encryption is a field of computer, and computer is a field of science. But in 20Newsgroup (refer to Table 2), science and encryption belong to one class, and computer belongs to another class. MLMs can capture general knowledge from training corpora such as Wikipedia. When given a document talking about encryption, an MLM probably generates words about encryption as well as computer. In this circumstance, generated words are misleading while extracted words are clean. We have detailed
Table 5: Mean and standard deviation of micro and macro F1 scores on 5 independent runs.
MethodDataset	WDDC	Baselines
Micro F1	Macro F1	Micro F1	Macro F1
Mean	Std	Mean	Std	Mean	Std	Mean	Std
AGNews	0.8826	0.0013	0.8825	0.0013	0.8630	0.0038	0.8626	0.0037
20Newsgroup	0.8570	0.0023	0.8250	0.0033	0.8153	0.0131	0.6896	0.0063
UCINews	0.8150	0.0012	0.8134	0.0014	0.7824	0.0141	0.7820	0.0148
Table 6: Some incorrect predictions in AGNews, 20Newsgroup, and UCINews.
Dataset	Text	Prediction	Ground Truth	Signal Words (MLM)
AGNews	Microsoft and Palmone today announced a partnership that will likely have a negative impact on good technology, a well capitalized startup.	technology	business	windows, microsoft, business, security, technology, linux, privacy
20News-group	For the system, or ‘family’, key would appear to be cryptographically useless. ... The same key is used for both encryption and decryption.	computer	science	software, virus, linux, encryption , ibm , nsa
UCINews	Paraplegic teenager to kick off World Cup thanks to robot suit.	entertainment	health	football, sport, soccer cricket, tennis
analysis in section 4.4.
Table 5 shows mean and standard deviation of micro and macro F1 scores of WDDC and best baselines on 5 independent runs. We also conducted t-tests, and p-values are all less than 0.001. We concluded that our method outperforms baselines significantly. Baselines refers to LOTClass, Pseudo-BERT, and Pseudo-BERT on AGNews, 20Newsgroup, and UCINews respectively.
4.4.1 Analysis of Incorrect Predictions
Table 6 shows some incorrect predictions. In the first example, some words in the original document such as "partnership" and "startup" indicate business while other words such as "Microsoft" and "technology" indicate technology. Signal words generated from an MLM are all related to technology. In AGNews dataset, there are a number of samples talking about the stock price of technology companies or cooperation between technology companies. An MLM inclines to focus on either technology or business and ignore the other one. Although the extraction method can cover all words,
the model is likely to be confused when signal words are related to two categories. In the second example, an MLM generates words related to encryption as well as computer. Generated words make sense because according to general knowledge, encryption is related to computer. Unfortunately, most of the signal words from an MLM are related to computer except one word "encryption." WDDC-MLM is likely to predict it as computer. Signal words extracted from the document are "encryption" and "key", which are more likely to guide the model to predict the correct category. In the third example, an MLM generates words that are all about sports because the term "World Cup" appears in the original document. The modifier "paraplegic" plays an important role in identifying the true category. Both generation and extraction methods fail to capture that.
4.4.2 Analysis of Word Distribution Learner
The word distribution learner aims to estimate the probability of a signal word  $w_{r}$  given a possible value of category  $c$ , i.e.,  $p(w_{r}|c)$ . A good word distribution learner should assign a high probability
Table 7: Top 15 signal words that have large inner product values with different latent variable vectors respectively on AGNews dataset. Signal words are generated by an MLM.
Label	Signal Words
Politics	iraq, syria, haiti, israel, murder, baghdad, suicide,torture, war, islam, iran, terrorist, afghanistan, religion, terrorism
Sports	injury, racing, baseball, soccer, boxing, player, relegation,cricket, quarterback, england, basketball, doping, football, golf, tennis
Business	profit, market, finance, agriculture, bankruptcy, energy, money,growth, price, insurance, recession, airline, oil, risk, inflation
Technology	ipod, genetics, encryption, microsoft, internet, hacking, virus,biotechnology, science, copyright, itunes, nasa, evolution, space, astronomy
to category-indicated words, so that by maximizing Eq. (3), a large value of  $p(w_r|c)$  leads to a large value of  $q(c|x)$ , which means if a document contains indicative words to category  $c$ , it possibly belongs to category  $c$ . Table 7 shows top 15 signal words that have large inner product values with different latent variable vectors respectively on AGNews dataset. As shown in Table 7, the selected words are category-indicated. For example, in the politics category, all words are about terrorism, war, and places where wars broke out, which are relevant to the politics topic. The word distribution learner can be consider as a category-indicated keywords expansion module.
4.5 Implementation
We use the BERT (bert-base-uncased) model to obtain supervision signals in AGNews and 20Newsgroup. We use the BERT (bert-base-cased) to obtain supervision signals in UCINews which contains many acronyms such as WHO and PTSD. We select top 20 predictions as supervision signals three datasets. To remove non-discriminated words, we set the threshold  $t$  to 2 in three datasets.
In the pseudo label generation process, we set the threshold  $\gamma$  to 0.6, 0.75, and 0.55 in AGNews, 20Newsgroup, and UCINews, respectively. Those pseudo labeled training data are used in PseudoCNN and Pseudo-BERT. A higher  $\gamma$  may result in more accurate pseudo labels. But we need to balance the size of pseudo labeled data because it will shrink when  $\gamma$  increases.
To train WDDC, in each batch, we randomly select 5 signal words among all signal words of a document. The number of negative samples in
the approximated word distribution learner is set to 10. For Pseudo-CNN, CNN, and WDDC methods, the CNN architectures are the same. Four different filter sizes  $\{2,3,4,5\}$  are applied. A max-pooling layer is applied to each convolutional layer, and each convolutional layer has 100 filters. The maximum length of input in the CNN is set to 64, 128, and 64 in AGNews, 20Newsgroup, and UCINews, respectively. The input in the CNN is contextualized embeddings generated by BERT (bert-base-uncased).
For WeSTClass, we use a CNN as the document classifier because it empirically outperforms LSTM in WeSTClass. The CNN architecture we used here is the same as the one described in their paper. We try our best to find good keywords and tune hyper-parameters for WeSTClass and LOTClass. For all methods, we tune hyper-parameters on development sets.
To solve the weakly supervised classification task, we propose to query a masked language model with cloze style prompts to obtain supervision signals. We design a prompt which combines the document itself and "this article is talking about [MASK]." The predictions for the "[MASK]" token are considered as supervision signals because they summarize the content of documents. We propose a latent variable model (WDDC) to learn word distributions given pre-defined categories and a neural document classifier simultaneously without using any annotated data. Evaluation on three datasets shows that our method can outperform weakly supervised learning baselines.
The authors of this paper were partially supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to Hong Kong Mediation and Arbitration Centre (HKMAAC) and California University, School of Business Law & Technology (CUSBLT), and the Jiangsu Province Science and Technology Collaboration Fund (BZ2021065). We also thank the anonymous reviewers for their valuable comments and suggestions that help improve the quality of this manuscript.
Cornelia Caragea, Nathan J. McNeese, Anuj R. Jaiswal, Greg Traylor, Hyun-Woo Kim, Prasenjit Mitra, Ding-hao Wu, Andrea H. Tapia, C. Lee Giles, Bernard J. Jansen, and John Yen. 2011. Classifying text messages for the haiti earthquake. In Proceedings of ISCRAM.
Mingwei Chang, Lev Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic rep-mentation: dataless classification. In Proceedings of AAAI, pages 830-835.
Wenhu Chen, Wenhan Xiong, Xifeng Yan, and William Wang. 2018. Variational knowledge graph reasoning. In Proceedings of NAACL-HLT, pages 1823-1832.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Dheeru Dua and Casey Graff. 2017. UCI machine learning repository.
Evgeniy Gabrilovich, Shaul Markovitch, et al. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In Proceedings of IJCAI, pages 1606-1611.
Johan Ludwig William Valdemar Jensen et al. 1906. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica, 30:175-193.
Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of ACL, pages 562-570.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746-1751.
Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proceedings of ICLR.
Toutanova Kristina, Klein Dan, Manning Christopher, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of NAACL-HLT, pages 252-259.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of ICML, pages 331-339.
Yan Liang, Xin Liu, Jianwen Zhang, and Yangqiu Song. 2019. Relation discovery with out-of-relation knowledge base as supervision. In Proceedings of NAACL-HLT, pages 3280-3290.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert Y. S. Lam. 2019. Reconstructing capsule networks for zero-shot intent classification. In Proceedings of EMNLP, pages 4798-4808.
Diego Marcheggiani and Ivan Titov. 2016. Discrete-state variational autoencoders for joint discovery and factorization of relations. Transactions of the Association for Computational Linguistics, 4:231-244.
Dheeraj Mekala and Jingbo Shang. 2020. Contextualized weak supervision for text classification. In Proceedings of ACL, pages 323-333.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of CIKM, pages 983-992.
Yu Meng, Jiaming Shen, Chao Zhang, , and Jiawei Han. 2019. Weakly-supervised hierarchical text classification. In Proceedings of AAAI, pages 6826-6833.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of EMNLP, pages 9006-9017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS, pages 3111-3119.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. In Proceedings of AKBC.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In Proceedings of EMNLP, pages 2463-2473.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of ICML, pages 1278-1286.
Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In Proceedings of EMNLP, pages 3132-3142.
Herbert L. Roitblat, Anne Kershaw, and Patrick Oot. 2010. Document categorization in legal electronic discovery: computer classification vs. manual review. Journal of the Association for Information Science and Technology, 61(1):70-80.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze questions for few shot text classification and natural language inference. In Proceedings of EACL, pages 255-269.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of NAACL-HLT, pages 2339-2352.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of EMNLP, pages 4222-4235.
Yangqiu Song and Dan Roth. 2014. On dataless hierarchical text classification. In Proceedings of AAAI, pages 1579-1585.
Simon Šuster, Ivan Titov, and Gertjan van Noord. 2016. Bilingual learning of multi-sense embeddings with discrete autoencoders. In Proceedings of NAACL-HLT, pages 1346-1356.
Ivan Titov and Ehsan Khoddam. 2015. Unsupervised induction of semantic roles within a reconstruction-error minimization framework. In Proceedings of NAACL-HLT, pages 1-10.
Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2021. X-class: Text classification with extremely weak supervision. In Proceedings of NAACL-HLT, pages 3043-3053.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S. Yu. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of EMNLP, pages 3090-3099.
Ziqian Zeng, Wenxuan Zhou, Xin Liu, and Yangqiu Song. 2019. A variational approach to weakly supervised document-level multi-aspect sentiment classification. In Proceedings of NAACL-HLT, pages 386-396.
Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019. Integrating semantic knowledge to tackle zero-shot text classification. In Proceedings of NAACL-HLT, pages 1031-1040.
Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. 2022. *Prboost: Prompt-based rule discovery and boosting for interactive weakly-supervised learning*. In *Proceedings of ACL*.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of NeurIPS, pages 649-657.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In Proceedings of AAAI, pages 6069-6076.
Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. In Proceedings of COLING, pages 3485-3495.