SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 3: Example seed words and class names for methods for NYT-Small.
class	Seed words	Class name for X-Class
arts	dance,art, ballet,museum	arts
business	shares,stocks, markets,trading	business
rences of the same word. For supervised model training, we follow BERT fine-tuning (Wolf et al., 2019) with all hyper-parameters unchanged.
For both WeSTClass and ConWea, we have tried our best to find keywords for the new datasets. Table 3 shows an example on the seed words selected for them on the NYT-Small dataset. For LOTClass, we tune their hyper-parameters match_threshold and mcp_epoch, and report the best performance during their self-train process.
4.4 Performance Comparison and Analysis
From Table 2, one can see that X-Class achieves the best overall performance. It is only  $1\%$  to  $2\%$  away from LOTClass and ConWea on AGNews and NYT-Topics, respectively. Note that, ConWea consumes at least 3 keywords per class.
It is noteworthy that X-Class can approach the supervised upper bound to a small spread, especially on the NYT-Small dataset.
Ablation on Modules. X-Class-Rep has achieved high scores (e.g., on both NYT-Topics and NYT-Locations) showing success of our class-oriented representations. The improvement of X-ClassAlign over X-Class-Rep demonstrates the usefulness of our clustering module. It is also clear that the classifier training is beneficial by comparing X-Class and X-Class-Align.
Ablation on Consistency Check. The consistency check in class representation estimation allows an adaptive number of keywords for each class. Without it leads to a diverged class understanding and degrading performance, as shown in Table 2.
Ablation on Clustering Methods. Table 2 also shows that K-Means performs poorly on most datasets. This matches our previous analysis as K-Means assumes a hard spherical boundary, while
Figure 6: Effects of Attention Mechanisms. We focus on X-Class-Align to show their direct effects.
GMM models the boundary softly like an ellipse.
4.5 Effect of Attention
In Figure 5, we visualize our class-oriented document representations and the unweighted variants using t-SNE (Rauber et al., 2016). We can see that while the simple-average representations are well-separated like class-oriented representations in NYT-Topics, they are much mixed up in NYT-Locations and Yelp. We conjecture that this is because BERT representations has topic information as its most significant feature.
We have also tried using different attention mechanisms in X-Class. From the results in Figure 6, one can see that using a single mechanism, though not under-performing much, is less stable than our proposed mixture. The unweighted case works well on all four datasets that focus on news topics but not good enough on locations and sentiments.
4.6 Hyper-parameter Sensitivity in X-Class
Figure 7 visualizes the performance trend w.r.t. to the three hyper-parameters in X-Class, i.e., the limit of class words  $T$  in class representation estimation, the PCA dimension  $P$  in document-class alignment, and the confidence threshold  $\delta$  in text classifier training.
Intuitively, a class doesn't have too many highly relevant keywords. One can confirm this in Figure 7(a) as the performance of X-Class is relatively stable unless  $T$  goes too large to 1000.
Choosing a proper PCA dimension could prune out redundant information in the embeddings and improve the running time. However, if  $P$  is too small or too large, it may hurt due to information
Figure 7: Hyper-parameter Sensitivity in X-Class. For  $T$  and  $P$ , we report the performance of X-Class-Align to explore their direct effects.
Table 4: Macro- $\mathbf{F}_1$  score changes of methods when removing all but one occurrence of a class name.
Model	20News	NYT-Small
Original	Removed	Original	Removed
X-Class	77.76	74.48	94.02	93.29
LOTClass	72.53	8.82	56.05	29.53
loss or redundancy. One can observe this expected trend in Figure 7(b) on all datasets.
Typically, we want to select a reasonable number of confident training samples for the text classifier training. Too few training samples (i.e., too large  $\delta$ ) would lead to insufficient training data. Too many training samples (i.e., too small  $\delta$ ) would lead to too noisy training data. Figure 7(c) shows that  $\delta \in [0.3, 0.9]$  is a good choice on all datasets.
4.7 Requirements on Class Names
Compared with previous works (Meng et al., 2018; Mekala and Shang, 2020; Meng et al., 2020b), our X-Class has a significantly more mild requirement on human-provided class names in terms of quantity and quality. We have conducted an experiment in Table 4 for X-Class on 20News and NYT-Small by deleting all but one occurrence of a class name from the input corpus. In other words, the user-provided class name only appears once in the corpus. Interestingly, the performance of X-Class only drops less than  $1\%$ , still outperforming all compared methods. In contrast, the most recent work, LOTClass (Meng et al., 2020b), requires a wide variety of contexts of class names from the input corpus to ensure the quality of generated class vocabulary in its very first step.
5 X-Class for Hierarchical Classification
There are two straightforward ways to extend X-Class for hierarchical classification (1) X-Class-End: We can give all fine-grained class names as input to X-Class and conduct classification in an end-to-end manner; and (2) X-Class-Hier: We can first
Table 5: Micro-/Macro- $\mathbf{F}_1$  scores for Fine-grained Classification on NYT-Small. All compared methods use 3 keywords per class. LOTClass failed to discover documents with category indicative terms, thus not reported.  $\S$  refers to numbers coming from other papers.
Model	Coarse (5 classes)	Fine (26 classes)
WeSTClass	91/84$	50/36$
WeSHClass	87.4/63.2$
ConWea	95.23/90.79	91/79$
X-Class-End	96.67/92.98	86.07/75.30
X-Class-Hier	92.66/80.92
give only coarse-grained class names to X-Class and obtain coarse-grained predictions. Then, for each coarse-grained class and its predicted documents, we further create a new X-Class classifier based on the fine-grained class names.
We experiment with hierarchical classification on the NYT-Small dataset, which has annotations for 26 fine-grained classes. We also introduce WeSHClass (Meng et al., 2019), the hierarchical version of WeSTClass, for comparison. LOTClass is not investigated here due to its poor coarse-grained performance on this dataset. The results in Table 5 show that X-Class-Hier performs the best, and it is a better solution than X-Class-End. We conjecture that this is because the fine-grained classes' similarities are drastically different (a pair of fine-grained classes can much similar than another pair). Overall, we show that we can apply our method to a hierarchy of classes.
We discuss related work from two angles.
Weakly supervised text classification. Weakly supervised text classification has attracted much attention from researchers (Tao et al., 2018; Meng et al., 2020a; Mekala and Shang, 2020; Meng et al., 2020b). The general pipeline is to generate a set of document-class pairs to train a supervised model above them. Most previous work utilizes keywords
to find such pseudo data for training, which requires an expert that understands the corpus well. In this paper, we show that it is possible to reach a similar, and often better, performance on various datasets without such guidance from experts.
A recent work (Meng et al., 2020b) also studied the same topic — extremely weak supervision on text classification. It follows a similar idea of (Meng et al., 2020a) and further utilizes BERT to query replacements for class names to find keywords for classes, identifying potential classes for documents via string matching. Compared with LoTClass, our X-Class has a less strict requirement of class names being existent in the corpus, and can work well even when there is only one occurrence (refer to Section 4.7).
BERT for topic clustering. Aharoni and Goldberg (2020) showed that document representations obtained by an average of token representations from BERT preserve domain information well. We borrow this idea to improve our document representations through clustering. Our work differs from theirs in that our document representations are guided by the given class names.
We propose our method X-Class for extremely weak supervision on text classification, which is to classify text with only class names as supervision. X-Class leverages BERT representations to generate class-oriented document presentations, which we then cluster to form document-class pairs, and in the end, fed to a supervised model to train on. We further set up benchmark datasets for this task that covers different data (news and reviews) and various class types (topics, locations, and sentiments). Through extensive experiments, we show the strong performance and stability of our method.
There are two directions that are possible to explore. First, focusing on the extremely weak supervision setting, we can extend to many other natural language tasks to eliminate human effort, such as Named Entity Recognition and Entity Linking. Second, based on the results on extremely weak supervision, we can expect an unsupervised version of text classification, where machines suggest class names and classify documents automatically.
We thank all reviewers for their constructive comments; Yu Meng for valuable discussions and com
ments. Our work is supported in part by NSF Convergence Accelerator under award OIA-2040727. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation hereon.
We do not anticipate any significant ethical concerns; Text Classification is a fundamental problem in Natural Language Processing. The intended use of this work would be to classify documents, such as news articles, efficiently. A minor consideration is the potential for certain types of hidden biases to be introduced into our results, such as a biased selection of class names or language model pre-trained on biased data. We did not observe this kind of issue in our experiments, and indeed these considerations seem low-risk for the specific datasets studied here.
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7747-7763. Association for Computational Linguistics.
Dana Angluin and Philip D. Laird. 1987. Learning from noisy examples. Mach. Learn., 2(4):343-370.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Richard O. Duda and Peter E. Hart. 1973. Pattern classification and scene analysis. A Wiley-Interscience publication. Wiley.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 55-65. Association for Computational Linguistics.
Jacob Goldberger and Ehud Ben-Reuven. 2017. Training deep neural-networks using a noise adaptation layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Hwiyeol Jo and Ceyda Cinarel. 2019. Delta-training: Simple semi-supervised text classification using pretrained word embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3456-3461. Association for Computational Linguistics.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, July 9-12, 1995, pages 331-339. Morgan Kaufmann.
Stuart P. Lloyd. 1982. Least squares quantization in PCM. IEEE Trans. Inf. Theory, 28(2):129-136.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421. The Association for Computational Linguistics.
Dheeraj Mekala and Jingbo Shang. 2020. Contextualized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 323-333. Association for Computational Linguistics.
Dheeraj Mekala, Xinyang Zhang, and Jingbo Shang. 2020. Meta: Metadata-empowered weak supervision for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020a. Discriminative topic mining via categoryname guided text embedding. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 2121-2132. ACM / IW3C2.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, pages 983-992. ACM.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2019. Weakly-supervised hierarchical text classification. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6826-6833. AAAI Press.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020b. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9006-9017. Association for Computational Linguistics.
David M. W. Powers. 1998. Applications and explanations of zipf's law. In Proceedings of the Joint Conference on New Methods in Language Processing and Computational Natural Language Learning, NeMLaP/CoNLL 1998, Macquarie University, Sydney, NSW, Australia, January 11-17, 1998, pages 151-160. ACL.
Paulo E. Rauber, Alexandre X. Falcao, and Alexandru C. Telea. 2016. Visualizing time-dependent data using dynamic t-sne. In 18th Eurographics Conference on Visualization, EuroVis 2016 - Short Papers, Groningen, The Netherlands, June 6-10, 2016, pages 73-77. Eurographics Association.
Fangbo Tao, Chao Zhang, Xiusi Chen, Meng Jiang, Tim Hanratty, Lance M. Kaplan, and Jiawei Han. 2018. Doc2cube: Allocating documents to text cube without labeled data. In IEEE International Conference on Data Mining, ICDM 2018, Singapore, November 17-20, 2018, pages 1260-1265. IEEE Computer Society.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3912-3921. Association for Computational Linguistics.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural
Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.