SlowGuess/NAACL_2021_OCR huggingface dataset:
- Hier-0Shot-TC (Yin et al., 2019): This zero-shot method uses a pre-trained textual entailment model to predict to what extent a document (as the premise text) can entail a template filled with the class name (as the hypothesis text). Similar to Hier-doc2vec, we select the class with the highest entailment score at each level in a top-down recursive fashion. For fair comparison, we change its internal BERT-base-uncased model to RoBERTa-large-mnli model as is used in our method.
- TaxoClass<sup>10</sup>: Our proposed weakly-supervised framework that identifies document core classes, leverages core classes to train a taxonomy-enhanced text classifier, and generalizes the classifier using multi-label self-training. We also evaluate two ablations: TaxoClass-NoST which removes the multi-label self-training step, and TaxoClass-NoGNN which replaces the GNN-based class encoder with a simple embedding layer initialized with pre-trained word embeddings (c.f. Sect. 3.3.1).
We follow previous studies (Partalas et al., 2015; Prabhu et al., 2018) and evaluate the multi-label classification results from different aspects using various metrics. The first metric is Example-F1<sup>11</sup> which calculates the average F1 scores for all documents as follows:
$$
\text {E x a m p l e - F 1} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {2 | \mathbb {C} _ {i} ^ {t r u e} \cap \mathbb {C} _ {i} ^ {p r e d} |}{| \mathbb {C} _ {i} ^ {t r u e} | + | \mathbb {C} _ {i} ^ {p r e d} |},
$$
where  $\mathbb{C}_i^{true}(\mathbb{C}_i^{pred})$  is the true (model predicted) class set of document  $D_{i}$ .
Moreover, as many applications formalize the HMTC as a class ranking problem (Jain et al., 2016; Guo et al., 2019), we convert predicted class set  $\mathbb{C}_i^{pred}$  into a rank list  $\mathbb{R}_i^{pred}$  based on each class's model predicted probability and calculate Precision at  $k(P@k)$  as follows:
$$
P @ k = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {| \mathbb {C} _ {i} ^ {t r u e} \cap \mathbb {R} _ {i , 1 : k} ^ {p r e d} |}{m i n (k , | \mathbb {C} _ {i} ^ {t r u e} |)},
$$
$^{5}$ https://radimrehurek.com/gensim/models/doc2vec.html
<sup>6</sup>We also test the Flat-doc2vec variant which directly ranks all classes in the taxonomy and returns top ranked classes. Its performance is significantly worse than Hier-doc2vec and thus we only report Hier-doc2vec results.
<sup>7</sup>https://github.com/yumeng5/WeSHClass
<sup>8</sup>https://github.com/HKUST-KnowComp/
PathPredictionForTextClassification
https://github.com/yinwenpeng/BenchmarkingZeroShot
$^{10}$ https://github.com/mickeystroller/TaxoClass
This metric is also called "micro-Dice coefficient".
Method	Amazon-531	DBPedia-298
Example-F1	P@1	P@3	MRR	Example-F1	P@1	P@3	MRR
Hier-doc2vec (Le and Mikolov, 2014)	0.3157	0.5805	0.3115	N/A	0.1443	0.2635	0.1443	N/A
WeSHClass (Meng et al., 2019)	0.2458	0.5773	0.2517	N/A	0.3047	0.5359	0.3048	N/A
TaxoClass-NoST	0.5431	0.7918	0.5414	0.5911	0.7712	0.8621	0.7712	0.8221
TaxoClass-NoGNN	0.5271	0.7642	0.5213	0.5621	0.7241	0.8154	0.7241	0.7692
TaxoClass	0.5934	0.8120	0.5894	0.6332	0.8156	0.8942	0.8156	0.8762
SS-PCEM (Xiao et al., 2019)	0.2921	0.5369	0.2948	0.3004	0.3845	0.7424	0.3845	0.4032
Hier-0Shot-TC (Yin et al., 2019)	0.4742	0.7144	0.4610	N/A	0.6765	0.7871	0.6765	N/A
where  $\mathbb{R}_{i,1:k}^{pred}$  is each method predicted top  $k$  most likely classes for  $D_{i}$ . Finally, for methods able to return the probability of a document being tagged with each class in the taxonomy, we calculate their Mean Reciprocal Rank (MRR) as follows:
$$
\mathrm {M R R} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{| \mathbb {C} _ {i} ^ {t r u e} |} \sum_ {c _ {j} \in \mathbb {C} _ {i} ^ {t r u e}} \frac {1}{R _ {i j}},
$$
where  $R_{ij}$  is the "rank" of document  $D_j$ 's true class  $c_j$  in model predicted rank list (over all classes).
4.4 Experiment Settings
For all baseline methods except Hier-doc2vec, we use the public implementations from their authors and leave the hyper-parameters unchanged. For both Hier-0Shot-TC and our method, we adopt the same public Roberta-Large-MNLI model as the textual entailment model and use the same hypothesis template: "this product is about _. ." for Amazon-531 dataset and "this example is _. ." for DBPedia-298 dataset. We use AdamW optimizer to train our model with batch size 64, learning rate 5e-5 for all parameters in BERT document encoder and learning rate 4e-3 for all remaining parameters. During the multi-label self-training stage (c.f. Sect. 3.4), we use learning rate 1e-6 for all parameters in the BERT document encoder and 5e-4 for all remaining parameters.
We run all experiments on a single cluster with 80 CPU cores and a Quadro RTX 8000 GPU. All deep learning models are moved to the GPU for faster inference speed. With batch size 64, the TaxoClass framework consumes about 10GB GPU memory. In principle, all methods should be runnable on CPU.
Table 2: Evaluation of all compared methods on two datasets. For some methods predicting a class path in a top-down fashion rather than returning all classes' probabilities, we cannot compute their MRR scores and indicate this using "N/A".
Core Class Mining Method	Example-F1	P@1	P@3	MRR
Explicit Mention	0.1611	0.2168	0.1564	0.2045
0Shot	0.4793	0.7361	0.4782	N/A
Ours	0.5431	0.7918	0.5414	0.5911
Ours-NoCS	0.3812	0.6254	0.3831	0.4366
Ours-NoConf	0.2603	0.4431	0.2521	0.3014
Table 3: Evaluation of core class mining algorithms on Amazon-531 dataset. We train the classifier using different training sets derived from different core class mining algorithm outputs. Please refer to Section 4.6 for detailed descriptions of each method.
4.5 Overall Performance Comparison
Table 2 presents the overall results of all compared methods. First, we find most weakly-supervised and zero-shot method can outperform the semi-supervised method SS-PCEM even the later has access to  $30\%$  of labeled documents. Second, we can see that TaxoClass has the overall best performance across all the metrics and defeats the second best method by a large margin. Comparing TaxoClass with TaxoClass-NoGNN, we show the importance of incorporating taxonomy structure into the class encoder. Moreover, the improvement of TaxoClass over TaxoClass-NoST demonstrates the effectiveness of our multi-label self-training.
4.6 Effectiveness of Core Class Mining
We evaluate the effectiveness of our core class mining method as follows. First, we define a set of rival methods and use them to generate various sets of "core classes". Then, we derive pseudo-training data for each generated core class set and use it to learn a text classifier with the same architecture as the one in TaxoClass. Finally, we report each model's performance on the test set. Note here we skip the self-training step to ensure the "core class based pseudo-training data" is the only variable.
Method	Example-F1	P@1	P@3	MRR
fastText	0.4472	0.7515	0.4521	0.4587
TextCNN	0.4787	0.7694	0.4771	0.4827
TaxoClass-NoGNN	0.5271	0.7642	0.5213	0.5621
TaxoClass	0.5934	0.8120	0.5894	0.6332
Table 4: Performance of different classifiers on Amazon-531 dataset. All methods use the same training set derived from our identified document core classes.
Table 3 lists all the results. First, we find that the "Explicit Mention" method, which treats all classes with names explicitly appear in the corpus as the core classes, does not perform well for our HMTC problem. One reason could be many class names are human-curated summarizing phrases that do not appear in the corpus naturally. Second, the "0Shot" method views the output classes of baseline method Hier-0Shot-TC as the core classes and trains a new classifier. Interestingly, this new classifier performs better than the original Hier-0Shot-TC classifier, which shows that transferring knowledge from a general zero-shot classifier to a domain-specific classifier is a possible and promising direction. Finally, we compare variants of our own methods. The "Ours-NoCS" method removes the candidate core class selection step (c.f. Sect. 3.2.1) and treats all classes with high confidence scores as core classes. The "Ours-NoConf" method skips the confident core class identification step (c.f. Sect. 3.2.2) and views all candidate core classes as the final output core classes. We can see a significant performance drop on both ablations, which shows the importance of our two core class mining steps.
4.7 Analysis of Classifier Architecture
We study whether we can use the identified document core classes to train other text classifiers with different architectures such as fastText (Joulin et al., 2016) and TextCNN (Kim, 2014). As shown in Table 4, both methods achieve reasonable performance. We can also see that TaxoClass with and without GNN-enhanced class encoder can outperform both methods. This shows the effectiveness of our dual-encoder style classifier architecture.
4.8 Supervision Signals in Class Names
We vary the percentage of labeled documents on Amazon-531 dataset for training a supervised fastText classifier and present its corresponding performance in Fig. 5. We can see the performance of our TaxoClass framework is equivalent to that of supervised fastText learned on roughly  $70\%$  of
Figure 5: Comparison between TaxoClass and supervised fastText method on Amazon-531 dataset. We train the fastText model using on different percentages of labeled training documents.
labeled documents in the training set (i.e., about 20,000 labeled documents).
Weakly-supervised Text Classification. There exist some previous studies that leverage a few labeled documents or class-indicative keywords as weak supervision signals for text classification. A pioneering method is dataless classification (Chang et al., 2008; Song and Roth, 2014) which embeds documents and classes into the same semantic space of Wikipedia concepts and performs classification using the embedding similarity. Li et al. (2018, 2019) extend this idea by mining concepts directly from the corpus rather than using the external Wikipedia. Along another line, Chen et al. (2015) and Li et al. (2016) propose to apply a seed-guided topic model to infer class-specific topics from class-indicative keywords and to predict document classes from posterior class-topic assignments. Compared with these methods, our TaxoClass framework neither restricts document and class embeddings to live in the same semantic space nor imposes strong statistical assumptions.
Recently, neural models are applied to weakly-supervised text classification. Meng et al. (2018, 2019) propose a pretrain-and-refine paradigm which first generates pseudo documents to pretrain a neural classifier and then refine this classifier via self-training. Mekala and Shang (2020); Meng et al. (2020); Wang et al. (2020) improve the above methods by introducing contextualized weak supervision and using a pre-trained language model to obtain better text representations. While achieving inspiring performance, these methods all assume each document has only one class and all class names (or class-indicative keywords) must appear in the corpus for pseudo training data generation. In this paper, we relax these assumptions and develop a new method for weakly-supervised hierarchical
multi-label text classification task.
Zero-shot Text Classification. Zero-shot text classification learns a text classifier based on training documents belonging to seen classes and applies the learned classifier to predict testing documents belonging to unseen classes (Wang et al., 2019). Nam et al. (2016) jointly embed documents and classes into a shared semantic space where knowledge from seen classes can be transferred to unseen classes. Such an idea is further developed in (Rios and Kavuluru, 2018; Srivastava et al., 2018; Yin et al., 2019; Chu et al., 2020) where external resources (e.g., knowledge graphs, natural language explanations of unseen classes, and open domain data) are introduced to help learn a better shared semantic space. Comparing with these methods, our TaxoClass framework does not require labeled data for a set of seen classes.
Hierarchical Text Classification. Hierarchical text classification leverages a class hierarchy to improve the standard text classification performance. Typical methods can be divided into two categories: (1) local approaches which learn a text classifier per class (Banerjee et al., 2019), per parent class (Liu et al., 2005), or per level (Wehrmann et al., 2018), and (2) global approaches which incorporate taxonomy structure information into one single classifier through recursive regularization (Gopal and Yang, 2013) or graph neural network (GNN) based encoder (Peng et al., 2018; Huang et al., 2019; Zhou et al., 2020). Our TaxoClass framework adopts the second global approach and uses a GNN-based encoder to obtain each class's representation.
This paper studies the hierarchical multi-label text classification problem when only class surface names, instead of massive labeled documents, are given. We propose a novel TaxoClass framework which leverages the class taxonomy structure to derive document core classes and learns taxonomy-enhanced text classifier for prediction. Extensive experiments demonstrate the effectiveness of TaxoClass on two real-world datasets from different domains. In the future, we plan to explore how TaxoClass framework can be integrated with semi-supervised methods and data augmentation methods, when some class surface names are too ambiguous to indicate class semantics. Moreover, we consider extending our multi-label self-training
method to other related NLP tasks such as fine-grained entity typing.
As text classification is a standard task in NLP, we do not see any significant ethical concerns. The expected usage of our work is to classify documents such as news articles, scientific literature, and etc.
Research was sponsored in part by US DARPA SocialSim Program No. W911NF-17-C0099, NSF IIS 16-18481, IIS 17-04532, and IIS 17-41317, and DTRA HDTRA11810026. Any opinions, findings or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of DARPA or the U.S. Government. We thank anonymous reviewers for valuable and insightful feedback.
S. Banerjee, Cem Akkaya, Francisco Perez-Sorrosal, and K. Tsioutsiouliklis. 2019. Hierarchical transfer learning for multi-label text classification. In ACL.
David Berthelot, Nicholas Carlini, I. Goodfellow, Nicolas Papernot, A. Oliver, and Colin Raffel. 2019. Mix-match: A holistic approach to semi-supervised learning. ArXiv, abs/1905.02249.
Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In AAAI.
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S. Dhillon. 2019. X-bert: extreme multi-label text classification with bert. In arXiv.
Xingyuan Chen, Yunqing Xia, Peng Jin, and John A. Carroll. 2015. Dataless text classification with descriptive lda. In AAAI.
Zewei Chu, K. Stratos, and Kevin Gimpel. 2020. Natcat: Weakly supervised text classification with naturally annotated datasets. ArXiv, abs/2009.14335.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
S. Gopal and Yiming Yang. 2013. Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In KDD.
Sylvain Goumy and Mohamed-Amine Mejri. 2018. Ecommerce product title classification. In SIGIR.
Chuan Fei Guo, Alireza Mousavi, Xiang Wu, Daniel N. Holtmann-Rice, Satyen Kale, Sashank J. Reddi, and Sanjiv Kumar. 2019. Breaking the glass ceiling for embedding-based classifiers for large output spaces. In NeurIPS.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. In ACL.
Wei Huang, Enhong Chen, Qi Liu, Yuying Chen, Zai Huang, Yang Liu, Zhou Zhao, Dan Zhang, and Shijin Wang. 2019. Hierarchical multi-label text classification: An attention-based recurrent network approach. In CIKM.
Himanshu Jain, Yashoteja Prabhu, and Manik Varma. 2016. Extreme multi-label loss functions for recommendation, tagging, ranking, and other missing label applications. In KDD.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Thomas Kipf and M. Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR.
Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML.
Jens Lehmann, Robert Isele, Max Jakob, A. Jentzsch, D. Kontokostas, Pablo N. Mendes, S. Hellmann, M. Morsey, Patrick van Kleef, S. Auer, and C. Bizer. 2015. Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:167-195.
Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. 2016. Effective document labeling with very few seed words: A topic model approach. In CIKM.
Keqian Li, Shiyang Li, Semih Yavuz, Hanwen Zha, Yu Su, and Xifeng Yan. 2019. Hiercon: Hierarchical organization of technical documents based on concepts. In ICDM.
Keqian Li, Hanwen Zha, Yu Su, and Xifeng Yan. 2018. Unsupervised neural categorization for scientific publications. In SDM.
T. Liu, Yiming Yang, H. Wan, H. Zeng, Z. Chen, and W. Ma. 2005. Support vector machines classification with a very large-scale taxonomy. SIGKDD, 7:36-43.
Julian J. McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In RecSys.
Dheeraj Mekala and Jingbo Shang. 2020. Contextualized weak supervision for text classification. In ACL.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In CIKM.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2019. Weakly-supervised hierarchical text classification. In AAAI.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Jinseok Nam, Eneldo Loza Mencia, and Johannes Furnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In AAAI.
Ioannis Partalas, A. Kosmopoulos, Nicolas Baskiotis, T. Artières, G. Paliouras, Éric Gaussier, Ion Androutsopoulos, M. Amini, and P. Gallinari. 2015. Lshtc: A benchmark for large-scale text classification. ArXiv, abs/1503.08581.
Hao Peng, Jianxin Li, Y. He, Yaopeng Liu, Mengjiao Bao, L. Wang, Y. Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. Proceedings of the 2018 World Wide Web Conference.
Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In WWW.
Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In EMNLP.
J. Shen, Zhihong Shen, Chenyan Xiong, Chunxin Wang, Kuansan Wang, and Jiawei Han. 2020. Taxoexpan: Self-supervised taxonomy expansion with position-enhanced graph neural network. Proceedings of The Web Conference 2020.
Yangqiu Song and Dan Roth. 2014. On dataless hierarchical text classification. In AAAI.
Shashank Srivastava, Igor Labutov, and Tom M. Mitchell. 2018. Zero-shot learning of classifiers from natural language quantification. In ACL.
Wei Wang, Vincent Wenchen Zheng, Han Yu, and Chunyan Miao. 2019. A survey of zero-shot learning: Settings, methods, and applications. In ACM TIST.
Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2020. X-class: Text classification with extremely weak supervision. *ArXiv*, abs/2010.12794.
Jonatas Wehrmann, R. Cerri, and Rodrigo C. Barros. 2018. Hierarchical multi-label classification networks. In ICML.
Huiru Xiao, Xin Liu, and Y. Song. 2019. Efficient path prediction for semi-supervised and weakly supervised hierarchical text classification. The World Wide Web Conference.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In ICML.
Peng Xu and Denilson Barbosa. 2018. Neural fine-grained entity type classification with hierarchy-aware loss. In The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018). ACL.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In EMNLP/IJCNLP.
Ronghui You, Suyang Dai, Zihan Zhang, Hiroshi
Mamitsuka, and Shanfeng Zhu. 2019. Attentionxml: Extreme multi-label text classification with multi-label attention based recurrent neural networks. In NeurIPS.
Yue Yu, Simiao Zuo, Haoming Jiang, W. Ren, Tuo Zhao, and C. Zhang. 2020. Fine-tuning pretrained language model with weak supervision: A contrastive-regularized self-training approach. ArXiv, abs/2010.07835.
Ziqian Zeng, Wenxuan Zhou, Xin Liu, and Yangqiu Song. 2019. A variational approach to weakly supervised document-level multi-aspect sentiment classification. In NAACL-HLT.
Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and G. Liu. 2020. Hierarchy-aware global model for hierarchical text classification. In ACL.