SlowGuess/ACL_2020_OCR huggingface dataset:
In this section, we introduce our experiments with datasets, evaluation metrics, implementation details, comparison, ablation study, and analysis of experimental results.
We experiment our proposed architecture on RCV1-V2, Web-of-Science (WOS) and NYTimes (NYT) datasets for comparison and ablation study.
Datasets RCV1-V2 (Lewis et al., 2004) and NYT (Sandhaus, 2008) are both news categorization corpora while WOS (Kowsari et al., 2017) includes abstracts of published papers from Web of Science. Those typical text classification datasets
Dataset	|L|	Depth	Avg(|Li|)	Train	Val	Test
RCV1	103	4	3.24	20,833	2,316	781,265
WOS	141	2	2.0	30,070	7,518	9,397
NYT	166	8	7.6	23,345	5,834	7,292
Table 1: Data Statistics:  $|L|$  is the number of classes.  $\mathrm{Avg}(|L_i|)$  is the average number of classes per sample. Depth indicates the maximum level of hierarchy.
are all annotated with the ground truth of hierarchical taxonomic labels. We use the benchmark split of RCV1-V2 and select a small partial training subset for validation. WOS dataset is randomly splitted into training, validation and test subsets. In NYT, we randomly select and split subsets from original raw data. We also remove samples with no label or only a single one-level label. Note that WOS is for single-path HTC while NYT and RCV1-V2 include multi-path taxonomic tags. The statistics of datasets is shown in Table 1.
Evaluation Metrics We measure the experimental results with standard evaluation metrics (Gopal and Yang, 2013), including Micro-F1 and Macro-F1. Micro-F1 takes the overall precision and recall of all the instances into account while Macro-F1 equals to the average F1-score of labels. So Micro-F1 gives more weight to frequent labels, while Macro-F1 equally weights all labels.
Implementation Details We use a one-layer bi-GRU with 64 hidden units and 3 parallel CNN layers with filter region size of  $\{2,3,4\}$ . The vocabulary is created by the most frequent words with the maximum size of 60,000. We use 300-dimensional pretrained word embedding from GloVe2 (Pennington et al., 2014) and randomly initialize the out-of-vocabulary words above the minimum count of 2. The key information pertaining to text classification could be extracted from the beginning statements. Thus, we set the maximum length of token inputs as 256. The fixed threshold for tagging is chosen as 0.5. Dropout is employed in the embedding layer and MLP layer with the rate of 0.5 while in the bi-GRU layer and node transformation with the rate of 0.1 and 0.05 respectively. Additionally, for HiAGM-LA, the label embedding is initialized by Kaiming uniform (He et al., 2015) while the other model parameters are initialized by Xavier uniform (Glorot and Bengio, 2010). We use the Adam optimizer in a mini-batch size of 64 with learning rate
$^{2}$ https://nlp.stanford.edu/projects/glove
Model	Micro	Macro
Local Models
HR-DGCNN-3 (Peng et al., 2018)	76.18	43.34
HMCN (Mao et al., 2019)	80.80	54.60
HFT(M) (Shimura et al., 2018)	80.29	51.40
Htrans (Banerjee et al., 2019)	80.51	58.49
Global Models
SGM4(Yang et al., 2018)	77.30	47.49
HE-AGCRCNN (Peng et al., 2019)	77.80	51.30
HiLAP-RL (Mao et al., 2019)	83.30	60.10
Baselines
TextRCNN	81.57	59.25
TextRCNN+LabelAttention	81.88	59.85
HiAGM-LA
TreeLSTM	82.54†‡	61.90†‡
GCN	82.21†‡	61.65†‡
GCN w/o Rec	82.26†‡	61.85†‡
HiAGM-TP
TreeLSTM	83.20†	62.32†
GCN	83.96†	63.35†
GCN w/o Rec	83.95†	63.23†
$\alpha = 1 \times 10^{-4}$ , momentum parameters  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and  $\epsilon = 1 \times 10^{-6}$ . The penalty coefficient of recursive regularization is set as  $1 \times 10^{-6}$ . Our model evaluates the test subset with the best model on the validation subset.
In Table 2, we compare the performance of Hi-AGM to traditional MLC models and the state-of-the-art HTC studies on RCV1-V2. With the recursive regularization for the last MLP layer, those conventional text classification models also obtain competitive performance. As for our proposed architecture, both HiAGM-LA and HiAGM-TP outperform most state-of-the-art results of global and local studies, especially in Macro-F1. It shows the strong advancement of our hierarchy encoders on HTC. HiAGM-LA achieves the performance of  $61.90\%$  Macro-F1 score and  $82.54\%$  Micro-F1 score while HiAGM-TP obtains the best performance of  $63.35\%$  Macro-F1 score and  $83.96\%$  Micro-F1 score.
To clarify the improvement of our proposed
Table 2: Comparison to previous models on RCV1-V2. Note that the prior probability matrix in HiAGM-TP is fine-tuned during training while the one in HiAGM-LA is fixed.  $w / o$  Rec denotes training without recursive regularization. "†" and "‡" indicate statistically significant difference (p<0.01) from TextRCNN and TextR-CNN+LabelAttention respectively.
Model	HiAGM-LA	HiAGM-TP
Micro	Macro	Time	Micro	Macro	Time
TreeLSTM	82.54	61.90	1.0 ×	83.24	62.60	3.2 ×
GCN	82.21	61.65	0.9 ×	83.92	63.01	1.1 ×
Table 3: Comparison of the HiAGM variants on RCV1-V2 with fixed prior probability. Note that Time denotes the time cost of one epoch during inference compared to TreeLSTM-based HiAGM-LA. Statistically significant difference  $(p < 0.01)$  compared to the best one.
HiAGM, we also experiment without recursive regularization. Compared with the state-of-the-art recent work (HiLAP) (Mao et al., 2019), our HiAGM-LA and HiAGM-TP without recursive regularization also achieve competitive improvement by  $1.75\%$  and  $3.13\%$  in terms of Macro-F1. It demonstrates that the recursive regularization is complementary but not necessary with our proposed architecture.
According to Table 4, HiAGM achieves consistent improvement on the performance of HTC among RCV1-V2, WOS and NYT datasets. It indicates the strong improvement of the label-wise text feature on HTC task. The results present that our proposed global model HiAGM has the advanced capability of enhancing text features for HTC.
All in all, HiAGM strongly improves the performance on the benchmark dataset RCV1-V2 and the other two classical text classification datasets. Especially, it obtains better results on Macro-F1 score. It indicates that HiAGM has a strong ability to tackle data-sparse classes deep in the hierarchy.
Hybrid Information Aggregation According to Table 2, both variants outperform the baseline models and previous studies. It denotes that the enhanced text feature is beneficial for HTC. We clarify the ablation study of two variants and structure encoders in Table 3. Both HiAGM-LA and HiAGM-TP are trained with fixed prior probability. With the help of the recursive computation process, bidirectional Tree-LSTM achieves better performance on learning hierarchy-aware label embedding. However, it additionally leads to lower computational efficiency when compared to Hierarchy-GCN. Regarding HiAGM-TP, hierarchy-GCN shows its better performance and efficiency than bidirectional Tree-LSTM.
These two variants have various advantages, respectively. To be specific, HiAGM-TP has better performance than HiAGM-LA in both Bi
4The result is reproduced with benchmark split upon the released project of SGM.
Model	RCV1-V2	RCV1-V2-R	WOS	NYT
Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1
Global Text Classification Baseline
TextRNN	81.10	51.09	87.78	70.42	77.94	69.65	70.29	53.06
TextCNN	79.37	55.45	84.97	68.06	82.00	76.18	70.11	56.84
TextRCNN	81.57	59.25	88.32	72.23	83.55	76.99	70.83	56.18
HiAGM-LA
GCN	82.21	61.65	88.49	73.14	84.61	79.37	72.35	58.67
TreeLSTM	82.54	61.90	88.47	72.81	84.82	79.51	72.50	58.86
HiAGM-TP
GCN	83.96	63.35	88.64	74.00	85.82	80.28	74.97	60.83
TreeLSTM	83.20	62.32	88.86	74.16	85.18	79.95	74.43	60.76
Table 4: Experimental results of our proposed HiAGM-LA and HiAGM-TP on various datasets. Note that RCV1-V2-R refers to the version that transpose original subset of the train and test set. All models are trained with the constraint of recursive regularization. HiAGM-LA is trained with fixed prior probability while HiAGM-TP is trained with trainable one.
TreeLSTM and Hierarchy-GCN encoders. The multi-label attention variant, HiAGM-LA, would somehow induce noises from the randomly initialized label embedding. Otherwise, HiAGM-TP aggregates the fusion of local structural information and text feature maps, without the negative impact of label embedding.
As for efficiency, HiAGM-LA is more computationally efficient than HiAGM-TP, especially in the inference process. The label representation from hierarchy encoders could be utilized as pretrained label embedding for multi-label attention during inference. Thus, HiAGM-LA omits the hierarchy-aware structure encoder module after training.
We recommend HiAGM-TP for high performance while we also suggest HiAGM-LA for empirically good performance and faster inference.
GCN Layers The impact of GCN layers is also an important issue for HiAGM. As illustrated in Figure 4, the one-layer structure encoder consistently performs best in both HiAGM-LA and HiAGM-TP. It indicates that the correlation between non-adjacent nodes is not essential for HTC but somehow noisy for hierarchical information aggregation. This empirical conclusion is consistent with the implementation of recursive regularization (Peng et al., 2018; Gopal and Yang, 2013) and transfer learning (Banerjee et al., 2019; Shimura et al., 2018) between adjacent labels or levels.
Prior Probability According to the aforementioned comparisons, our simplified structure encoders with prior probabilities is undoubtedly beneficial for HTC. We also investigate different choices of prior probabilities with hierarchy-GCN encoder
on the HiAGM-TP variant, clarified as Table 5. Note that the weighted adjacent matrix is initialized by prior probabilities.
The simple weighted adjacent matrix performs better than the complex edge-wise weight matrix for node transformation. The fixed weighted adjacent matrix also achieves better results than the original unweighted adjacent matrix and the trainable randomly initialized one. It demonstrates that the prior probability of the hierarchy is capable of representing hierarchical label dependencies. Furthermore, the best result is obtained by the setting that obeys the calculating direction of prior probability. When comparing the results of the fixed adjacent matrix and trainable one, we can find that the weighted adjacent matrix could be finetuned for higher flexibility and better performance.
In Table 5, the settings that allow all interac
Figure 4: Ablation study on the depth of GCN.
Top-Down	Bottom-Up	Fixed	Trainable
Micro	Macro	Micro	Macro
Edge-Wise Matrix	-	-	82.75	60.81
Randomly Initialized	-	-	83.86	62.12
Randomly Initialized*	-	-	82.80	62.51
1	1	83.77	62.31	83.86	62.96
P	P	83.61	63.65	83.83	63.14
1	P	83.65	62.46	83.95	63.23
P	1	83.92	63.01	83.96	63.35
P*	1*	-	-	83.33	62.86
Table 5: Ablation study of the fine-grained hierarchy information on RCV1-V2 based on GCN-based HiAGM-TP. Edge-Wise Matrix denotes that each directional edge has a distinct trainable weight matrix for transformation while the others use the weighted adjacent matrix.  $P$  is  ${f}_{c}\left( {e}_{i,j}\right)  = \frac{{N}_{j}}{{N}_{i}}$  and  $I$  is  ${f}_{p}\left( {e}_{i,j}\right)  = {1.0}$  . "  $*$  " allows the information propagation between all nodes while the others obey the constraint of hierarchy.
tions perform worse than the others that allow propagation throughout the hierarchy paths. As analyzed on GCN layers, the interaction between non-adjacent nodes would lead to negative impact on the HTC. We also validate this conclusion based on the ablation study of prior probability.
Performance Study We analyze the improvement on performance by dividing labels based on their levels. We compute level-based MicroF1 scores of NYT on baseline, HiAGM-LA, and HiAGM-TP. Figure 5 shows that our models retain a better performance than the baseline on all levels, especially among deep levels.
Figure 5: Evaluation of labels among different levels. Note that we observe similar results for other datasets and omit them for a cleaner view.
In this paper, we propose a novel end-to-end hierarchy-aware global model that extracts the label structural information for aggregating label-wise text features. We present a bidirectional TreeLSTM and a hierarchy-GCN as the hierarchy-aware structure encoder. Furthermore, our framework is extended into a parallel variant based on multi-label attention and a serial variant of text feature propagation. Our approaches empirically achieve significant and consistent improvement on three distinct datasets, especially on the low-frequency labels. Specifically, both variants outperform the state-of-the-art model on the RCV1-V2 benchmark dataset. And our best model obtains a Macro-F1 score of  $63.35\%$  and a Micro-F1 score of  $83.96\%$ .
We thank all the anonymous reviewers for their valuable suggestions. This research work was supported by the National Natural Science Foundation of China (Grant No.61772337, U1736207).
Siddhartha Banerjee, Cem Akkaya, Francisco Perez-Sorrosal, and Kostas Tsiousiouliklis. 2019. Hierarchical transfer learning for multi-label text classification. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 6295-6300. Association for Computational Linguistics.
Lijuan Cai and Thomas Hofmann. 2004. Hierarchical document categorization with support vector machines. In Proceedings of the 2004 ACM CIKM International Conference on Information and Knowledge Management, Washington, DC, USA, November 8-13, 2004, pages 78-87. ACM.
Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017a. Improved neural machine translation with a syntax-aware encoder and decoder. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1936-1945. Association for Computational Linguistics.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1657-1668. Association for Computational Linguistics.
Cunxiao Du, Zhaozheng Chen, Fuli Feng, Lei Zhu, Tian Gan, and Liqiang Nie. 2019. Explicit interaction model towards text classification. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6359-6366. AAAI Press.
Susan T. Dumais and Hao Chen. 2000. Hierarchical classification of web content. In SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece, pages 256-263. ACM.
David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2224-2232.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1263-1272. PMLR.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pages 249-256. JMLR.org.
Siddharth Gopal and Yiming Yang. 2013. Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013, pages 257-265. ACM.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance onImagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1026-1034. IEEE Computer Society.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics,
ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2061-2071. Association for Computational Linguistics.
Wei Huang, Enhong Chen, Qi Liu, Yuying Chen, Zai Huang, Yang Liu, Zhou Zhao, Dan Zhang, and Shijin Wang. 2019. Hierarchical multi-label text classification: An attention-based recurrent network approach. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pages 1051-1060. ACM.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber, and Laura E. Barnes. 2017. Hdtex: Hierarchical deep learning for text classification. In 16th IEEE International Conference on Machine Learning and Applications, ICMLA 2017, Cancun, Mexico, December 18-21, 2017, pages 364-371. IEEE.
David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res., 5:361-397.
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2401-2411. Association for Computational Linguistics.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In *NAACL HLT* 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015, pages 912-921. The Association for Computational Linguistics.
Yuning Mao, Jingjing Tian, Jiawei Han, and Xiang Ren. 2019. Hierarchical text classification with reinforced label assignment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, pages 445-455. Association for Computational Linguistics.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017
Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1506-1515. Association for Computational Linguistics.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1101-1111. Association for Computational Linguistics.
Bo Pang and Lillian Lee. 2007. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1-135.
Hao Peng, Jianxin Li, Qiran Gong, Senzhang Wang, Lifang He, Bo Li, Lihong Wang, and Philip S. Yu. 2019. Hierarchical taxonomy-aware and attentional graph capsule rcnns for large-scale multi-label text classification. CoRR, abs/1906.04898.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1063-1072. ACM.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar. A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3132-3142. Association for Computational Linguistics.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Kazuya Shimura, Jiyi Li, and Fumiyo Fukumoto. 2018. HFT-CNN: learning hierarchical category structure for multi-label short text categorization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 811-816. Association for Computational Linguistics.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations