SlowGuess/ACL_2020_OCR huggingface dataset:
We evaluate Graph2Taxo on Semeval-2016 Task 13: Taxonomy Extraction Evaluation<sup>3</sup>, otherwise known as TExEval-2 task (Bordea et al., 2016). All experiments are implemented in PyTorch. Code is publicly available at https://github.com/IBM/gnn-taxo-construction.
Domain	Source	V	E
Science	WordNet	429	452
Eurovoc	125	124
Combined	453	465
Environment	Eurovoc	261	261
Table 1: Dataset statistics for TExEval-2 task obtained from Bordea et al. (2016). The Vertices(V) and Edges(E) columns represent structural measures of taxonomies for English language only.
4.1 Benchmark Datasets
For experiments, we used the English environment and the science taxonomies within the TExEval-2 benchmark datasets. These datasets do not come with any training data, but a list of terms and the task is to build a meaningful taxonomy using these terms. The science domain terms come from WordNet, Eurovoc and a manually constructed taxonomy (henceforth referred to as combined), whereas the terms for environment domain comes from Eurovoc taxonomy only. Table 1 shows the dataset statistics.
We chose to evaluate our proposed approach on environment and science taxonomies only, because we wanted to compare our approach with the existing state-of-the-art system named TaxoRL (Mao et al., 2018) as well as with TAXI, the winning system in the TExEval-2 task. Note that we use the same datasets with TaxoRL (Mao et al., 2018) for TExEval-2 task.
In addition, we used the dataset from Bansal et al. (2014) as gold taxonomies (i.e. sources of additional knowledge),  $G_{gold} = \{G_{gold,i}, 1 \leq i \leq N_{known}\}$  that are known apriori. This dataset is a set of medium-sized full-domain taxonomies consisting of bottom-out full subtrees sampled from Wordnet, and contains 761 taxonomies in total.
To test our model for taxonomy prediction (and to remove overlap), we removed any taxonomy from  $G_{gold}$  which had term overlap with the set of provided terms for science and environment domains within TExEval-2 task. Because of this, we get 621 non-overlapping taxonomies in total, partitioned by 80-20 ratio to create training and validation datasets respectively.
We ran our experiments in two different settings. In each of them, we train on a different noisy input graph (and the same gold taxonomies as mentioned before), and evaluate on the science and environ-
3Semeval-2016 Task 13: http://alt.qcri.org/ semeval2016/task13
Science
(Combined)	Science
(Eurovoc)	Science
(WordNet)	Science
(Average)	Environment
(Eurovoc)
Model	Pe	Re	Fe	Pe	Re	Fe	Pe	Re	Fe	Pe	Re	Fe	Pe	Re	Fe
Baseline	0.63	0.29	0.39	0.62	0.21	0.31	0.69	0.27	0.38	0.65	0.26	0.36	0.50	0.21	0.30
JUNLP	0.14	0.31	0.19	0.13	0.36	0.19	0.21	0.31	0.25	0.16	0.33	0.21	0.13	0.23	0.17
USAAR	0.38	0.26	0.31	0.63	0.15	0.25	0.82	0.19	0.31	0.61	0.20	0.29	0.81	0.15	0.25
TAXI	0.39	0.35	0.37	0.30	0.33	0.31	0.37	0.38	0.38	0.35	0.35	0.35	0.34	0.27	0.30
TaxoRLA	-	-	-	-	-	-	-	-	-	0.57	0.33	0.42	0.38	0.24	0.29
TaxoRLB	-	-	-	-	-	-	-	-	-	0.38	0.38	0.38	0.32	0.32	0.32
Graph2Taxo1	0.91	0.31	0.46	0.78	0.26	0.39	0.82	0.32	0.46	0.84	0.30	0.44	0.89	0.24	0.37
Graph2Taxo2	0.90	0.33	0.48	0.79	0.33	0.46	0.77	0.32	0.46	0.82	0.33	0.47	0.67	0.28	0.39
Table 2: Results on TExEval-2 task: Taxonomy Extraction Evaluation (a.k.a TExEval-2). First four rows represent participating systems in the TExEval-2 task, whose performances are taken from Bordea et al. (2016). TaxoRL  ${}^{A/B}$  illustrate the performance of a Reinforcement Learning system by Mao et al. (2018) under the Partial and Full setting respectively. Graph  $2{\operatorname{Taxo}}^{1/2}$  represent our proposed algorithm under both the settings as described in Section 4.2. All results reported above are rounded to 2 decimal places.
ment domains, within TExEval-2 task. In the first setting, we used the same input as TaxoRL (Mao et al., 2018) for a fair comparison. This input of TaxoRL consists of term pairs and associated dependency path information between them, which has been extracted from three public web-based corpora. For Graph2Taxo, we only make use of the term pairs to create a noisy input graph.
In the second setting, we used data<sup>4</sup> provided by TAXI (Panchenko et al., 2016), which comprises of a list of candidate is-a pairs extracted based on substrings and lexico-syntactic patterns. We used these noisy candidate pairs to create a noisy graph.
A Graph2Taxo model is then trained on the noisy graph obtained in each of the two settings. In the test phase, all candidate term-pairs for which both terms are present in the test vocabulary are scored (between 0 and 1) by the trained Graph2Taxo model. A threshold of 0.5 is applied, and the candidate pairs scoring beyond this threshold are accumulated together as the predicted taxonomy  $G_{pred}$ . Notice that there are different optimal thresholds for different tasks. We get better performance if we tune the thresholds. However, we chose a harder task and proved our model has better performance than others even we simply use 0.5 as the threshold. In addition, We specify the hyper-parameter ranges for our experiments: learning rate {0.01, 0.005, 0.001}, number of kernels {5, 10, 20} and number of clusters {10, 30, 50, 100}. Finally, Adam optimizer (Kingma and Ba, 2015) is used for all experiments.
Evaluation Metrics. Given a gold taxonomy
$G_{gold}$  (as part of the TExEval-2 benchmark dataset) and a predicted taxonomy  $G_{pred}$  (by our proposed Graph2Taxo approach), we evaluate  $G_{pred}$  using Edge Precision, Edge Recall and F-score measures as defined in Bordea et al. (2016).
4.3 Hyper-parameters
We use the following hyper-parameter configuration for training the model. We set dropout to 0.3, number of kernels  $C$  to 10, kernel size  $K$  to 5, learning rate to 0.001 and initial embedding size to 300. For the loss function, we use the  $\lambda = 1.0$  and  $\rho = 0.5$ . In addition, number of clusters  $n_c$  is set to 50 for all our experiments. In the scenario wherein the input resource comes from TAXI, only hyponym-hypernym candidate pairs observed more than 10 times are used to create a noisy graph. Also, we use one pooling and one unpooling layer for our experiments.
We use dropouts in two places, one at the end of the cross-domain encoder module, and the other after the Conv1D operation. Our models are trained using NVIDIA Tesla P100 GPUs.
4.4 Results and Discussions
Table 2 shows the results on the TExEval-2 task Evaluation on science and environment domains. The first row represents a string-based baseline method (Bordea et al., 2016), that exploits term compositionality to hierarchically relate terms. For example, it extracts pairs such as (Statistics Department, Department) from the provided Wikipedia corpus, and utilizes aforementioned technique to construct taxonomy.
The next three rows in Table 2, namely, TAXI, JUNLP and USAAR are some of the top perform
4Data is available at http://panchenko.me/data/joint/taxi/res/resources.tgz
Science
(Combined)	Science
(Eurovoc)	Science
(WordNet)	Environment
(Eurovoc)
Model	Pe	Re	Fe	Pe	Re	Fe	Pe	Re	Fe	Pe	Re	Fe
Graph2Taxo(2GNN+SC+Res)	0.90	0.33	0.48	0.79	0.33	0.46	0.77	0.32	0.46	0.67	0.28	0.39
Graph2Taxo(2GNN+Res)	0.92	0.32	0.48	0.83	0.29	0.43	0.80	0.31	0.45	0.73	0.26	0.38
Graph2Taxo(2GNN)	0.90	0.33	0.48	0.81	0.29	0.42	0.81	0.31	0.45	0.74	0.25	0.37
Graph2Taxo(NoConstraint)	0.92	0.32	0.48	0.81	0.28	0.41	0.83	0.31	0.45	0.76	0.25	0.37
Graph2Taxo(Without Feas)	0.82	0.33	0.47	0.73	0.27	0.39	0.70	0.33	0.45	0.61	0.23	0.33
Graph2Taxo(AddEmbeddings)	0.90	0.33	0.48	0.80	0.33	0.47	0.77	0.32	0.46	0.71	0.28	0.40
Table 3: Ablation tests reporting the Precision, Recall and  $F$ -score, across Science and Environment domains. The first block of values reports results by ablating each layer utilized within Graph2Taxo model. In the second block, we demonstrate that addition of constraint does in fact improve performance. In the third block, we illustrate that the importance of features  $v_{\text{feas}}$  for improving performance. The final block uses pretrained fastText embeddings to initialize our Graph2Taxo model, and then fine tunes based on our training data. All results reported above are rounded off to 2 decimal places.
ing systems that participated in the TExEval-2 task. Furthermore,  $\mathrm{TaxoRL}^{A,B}$  illustrates the performance of a Reinforcement Learning system by under the Partial induction and Full induction settings respectively (Mao et al., 2018). Since Mao et al. (2018) has shown that it outperforms other methods such as Gupta et al. (2017); Bansal et al. (2014), we only compare the results of our proposed Graph2Taxo approach against the state-of-the-art system TaxoRL.
Finally, Graph2Taxo $^1$  and Graph2Taxo $^2$  depict the results of our proposed algorithm under both aforementioned settings, i.e. using the input resources of TaxoRL in the first scenario, and using the resources of TAXI in the second scenario. In each of these settings, we find that the overall precision of our proposed Graph2Taxo approach is far better than all the other existing approaches, demonstrating the strong ability of Graph2Taxo to find true relations. Meanwhile, the recall of our proposed Graph2Taxo approach is comparable to that of the existing state-of-the-art approaches. Combining the precision and recall metrics, we observe that Graph2Taxo outperforms existing state-of-the-art approaches on the F-score, by a significant margin. For example, for the Science (Average) domain, Graph2Taxo $^2$  improves over TaxoRL's F-score by  $5\%$ . For the Environment (Eurovoc) domain, our model improves TaxoRL's F-score by  $7\%$  on the TExEval-2 task.
Besides, our proposed model has high scalability. For example, the GNN method has been trained for a large graph, including about 1 million nodes (Kipf and Welling, 2017). Besides, the GNN part can be replaced by any improved GNN methods
(Hamilton et al., 2017; Gao et al., 2018) designed for large-scale graphs.
Ablation Tests. Table 3 shows the results of proposed Graph2Taxo in the second setting for the ablation experiments (divided into four blocks), which indicates the contribution of each layer used in our Graph2Taxo model. In Table 3, all the experiments are run three times, and the average values of the three runs are reported. Furthermore, in Figure 3, we randomly choose Science (Eurovoc) domain as the one to report the error-bars (corresponding to the standard-deviation values) for our experiments.
Figure 3: Results on Science (Eurovoc) domain: The average Precision, Recall and  $F$ -score values and their standard error values. It is clear that addition of Residual Layer and SC Layer lowers the variance of the results.
The first block of values in Table 3 illustrates results by ablating layers from within our Graph2Taxo model. Comparing the first two rows, it's evident that adding a Semantic Cluster (SC) layer improves recall at the cost of precision, however improving the overall F-score. This improve
ment is clearly seen for the Science (Eurovoc) domain, wherein we have an increase of  $3\%$ .
In the second block, we show that the addition of constraints improves performance. Row 4 represents a Graph2Taxo i.e.  $2\mathrm{GNN} + \mathrm{SC} + \mathrm{Res}$  setup, but without any constraint. Adding the DAG Constraint (Row 1) to this yields can get a better F-score. Specifically, we observe a major increase of  $+5\%$  F1 for the Science (Eurovoc) domain.
In the third block, we remove the features  $v_{feas}$  as mentioned in section 3.3.1. The results, i.e. row 5 in Table 3 shows that these features are critical in improving the performance of our proposed system on both Science (Eurovoc) and Environment (Eurovoc) domains. Note that these features denoted as  $v_{feas}$  are not a novelty of our proposed method, but rather have been used by existing state-of-the-art approaches.
Finally, we study the effect of initializing our model using pre-trained embeddings, rather than initializing at random. Specifically, we initialize the input matrix  $H_0$  of our Graph2Taxo model with pre-trained fastText $^5$  embeddings. Our model using fastText embeddings improves upon Row 1 by a margin of  $4\%$  in precision values for the Environment (Eurovoc) domain, but unfortunately has no significant effect on the F-score. Hence, we have not used pre-trained embeddings in reporting the results in Table 2.
We provide an illustration of the output of the Graph2Taxo model in Figure 4, for the Environment domain. The generated taxonomy in this example contains multiple trees, which serve the purpose of generating taxonomical classifications. As future work, we plan to figure out different strategies to connect the subtrees into a large graph for better DAG generation.
Figure 4: A simple example of the taxonomy generated by Graph2Taxo in the environment domain.
We have introduced a GNN-based cross-domain knowledge transfer framework Graph2Taxo, which makes use of a cross-domain graph structure, in conjunction with an acyclicity constraint-based DAG learning for taxonomy construction. Furthermore, our proposed model encodes acyclicity as a soft constraint and shows that the overall model outperforms state of the art.
In the future, we would like to figure out different strategies to merge individual gains, obtained by separate application of the DAG constraint, into a setup that can take the best of both precision and recall improvements, and put forth a better performing system. We also plan on looking into strategies to improve recall of the constructed taxonomy.
The authors would like to thank Dr. Jie Chen from MIT-IBM Watson AI Lab and Prof. Jinbo Bi from the University of Connecticut for in-depth discussions on model construction.
Daniele Alfarone and Jesse Davis. 2015. Unsupervised learning of an IS-A taxonomy from a limited domain-specific corpus. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 1434-1441. AAAI Press.
Mohit Bansal, David Burkett, Gerard de Melo, and Dan Klein. 2014. Structured learning for taxonomy induction with belief propagation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1041-1051, Baltimore, Maryland. Association for Computational Linguistics.
Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 23-32, Avignon, France. Association for Computational Linguistics.
Rianne van den Berg, Thomas N. Kipf, and Max Welling. 2017. Graph convolutional matrix completion. CoRR, abs/1706.02263.
Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. SemEval-2016 task 13: Taxonomy extraction evaluation (TExEval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1081-1091, San Diego,
$^{5}$ https://fasttext.cc
California. Association for Computational Linguistics.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
David Maxwell Chickering. 1995. Learning bayesian networks is np-complete. In Learning from Data - Fifth International Workshop on Artificial Intelligence and Statistics, AISTATS 1995, Key West, Florida, USA, January, 1995. Proceedings, pages 121-130. Springer.
David Maxwell Chickering, David Heckerman, and Christopher Meek. 2004. Large-sample learning of bayesian networks is np-hard. J. Mach. Learn. Res., 5:1287-1330.
Sarthak Dash, Md Faisal Mahbub Chowdhury, Alfio Gliozzo, Nandana Mihindukulasooriya, and Nicolas Rodolfo Fauceglia. 2020. Hypernym detection using strict partial order networks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, USA, February 7 - February 12, 2020. AAAI Press.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, USA, February 2-7, 2018, pages 1811-1818. AAAI Press.
Gerhard Friedrich and Markus Zanker. 2011. A taxonomy for generating explanations in recommender systems. AI Magazine, 32(3):90-98.
Chuang Gan, Yi Yang, Linchao Zhu, Deli Zhao, and Yueting Zhuang. 2016. Recognizing an action using its name: A knowledge-based approach. Int. J. Comput. Vis., 120(1):61-77.
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. 2018. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pages 1416-1424. ACM.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70, pages 1263-1272. PMLR.
Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729-734. IEEE.
Amit Gupta, Rémi Lebret, Hamza Harkous, and Karl Aberer. 2017. Taxonomy induction using hypernym subsequences. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 - 10, 2017.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, NeurIPS 2017, 4-9 December 2017, Long Beach, CA, USA, pages 1024-1034.
Sanda M. Harabagiu, Steven J. Maiorano, and Marius Pasca. 2003. Open-domain textual question answering techniques. Nat. Lang. Eng., 9(3):231-267.
Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In 14th International Conference on Computational Linguistics, COLING 1992, Nantes, France, August 23-28, 1992, pages 539-545.
Wen Hua, Zhongyuan Wang, Haixun Wang, Kai Zheng, and Xiaofang Zhou. 2017. Understand short texts by harvesting and analyzing semantic knowledge. IEEE Trans. Knowl. Data Eng., 29(3):499-512.
Hao Huang, Haihua Xu, Xianhui Wang, and Wushour Silamu. 2015. Maximum f1-score discriminative training criterion for automatic mispronunciation detection. IEEE ACM Trans. Audio Speech Lang. Process., 23(4):787-797.
Richard M. Karp. 1971. A simple derivation of edmonds' algorithm for optimum branchings. Networks, 1(3):265-272.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Zornitsa Kozareva and Eduard Hovy. 2010. A semi-supervised method to learn and construct taxonomies using the web. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1110-1118, Cambridge, MA. Association for Computational Linguistics.
Anh Tuan Luu, Jung-jae Kim, and See Kiong Ng. 2014. Taxonomy construction using syntactic contextual evidence. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 810-819, Doha, Qatar. Association for Computational Linguistics.
Yuning Mao, Xiang Ren, Jiaming Shen, Xiaotao Gu, and Jiawei Han. 2018. End-to-end reinforcement learning for automatic taxonomy induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2462-2472, Melbourne, Australia. Association for Computational Linguistics.
George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. 2017. Geometric deep learning on graphs and manifolds using mixture model cnns. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society.
Alexander Panchenko, Stefano Faralli, Eugen Ruppert, Steffen Remus, Hubert Naets, Cedrick Fairon, Simone Paolo Ponzetto, and Chris Biemann. 2016. TAXI at SemEval-2016 task 13: a taxonomy induction method based on lexico-syntactic patterns, substrings and focused crawling. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), San Diego, California. Association for Computational Linguistics.
Alan Ritter, Stephen Soderland, and Oren Etzioni. 2009. What is this, anyway: Automatic hypernym discovery. In Learning by Reading and Learning to Read, Papers from the 2009 AAAI Spring Symposium, Technical Report SS-09-07, Stanford, California, USA, March 23-25, 2009, pages 88-93. AAAI.
Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1025-1036, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 3060-3067. AAAI Press.
Wei Shen, Jianyong Wang, Ping Luo, and Min Wang. 2012. A graph-based approach for ontology population with named entities. In 21st ACM International Conference on Information and Knowledge Management, CIKM'12, Maui, HI, USA, October 29 - November 02, 2012, pages 345-354. ACM.
Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In Advances in Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada.