SlowGuess/NAACL_2021_OCR huggingface dataset:
We compare the predictions of ROBERTA-Large, with and without web glosses, to understand what kind of glosses help. We split the parenthood edges in the gold trees into two groups based on the glosses: (1) lexical overlap (the parent term appears in the child gloss and/or the child term appears in the parent gloss) and (2) no lexical overlap (neither the parent term nor the child term appears in the other term's gloss). We find that for edges in the "lexical overlap" group, glosses increase the recall of the gold edges from 60.9 to 67.7. For edges in the "no lexical overlap" group, retrieval decreases the recall (edge recall changes from 32.1 to 27.3).
5.7 Pretraining and Tree Reconciliation Both Contribute to Taxonomy Construction
We performed an ablation study in which we ablated either the pretrained language models for the parenthood prediction step or we ablated the tree reconciliation step. We ablated the pretrained language models in two ways. First, we used a one-layer LSTM on top of GloVe vectors instead of a pretrained language model as the input to the fine-tuning step, and then performed tree reconciliation as before. Second, we used a randomly initialized ROBERTA-Large model in place of a pretrained network, and then performed tree reconciliation as before. We ablated the tree reconciliation step by substituting the graph-based reconciliation step with a simpler threshold step, where we output a parenthood-relation between all pairs of words with softmax score greater than 0.5. We used the parenthood prediction scores from the fine-tuned ROBERTA-Large model, and substituted tree reconciliation with thresholding.
In Table 6, we show the results of our ablation experiments. These results show that both steps (using pretrained language models for parenthood-prediction and performing tree reconciliation) are
<sup>2</sup>Since the original pretraining corpus is not available, we follow Devlin et al. (2019) and recreate the dataset by crawling http://smashwords.com and Wikipedia.
Figure 6: A fragment of a subtree from the WORDNET hierarchy. Orange indicates incorrectly predicted edges and blue indicates edges that were missed.
P	R	F1
ROBERTA-Large	71.2	65.9	67.4
w/o tree reconciliation	70.8	45.8	51.1
ROBERTA-Random-Init	32.6	28.2	29.3
LSTM GloVe	32.5	23.6	26.6
Table 6: Ablation study. Pretraining and tree reconciliation both contribute to taxonomy construction.
important for taxonomy construction. Moreover, these results show that the incorporation of a new information source (knowledge learned by pretrained language models) produces the majority of the performance gains.
5.8 Models Struggle to Generalize to Large Taxonomies
To test generalization to large subtrees, we tested our models on the English environment and science taxonomies from SemEval-2016 Task 13 (Bordea et al., 2016a). Each of these taxonomies consists of a single large taxonomic tree with between 125 and 452 terms. Following Mao et al. (2018) and Shang et al. (2020), we used the medium-sized trees from Bansal et al. (2014) to train our models. During training, we excluded all medium-sized trees from the Bansal et al. (2014) dataset that overlapped with the terms in the SemEval-2016 Task 13 environment and science taxonomies.
In Table 7 we show the performance of the ROBERTA-Large CTP model. We show the EdgeF1 score rather than the Ancestor-F1 score in order to compare to previous work. Although the CTP model outperforms previous work in constructing medium-sized taxonomies, this model is limited in its ability to generalize to large taxonomies. Future
work can incorporate modeling of the global tree structure into CTP.
Taxonomy induction has been studied extensively, with both pattern-based and distributional approaches. Typically, taxonomy induction involves hypernym detection, the task of extracting candidate terms from corpora, and hypernym organization, the task of organizing the terms into a hierarchy.
While we focus on hypernym organization, many systems have studied the related task of hypernym detection. Traditionally, systems have used pattern-based features such as Hearst patterns to infer hypernym relations from large corpora (e.g. Hearst, 1992; Snow et al., 2005; Kozareva and Hovy, 2010). For example, Snow et al. (2005) propose a system that extracts pattern-based features from a corpus to predict hypernymy relations between terms. Kozareva and Hovy (2010) propose a system that similarly uses pattern-based features to predict hypernymy relations, in addition to harvesting relevant terms and using a graph-based longest-path approach to construct a legal taxonomic tree.
Later work suggests that, for hypernymy detection tasks, pattern-based approaches outperform those based on distributional models (Roller et al., 2018). Subsequent work pointed out the sparsity that exists in pattern-based features derived from corpora, and showed that combining distributional and pattern-based approaches can improve hypernymy detection by addressing this problem (Yu et al., 2020).
In this work we consider the task of organizing a set of terms into a medium-sized taxonomic tree. Bansal et al. (2014) treat this as a structured learning problem and use belief propagation to incorpo
Dataset	Model	P	R	F1
Science (Averaged)	CTP	29.4	28.8	29.1
Mao et al. (2018)	37.9	37.9	37.9
Shang et al. (2020)	84.0	30.0	44.0
Environment (Eurovoc)	CTP	23.1	23.0	23.0
Mao et al. (2018)	32.3	32.3	32.3
Shang et al. (2020)	89.0	24.0	37.0
Table 7: Generalization to large taxonomic trees. Models trained on medium-sized taxonomies generalize poorly to large taxonomies. Future work can improve the usage of global tree structure with CTP.
rate siblinghood information. Mao et al. (2018) propose a reinforcement learning based approach that combines the stages of hypernym detection and hypernym organization. In addition to the task of constructing medium-sized WORDNET subtrees, they show that their approach can leverage global structure to construct much larger taxonomies from the SemEval-2016 Task 13 benchmark dataset, which contain hundreds of terms (Bordea et al., 2016b). Shang et al. (2020) apply graph neural networks and show that they improve performance in constructing large taxonomies in the SemEval-2016 Task 13 dataset.
Another relevant line of work involves extracting structured declarative knowledge from pretrained language models. For instance, Bouraoui et al. (2019) showed that a wide range of relations can be extracted from pretrained language models such as BERT. Our work differs in that we consider tree structures and incorporate web glosses. Bosse-lut et al. (2019) use pretrained models to generate explicit open-text descriptions of commonsense knowledge. Other work has focused on extracting knowledge of relations between entities (Petroni et al., 2019; Jiang et al., 2020). Blevins and Zettlemoyer (2020) use a similar approach to ours for word sense disambiguation, and encode glosses with pretrained models.
Our experiments show that pretrained language models can be used to construct taxonomic trees. Importantly, the knowledge encoded in these pretrained language models can be used to construct taxonomies without additional web-based information. This approach produces subtrees with higher mean  $F_{1}$  scores than previous approaches, which used information from web queries.
When given web-retrieved glosses, pretrained
language models can produce improved taxonomic trees. The gain from accessing web glosses shows that incorporating both implicit knowledge of input terms and explicit textual descriptions of knowledge is a promising way to extract relational knowledge from pretrained models. Error analyses suggest specific avenues of future work, such as improving predictions for subtrees corresponding to abstractions, or explicitly modeling the global structure of the subtrees.
Experiments on aligned multilingual WORDNET datasets emphasize that more work is needed in investigating the differences between taxonomic relations in different languages, and in improving pretrained language models in non-English languages. Our results provide strong baselines for future work on constructing taxonomies for different languages.
While taxonomies (e.g., WORDNET) are often used as ground-truth data, they have been shown to contain offensive and discriminatory content (e.g., Broughton, 2019). Automatic systems created by pretrained language models can reflect and exacerbate the biases contained by their training corpora. More work is needed to detect and combat biases that arise when constructing and evaluating taxonomies.
Furthermore, we used previously constructed alignments to extend our results to wordnets in multiple languages. While considering English WORDNET as the basis for the alignments allows for convenient comparisons between languages and is the standard method for aligning wordnets across languages, continued use of these alignments to evaluate taxonomy construction imparts undue bias towards conceptual relations found in English.
We thank the members of the Berkeley NLP group and the anonymous reviewers for their insightful feedback. CC and KL are supported by National Science Foundation Graduate Research Fellowships. This research has been supported by DARPA under agreement HR00112020054. The content does not necessarily reflect the position or the policy of the government, and no official endorsement should be inferred.
A. Agirre, Egoitz Laparra, and German Rigau. 2011. Multilingual central repository version 3.0: upgrading a very large lexical knowledge base.
Mohit Bansal, David Burkett, Gerard De Melo, and Dan Klein. 2014. Structured learning for taxonomy induction with belief propagation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1041-1051.
Emily M. Bender. 2011. On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology, 6.
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss-informed biencoders. In ACL.
Francis Bond and Kyonghee Paik. 2012. A survey of wordnets and their licenses.
Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016a. SemEval-2016 task 13: Taxonomy extraction evaluation (TExEval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1081-1091, San Diego, California. Association for Computational Linguistics.
Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016b. Semeval-2016 task 13: Taxonomy extraction evaluation (texeval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, A. Çelikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In ACL.
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2019. Inducing relational knowledge from bert. arXiv preprint arXiv:1911.12753.
Vanda Broughton. 2019. The respective roles of intellectual creativity and automation in representing diversity: human and machine generated bias.
Jose Cañete, Gabriel Chaperon, Rodrigo Fuentes, JouHui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish pre-trained bert model and evaluation data. In PML4DC at ICLR 2020.
Yoeng-Jin Chu. 1965. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396-1400.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
A. Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of BlackBoxNLP 2020.
Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In *Coling* 1992 volume 2: The 15th international conference on computational linguistics.
E. Hovy, Zornitsa Kozareva, and E. Riloff. 2009. Toward completeness in concept extraction and classification. In EMNLP.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monjit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095.
Zornitsa Kozareva and Eduard Hovy. 2010. A semi-supervised method to learn and construct taxonomies using the web. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 1110-1118.
Krister Lindén and Jyrki Niemi. 2014. Is it possible to create a very large wordnet in 100 days? an evaluation. Language Resources and Evaluation, 48:191-201.
B. Magnini, C. Strapparava, F. Ciravegna, and E. Pi-anta. 1994. A project for the construction of an Italian lexical knowledge base in the framework of wordnet.
Yuning Mao, Xiang Ren, Jiaming Shen, Xiaotao Gu, and Jiawei Han. 2018. End-to-end reinforcement learning for automatic taxonomy induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2462-2472, Melbourne, Australia. Association for Computational Linguistics.
Sabrina J. Mielke. 2016. Language diversity in acl 2004 - 2016.
George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
Valeria de Paiva and Alexandre Rademaker. 2012. Revisiting a brazilian wordnet. Scopus.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473.
M. Piasecki, Stanisław Szpakowicz, and Bartosz Broda. 2009. A wordnet from the ground up.
Marten Postma, Emiel van Miltenburg, Roxane Segers, Anneleen Schoen, and Piek Vossen. 2016. Open Dutch WordNet. In Proceedings of the Eight Global Wordnet Conference, Bucharest, Romania.
Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst patterns revisited: Automatic hypernym detection from large text corpora. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 358-363, Melbourne, Australia. Association for Computational Linguistics.
Benoit Sagot. 2008. Building a free french wordnet from multilingual resources.
Chao Shang, Sarthak Dash, Md Faisal Mahbub Chowdhury, Nandana Mihindukulasooriya, and Alfio Gliozzo. 2020. Taxonomy construction of unseen domains via graph-based cross-domain knowledge transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2198-2208.
Rion Snow, Daniel Jurafsky, and Andrew Y Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Advances in neural information processing systems, pages 1297-1304.
Alon Talmor, Oyvind Tafjord, P. Clark, Y. Goldberg, and Jonathan Berant. 2020. Teaching pre-trained models to systematically reason over implicit knowledge. *ArXiv*, abs/2006.06609.
A. Virtanen, J. Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, T. Salakoski, F. Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. *ArXiv*, abs/1912.07076.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and M. Nissim. 2019. Bertje: A dutch bert model. ArXiv, abs/1912.09582.
Shan Wang and Francis Bond. 2013. Building the chinese open wordnet (cow): Starting from core synsets.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, pages arXiv-1910.
CheYu Yang and Shih-Jung Wu. 2012. Semantic web information retrieval based on the wordnet. International Journal of Digital Content Technology and Its Applications, 6:294-302.
Changlong Yu, Jialong Han, Peifeng Wang, Yangqiu Song, Hongming Zhang, Wilfred Ng, and Shuming Shi. 2020. When hearst is not enough: Improving hypernymy detection from corpus with distributional models. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP).
Language-Specific Pretrained Models
We used pretrained models from the following sources:
https://github.com/codegram/calbert,
https://github.com/google-research/bert/ blob/master/multilingual.md (Devlin et al. 2019),
http://turkunlp.org/FinBERT/ (Virtanen et al., 2019),
https://github.com/dbmdz/berts,
https://github.com/wietsedv/bertje
(de Vries et al., 2019),
https://huggingface.co/dkleczek/
bert-base-polish-uncased-v1,
https://github.com/neuralmind-ai/ portuguese-bert,
https://github.com/dccuchile/beto/blob/master/README.md (Cañete et al., 2020)
Multilingual WORDNET Dataset Statistics
Table 8 details the datasets we created by using synset alignments to the English dataset proposed in Bansal et al. (2014). The data construction method is described in Section 3.1.
Train	Num Trees Dev	Test	Average Nodes per Tree
Train	Dev	Test
ca	391	94	90	9.2	9.3	8.7
zh	216	48	64	10.0	12.4	9.2
en	533	114	114	19.7	20.3	19.8
fi	532	114	114	17.8	18.8	18.1
fr	387	82	76	8.7	9.1	8.3
it	340	85	75	6.3	7.2	6.2
nl	308	58	64	6.6	6.7	6.3
pl	283	73	72	7.7	8.0	7.4
pt	347	68	77	7.1	8.2	7.2
es	280	60	60	6.5	6.1	5.8
Table 8: Dataset Statistics. For each language, we show the number of train, dev, and test subtrees that remain after the subsetting procedure described in Section 3.1.1. In addition, we show the mean number of nodes per tree in each language. We use ISO 639-1 language acronyms.
Ablation Results
Table 9 shows the results for the learning rate trials for the ablation experiment.
1e-5	1e-6	1e-7
ROBERTA-Large	59.5	67.3	60.7
w/o tree reconciliation	38.6	51.2	18.2
ROBERTA-Random-Init	17.4	26.4	27.0
Table 9: Dev F1 Scores for Different Learning Rates, Ablation Experiments.
Table 10 shows the results for the test trials for the ablation experiment.
Run 0	Run 1	Run 2
ROBERTA-Large	67.1	67.3	67.7
w/o tree reconciliation	51.2	51.4	50.6
ROBERTA-Random-Init	27.0	29.9	31.1
LSTM GloVe	24.6	27.7	27.6
Table 10: Dev F1 Scores for Three Trials, Ablation Experiments.
SemEval Results
Dataset	Run 0	Run 1	Run 2
Science (Combined)	28.6	31.7	25.1
Science (Eurovoc)	26.6	37.1	31.5
Science (WordNet)	26.5	28.8	25.8
Environment (Eurovoc)	23.4	21.5	24.2
Table 11: Test F1 Scores for Three Trials, Semeval. We show the Edge-F1 score rather than the Ancestor-F1 score in order to compare to previous work.
Table 11 shows the results for the test trials for the SemEval experiment. These results all use the ROBERTA-Large model in the parenthood prediction step.
Random Baseline for Multilingual WORDNET Datasets
To compute the random baseline in each language, we randomly construct a tree containing the nodes in each test tree and compute the ancestor precision, recall and  $F_{1}$  score on the randomly constructed
trees. We include the  $F_{1}$  scores for three trials in Table 12.
Model	Run 0	Run 1	Run 2
Catalan	19.7	19.1	21.2
Chinese	23.5	26.8	27.0
English	8.1	8.9	9.7
Finnish	10.6	10.0	9.8
French	22.1	24.7	19.4
Italian	28.0	27.1	31.6
Dutch	29.7	27.9	22.8
Polish	20.5	22.1	27.5
Portuguese	27.9	28.1	22.2
Spanish	32.6	24.1	24.3
Table 12: Test F1 Scores for Three Trials Using a Random Baseline.
Subtree Construction Results, English WordNet
Table 13 shows the results for the learning rate trials for the English WORDNET experiment.
Model	1e-5	1e-6	1e-7
BERT	60.0	63.3	60.7
BERT-Large	59.5	67.3	65.8
ROBERTA-Large	56.3	67.1	65.5
ROBERTA-Large
(Web-retrieved Glosses)	58.6	71.5	64.7
ROBERTA Large
(WordNet Glosses)	63.0	83.7	82.9
Table 14 shows the results for the test trials for the English WORDNET experiment.
Subtree Construction Results, Multilingual WordNet
Table 15 shows the results for the learning rate trials for the non-English WORDNET experiments.
Table 16 shows the results for the test trials for the non-English WORDNETexperiments.
Table 13: Dev Results for Different Learning Rates, English Models. We highlight in bold the best learning rate for each model.
Model	Run 0	Run 1	Run 2
BERT	53.6	54.0	52.5
BERT-Large	58.9	61.5	63.8
ROBERTA-Large	62.9	64.2	63.3
ROBERTA-Large (Web-retrieved glosses)	66.6	66.3	67.1
ROBERTA-Large (WordNet glosses)	82.4	84.0	83.2
Table 14: Test F1 Scores for Three Trials, English.
Language	Model	1e-5	1e-6	1e-7
Catalan	Calbert	39.9	37.9	24.5
mBERT	39.7	43.5	32.6
Chinese	Chinese BERT	56.9	59.0	54.3
mBERT	57.4	60.6	44.7
Finnish	FinBERT	45.6	50.1	47.0
mBERT	24.6	30.2	28.9
French	French BERT	48.9	50.6	46.9
mBERT	40.3	41.1	32.5
Italian	Italian BERT	52.6	52.2	46.9
mBERT	50.7	51.8	41.3
Dutch	BERTje	49.0	48.8	38.1
mBERT	44.9	44.5	32.9
Polish	Polbert	54.2	52.9	48.2
mBERT	53.0	50.7	36.4
Portuguese	BERTimbau	51.2	52.0	42.1
mBERT	38.5	37.8	28.0
Spanish	BETO	56.7	57.4	52.8
mBERT	49.5	41.5	40.4
Table 15: Dev Results for Different Learning Rates, Multilingual. We highlight in bold the best learning rate for each model.
Table 17 shows the results for all tested models for the non-English WORDNET experiments.
Language	Model	Run 0	Run 1	Run 2
Catalan	Calbert	36.5	34.1	33.6
mBERT	39.4	41.8	32.7
Chinese	Chinese BERT	57.1	62.3	56.8
mBERT	55.2	59.4	58.0
Finnish	FinBERT	43.6	44.6	43.2
mBERT	25.5	26.3	26.7
French	French BERT	47.5	49.5	50.4
mBERT	41.0	40.9	38.9
Italian	Italian BERT	43.2	47.2	47.8
mBERT	42.9	43.6	49.3
Dutch	BERTje	43.8	44.9	42.4
mBERT	35.9	33.0	27.1
Polish	Polbert	51.2	49.9	47.3
mBERT	40.1	42.0	41.5
Portuguese	BERTimbau	57.6	57.4	55.8
mBERT	38.4	38.2	34.3
Spanish	BETO	50.8	53.4	50.9
mBERT	48.7	49.3	44.0
Table 16: Test F1 Scores for Three Trials, Multilingual.
Language	Model	P	R	F1
Catalan	Calbert	39.3	32.4	34.7
mBERT	38.7	39.7	38.0
Chinese	Chinese BERT	62.2	57.3	58.7
mBERT	61.9	56.0	57.5
Finnish	FinBERT	47.9	42.6	43.8
mBERT	29.6	25.4	26.2
French	French BERT	51.3	49.1	49.1
mBERT	43.3	40.0	40.3
Italian	Italian BERT	48.3	45.5	46.1
mBERT	47.6	44.6	45.3
Dutch	BERTje	44.6	44.8	43.7
mBERT	34.3	31.6	32.0
Polish	Polbert	51.9	49.7	49.5
mBERT	43.7	41.4	41.2
Portuguese	BERTimbau	59.3	57.1	56.9
mBERT	38.7	38.2	37.0
Spanish	BETO	53.1	51.7	51.7
mBERT	47.3	49.4	47.3
Table 17: Multilingual WORDNET Test Results. We use ISO 639-1 acronyms to indicate languages.