SlowGuess/NAACL_2021_OCR huggingface dataset:
Catherine Chen* Kevin Lin* Dan Klein
{cathychen,k-lin,klein}@berkeley.edu
We present a method for constructing taxonomic trees (e.g., WORDNET) using pretrained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that reconciles those predictions into trees. The parenthood prediction module produces likelihood scores for each potential parent-child pair, creating a graph of parent-child relation scores. The tree reconciliation module treats the task as a graph optimization problem and outputs the maximum spanning tree of this graph. We train our model on subtrees sampled from WORDNET, and test on non-overlapping WORDNET subtrees. We show that incorporating web-retrieved glosses can further improve performance. On the task of constructing subtrees of English WORDNET, the model achieves 66.7 ancestor  $F_{1}$ , a  $20.0\%$  relative increase over the previous best published result on this task. In addition, we convert the original English dataset into nine other languages using OPEN MULTILINGUAL WORDNET and extend our results across these languages.
A variety of NLP tasks use taxonomic information, including question answering (Miller, 1998) and information retrieval (Yang and Wu, 2012). Taxonomies are also used as a resource for building knowledge and systematicity into neural models (Peters et al., 2019; Geiger et al., 2020; Talmore et al., 2020). NLP systems often retrieve taxonomic information from lexical databases such as WORDNET (Miller, 1998), which consists of taxonomies that contain semantic relations across many domains. While manually curated taxonomies provide useful information, they are incomplete and expensive to maintain (Hovy et al., 2009).
Traditionally, methods for automatic taxonomy construction have relied on statistics of web-scale corpora. These models generally apply lexicosyntactic patterns (Hearst, 1992) to large corpora, and use corpus statistics to construct taxonomic trees (e.g., Snow et al., 2005; Kozareva and Hovy, 2010; Bansal et al., 2014; Mao et al., 2018; Shang et al., 2020).
Figure 1: An example subtree from the WORDNET hierarchy.
In this work, we propose an approach that constructs taxonomic trees using pretrained language models (CTP). Our results show that direct access to corpus statistics at test time is not necessary. Indeed, the re-representation latent in large-scale models of such corpora can be beneficial in constructing taxonomies. We focus on the task proposed by Bansal et al. (2014), where the task is to organize a set of input terms into a taxonomic tree. We convert this dataset into nine other languages using synset alignments collected in OPEN MULTILINGUAL WORDNET and evaluate our approach in these languages.
CTP first finetunes pretrained language models to predict the likelihood of pairwise parent-child relations, producing a graph of parenthood scores. Then it reconciles these predictions with a maximum spanning tree algorithm, creating a tree-structured taxonomy. We further test CTP in a setting where models have access to web-retrieved glosses. We reorder the glosses and finetune the
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4687-4700 June 6-11, 2021. ©2021 Association for Computational Linguistics
model on the reordered glosses in the parenthood prediction module.
We compare model performance on subtrees across semantic categories and subtree depth, provide examples of taxonomic ambiguities, describe conditions for which retrieved glosses produce greater increases in tree construction  $F_{1}$  score, and evaluate generalization to large taxonomic trees (Bordea et al., 2016a). These analyses suggest specific avenues of future improvements to automatic taxonomy construction.
Even without glosses, CTP achieves a 7.9 point absolute improvement in  $F_{1}$  score on the task of constructing WORDNET subtrees, compared to previous work. When given access to the glosses, CTP obtains an additional 3.2 point absolute improvement in  $F_{1}$  score. Overall, the best model achieves a 11.1 point absolute increase (a  $20.0\%$  relative increase) in  $F_{1}$  score over the previous best published results on this task.
Our paper is structured as follows. In Section 2 we describe CTP, our approach for taxonomy construction. In Section 3 we describe the experimental setup, and in Section 4 we present the results for various languages, pretrained models, and glosses. In Section 5 we analyze our approach and suggest specific avenues for future improvement. We discuss related work and conclude in Sections 6 and 7.
2 Constructing Taxonomies from Pretrained Models
2.1 Taxonomy Construction
We define taxonomy construction as the task of creating a tree-structured hierarchy  $T = (V, E)$ , where  $V$  is a set of terms and  $E$  is a set of directed edges representing hypernym relations. In this task, the model receives a set of terms  $V$ , where each term can be a single word or a short phrase, and it must construct the tree  $T$  given these terms. CTP performs taxonomy construction in two steps: parenthood prediction (Section 2.2) followed by graph reconciliation (Section 2.3).
We provide a schematic description of CTP in Figure 2 and provide details in the remainder of this section.
2.2 Parenthood Prediction
We use pretrained models (e.g., BERT) to predict the edge indicators  $\mathbb{I}[\text{parent}(v_i, v_j)]$ , which denote whether  $v_i$  is a parent of  $v_j$ , for all pairs  $(v_i, v_j)$  in
the set of terms  $V = \{v_{1},\dots,v_{n}\}$  for each subtree  $T$
To generate training data from a tree  $T$  with  $n$  nodes, we create a positive training example for each of the  $n - 1$  parenthood edges and a negative training example for each of the  $\frac{n(n - 1)}{2} - (n - 1)$  pairs of nodes that are not connected by a parenthood edge.
We construct an input for each example using the template  $v_{i}$  is a  $v_{j}$ , e.g., "A dog is a mammal." Different templates (e.g., [TERM_A] is an example of [TERM_B] or [TERM_A] is a type of [TERM_B]) did not substantially affect model performance in initial experiments, so we use a single template. The inputs and outputs are modeled in the standard format (Devlin et al., 2019).
We fine-tune pretrained models to predict  $\mathbb{I}[\text{parent}(v_i, v_j)]$ , which indicates whether  $v_i$  is the parent of  $v_j$ , for each pair of terms using a sentence-level classification task on the input sequence.
2.3 Tree Reconciliation
We then reconcile the parenthood graph into a valid tree-structured taxonomy. We apply the Chu-Liu-Edmonds algorithm to the graph of pairwise parenthood predictions. This algorithm finds the maximum weight spanning arborescence of a directed graph. It is the analog of MST for directed graphs, and finds the highest scoring arborescence in  $O(n^{2})$  time (Chu, 1965).
2.4 Web-Retrieved Glosses
We perform experiments in two settings: with and without web-retrieved glosses. In the setting without glosses, the model performs taxonomy construction using only the set of terms  $V$ . In the setting with glosses, the model is provided with glosses retrieved from the web. For settings in which the model receives glosses, we retrieve a list of glosses  $d_v^1, \ldots, d_v^n$  for each term  $v \in V$ .
Many of the terms in our dataset are polysemous, and the glosses contain multiple senses of the word. For example, the term dish appears in the subtree we show in Figure 1. The glosses for dish include (1) (telecommunications) A type of antenna with
<sup>1</sup>We scrape glosses from wiktionary.com, merriam-webster.com, and wikipedia.org. For wiktionary.com and merriam-webster.com we retrieve a list of glosses from each site. For wikipedia.org we treat the first paragraph of the page associated with the term as a single gloss. The glosses were scraped in August 2020.
1. A type of antenna with a similar shape to a plate or bowl.
2. A specific type of prepared food.  
3. A trough in which ore is measured
W
X
A specific type of prepared food. A trough in which ore is measured
A type of antenna with a similar shape to a plate or bowl.
C
Figure 2: A schematic depiction of CTP. We start with a set of terms (A). We fine-tune a pretrained language model to predict pairwise parenthood relations between pairs of terms (B), creating a graph of parenthood predictions (C) (Section 2.2). We then reconcile the edges of this graph into a taxonomic tree (E) (Section 2.3). Optionally, we provide the model ranked web-retrieved glosses (Section 2.4). We re-order the glosses based on relevance to the current subtree (Z).
a similar shape to a plate or bowl, (2) (metonymically) A specific type of prepared food, and (3) (mining) A trough in which ore is measured.
We reorder the glosses based on their relevance to the current subtree. We define relevance of a given context  $d_v^i$  to subtree  $T$  as the cosine similarity between the average of the GloVe embeddings (Pennington et al., 2014) of the words in  $d_v^i$  (with stopwords removed), to the average of the GloVe embeddings of all terms  $v_{1},\ldots ,v_{n}$  in the subtree. This produces a reordered list of glosses  $d_v^{(1)},\dots,d_v^{(n)}$ .
We then use the input sequence containing the reordered glosses “[CLS]  $v_{i}d_{v_{i}}^{(1)}, \ldots, d_{v_{i}}^{(n)}$ . [SEP]  $v_{j}d_{v_{j}}^{(1)}, \ldots, d_{v_{j}}^{(n)}$ ” to fine-tune the pretrained models on pairs of terms  $(v_{i}, v_{j})$ .
In this section we describe the details of our datasets (Section 3.1), and describe our evaluation metrics (Section 3.2). We ran our experiments on a cluster with 10 Quadro RTX 6000 GPUs. Each training runs finishes within one day on a single GPU.
We evaluate CTP using the dataset of medium-sized WORDNET subtrees created by Bansal et al. (2014). This dataset consists of bottomed-out full
subtrees of height 3 (this corresponds to trees containing 4 nodes in the longest path from the root to any leaf) that contain between 10 and 50 terms. This dataset comprises 761 English trees, with 533/114/114 train/dev/test trees respectively.
3.1.1 Multilingual WORDNET
WORDNET was originally constructed in English, and has since been extended to many other languages such as Finnish (Magnini et al., 1994), Italian (Lindén and Niemi, 2014), and Chinese (Wang and Bond, 2013). Researchers have provided alignments from synsets in English WORDNET to terms in other languages, using a mix of automatic and manual methods (e.g., Magnini et al., 1994; Lindén and Niemi, 2014). These multilingual wordnets are collected in the OPEN MULTILINGUAL WORDNET project (Bond and Paik, 2012). The coverage of synset alignments varies widely. For instance, the alignment of ALBANET (Albanian) to English WORDNET covers  $3.6\%$  of the synsets in the Bansal et al. (2014) dataset, while the FINNWORDNET (Finnish) alignment covers  $99.6\%$  of the synsets in the dataset.
We convert the original English dataset to nine other languages using the synset alignments. (We create datasets for Catalan (Agirre et al., 2011), Chinese (Wang and Bond, 2013), Finnish (Lindén and Niemi, 2014), French (Sagot, 2008), Italian (Magnini et al., 1994), Dutch (Postma et al., 2016),
Polish (Piasecki et al., 2009), Portuguese (de Paiva and Rademaker, 2012), and Spanish (Agirre et al., 2011)).
Since these wordnets do not include alignments to all of the synsets in the English dataset, we convert the English dataset to each target language using alignments specified in WORDNET as follows. We first exclude all subtrees whose roots are not included in the alignment between the WORDNET of the target language and English WORDNET. For each remaining subtree, we remove any node that is not included in the alignment. Then we remove all remaining nodes that are no longer connected to the root of the corresponding subtrees. We describe the resulting dataset statistics in Table 8 in the Appendix.
As with previous work (Bansal et al., 2014; Mao et al., 2018), we report the ancestor  $F_{1}$  score  $\frac{2PR}{P + R}$  where
$$
P = \frac {\left| \mathrm {I S} _ {-} \mathrm {A P R E D I C T E D} \cap \mathrm {I S} _ {-} \mathrm {A G O L D} \right|}{\left| \mathrm {I S} _ {-} \mathrm {A P R E D I C T E D} \right|}
$$
$$
R = \frac {\left| \mathrm {I S} _ {-} \mathrm {A P R E D I C T E D} \cap \mathrm {I S} _ {-} \mathrm {A G O L D} \right|}{\left| \mathrm {I S} _ {-} \mathrm {A G O L D} \right|}
$$
IS_APREDICTED and IS_AGOLD denote the set of predicted and gold ancestor relations, respectively. We report the mean precision  $(P)$ , recall  $(R)$ , and  $F_{1}$  score, averaged across the subtrees in the test set.
In our experiments, we use pretrained models from the Huggingface library (Wolf et al., 2019). For the English dataset we experiment with BERT, BERT-Large, and ROBERTA-Large in the parenthood prediction module. We experiment with multilingual BERT and language-specific pretrained models (detailed in Section 9 in the Appendix). We finetuned each model using three learning rates {1e-5, 1e-6, 1e-7}. For each model, we ran three trials using the learning rate that achieved the highest dev  $F_{1}$  score. In Section 4, we report the average scores over three trials. We include full results in Tables 13 and 15 in the Appendix. The code and datasets are available at https://github.com/cchen23/ctp.
Our approach, CTP, outperforms existing state-of-the-art models on the WORDNET subtree construction task. In Table 1 we provide a comparison of our results to previous work. Even without retrieved glosses, CTP with ROBERTA-LARGE in the parenthood prediction module achieves higher  $F_{1}$  than previously published work. CTP achieves additional improvements when provided with the web-retrieved glosses described in Section 2.4.
We compare different pretrained models for the parenthood prediction module, and provide these comparisons in Section 4.3.
P	R	F1
Bansal et al. (2014)	48.0	55.2	51.4
Mao et al. (2018)	52.9	58.6	55.6
CTP (no glosses)	67.3	62.0	63.5
CTP (web glosses)	69.3	66.2	66.7
Table 1: English Results, Comparison to Previous Work. Our approach outperforms previous approaches on reconstructing WORDNET subtrees, even when the model is not given web-retrieved glosses.
4.2 Web-Retrieved Glosses
In Table 2 we show the improvement in taxonomy construction with two types of glosses - glosses retrieved from the web (as described in Section 2.4), and those obtained directly from WORDNET. We consider using the glosses from WORDNET as an oracle setting since these glosses are directly generated from the gold taxonomies. Thus, we focus on the web-retrieved glosses as the main setting. Models produce additional improvements when given WORDNET glosses. These improvements suggest that reducing the noise from web-retrieved glosses could improve automated taxonomy construction.
4.3 Comparison of Pretrained Models
For both settings (with and without web-retrieved glosses), CTP attains the highest  $F_{1}$  score when ROBERTA-Large is used in the parenthood prediction step. As we show in Table 3, the average  $F_{1}$  score improves with both increased model size and with switching from BERT to ROBERTA.
P	R	F1
CTP	67.3	62.0	63.5
+ web glosses	69.3	66.2	66.7
+ oracle glosses	84.0	83.8	83.2
Table 2: English Results, Gloss Comparison on Test Set. Adding web glosses improves performance over only using input terms. Models achieve additional improvements in subtree reconstruction when given oracle glosses from WORDNET, showing possibilities for improvement in retrieving web glosses.
P	R	F1
CTP (BERT-Base)	57.9	51.8	53.4
CTP (BERT-Large)	65.5	59.8	61.4
CTP (ROBERTA-Large)	67.3	62.0	63.5
4.4 Aligned Wordnets
We extend our results to the nine non-English alignments to the Bansal et al. (2014) dataset that we created. In Table 4 we compare our best model in each language to a random baseline. We detail the random baseline in Section 9 in the Appendix and provide results from all tested models in Section 17 in the Appendix.
CTP's  $F_{1}$  score non-English languages is substantially worse than its  $F_{1}$  score on English trees. Lower  $F_{1}$  scores in non-English languages are likely due to multiple factors. First, English pretrained language models generally perform better than models in other languages because of the additional resources devoted to the development of English models. (See e.g., Bender, 2011; Mielke, 2016; Joshi et al., 2020). Second, OPEN MULTILINGUAL WORDNET aligns wordnets to English WORDNET, but the subtrees contained in English WORDNET might not be the natural taxonomy in other languages. However, we note that scores across languages are not directly comparable as dataset size and coverage vary across languages (as we show in Table 8).
These results highlight the importance of evaluating on non-English languages, and the difference in available lexical resources between languages. Furthermore, they provide strong baselines for fu
Table 3: English Results, Comparison of Pretrained Models on Test Set. Larger models perform better and ROBERTA outperforms BERT.
Model	P	R	F1
ca	Random Baseline	20.0	31.3	23.6
CTP (MBERT)	38.7	39.7	38.0
zh	Random Baseline	25.8	35.9	29.0
CTP (CHINESE BERT)	62.2	57.3	58.7
en	Random Baseline	8.9	22.2	12.4
CTP (ROBERTA-Large)	67.3	62.0	63.5
fi	Random Baseline	10.1	22.5	13.5
CTP (FINBERT)	47.9	42.6	43.8
fr	Random Baseline	22.1	34.4	25.9
CTP (FRENCH BERT)	51.3	49.1	49.1
it	Random Baseline	28.9	39.4	32.3
CTP (ITALIAN BERT)	48.3	45.5	46.1
nl	Random Baseline	26.8	38.4	30.6
CTP (BERTJE)	44.6	44.8	43.7
pl	Random Baseline	23.4	33.6	26.8
CTP (POLBERT)	51.9	49.7	49.5
pt	Random Baseline	26.1	37.6	29.8
CTP (BERTIMBAU)	59.3	57.1	56.9
es	Random Baseline	27.0	37.2	30.5
CTP (BETO)	53.1	51.7	51.7
Table 4: Multilingual WORDNET Test Results. We extend our model to datasets in nine other languages, and evaluate our approach on these datasets. We use ISO 639-1 acronyms to indicate languages.
ture work in constructing wordnets in different languages.
In this section we analyze the models both quantitatively and qualitatively. Unless stated otherwise, we analyze our model on the dev set and use ROBERTA-Large in the parenthood prediction step.
5.1 Models Predict Flatter Trees
In many error cases, CTP predicts a tree with edges that connect terms to their non-parent ancestors, skipping the direct parents. We show an example of this error in Figure 3. In this fragment (taken from one of the subtrees in the dev set), the model predicts a tree in which botfly and horsefly are direct children of fly, bypassing the correct parent gadfly. On the dev set,  $38.8\%$  of incorrect parenthood edges were cases of this type of error.
Figure 3: A fragment of a subtree from the WORD-NET hierarchy. Orange indicates incorrectly predicted edges and blue indicates missed edges.
Missing edges result in predicted trees that are generally flatter than the gold tree. While all the gold trees have a height of 3 (4 nodes in the longest path from the root to any leaf), the predicted dev trees have a mean height of 2.61. Our approach scores the edges independently, without considering the structure of the tree beyond local parenthood edges. One potential way to address the bias towards flat trees is to also model the global structure of the tree (e.g., ancestor and sibling relations).
5.2 Model Struggle Near Leaf Nodes
d=1	d=2	d=3
l=1	81.2	52.3	39.7
l=2	74.4	48.9
l=3	66.0
Table 5: Ancestor Edge Recall, Categorized by Descendant Node Depth  $d$  and Parent Edge Length  $l$ . Ancestor edge prediction recall decreases with deeper descendant nodes and closer ancestor-descendant relations.
CTP generally makes more errors in predicting edges involving nodes that are farther from the root of each subtree. In Table 5 we show the recall of ancestor edges, categorized by the number of parent edges  $d$  between the subtree root and the descendant of each edge, and the number of parent edges  $l$  between the ancestor and descendant of each edge. The model has lower recall for edges involving descendants that are farther from the root (higher  $d$ ). In permutation tests of the correlation between edge recall and  $d$  conditioned on  $l$ , 0 out of 100,000 permutations yielded a correlation at least as extreme as the observed correlation.
5.3 Subtrees Higher Up in WORDNET are Harder, and Physical Entities are Easier than Abstractions
Subtree performance also corresponds to the depth of the subtree in the entire WORDNET hierarchy. The  $F_{1}$  score is positively correlated with the depth of the subtree in the full WORDNET hierarchy, with a correlation of 0.27 (significant at  $\mathsf{p} = 0.004$  using a permutation test with 100,000 permutations).
The subtrees included in this task span many different domains, and can be broadly categorized into subtrees representing concrete entities (such as telephone) and those representing abstractions (such as sympathy). WORDNET provides this categorization using the top-level synsets physical_entity.n.01 and abstraction.n.06. These categories are direct children of the root of the full WORDNET hierarchy (entity.n.01), and split almost all WORDNET terms into two subsets. The model produces a mean  $F_{1}$  score of 60.5 on subtrees in the abstraction subsection of WORDNET, and a mean  $F_{1}$  score of 68.9 on subtrees in the physical-entity subsection. A one-sided Mann-Whitney rank test shows that the model performs systematically worse on abstraction subtrees (compared to physical entity subtrees)  $(p = 0.01)$ .
5.4 Pretraining Corpus Covers Most Terms
Figure 4: Frequency of terms in the WORDNET dataset in the pretraining corpus. Over  $97\%$  of terms in the Bansal et al. (2014) dataset occur at least once in the pretraining corpus. Over  $80\%$  of terms occur less than 50k times.
With models pretrained on large web corpora, the distinction between the settings with and without access to the web at test time is less clear, since large pretrained models can be viewed as a com
pressed version of the web. To quantify the extent the evaluation setting measures model capability to generalize to taxonomies consisting of unseen words, we count the number of times each term in the WORDNET dataset occurs in the pretraining corpus. We note that the WORDNET glosses do not directly appear in the pretraining corpus. In Figure 4 we show the distribution of the frequency with which the terms in the Bansal et al. (2014) dataset occur in the BERT pretraining corpus.<sup>2</sup> We find that over  $97\%$  of the terms occur at least once in the pretraining corpus. However, the majority of the terms are not very common words, with over  $80\%$  of terms occurring less than 50k times. While this shows that the current setting does not measure model ability to generalize to completely unseen terms, we find that the model does not perform substantially worse on edges that contain terms that do not appear in the pretraining corpus. Furthermore, the model is able do well on rare terms. Future work can investigate model ability to construct taxonomies from terms that are not covered in pretraining corpora.
5.5 WORDNET Contains Ambiguous Subtrees
Figure 5: A fragment of a subtree from the WORD-NET hierarchy. Orange indicates incorrectly predicted edges and blue indicates edges that were missed.
Some trees in the gold WORDNET hierarchy contain ambiguous edges. Figure 5 shows one example. In this subtree, the model predicts arteriography as a sibling of arthrography rather than as its child. The definitions of these two terms suggest why the model may have considered these terms as siblings: arteriograms produce images of arteries while
arthrograms produce images of the inside of joints. In Figure 6 we show a second example of an ambiguous tree. The model predicts good faith as a child of sincerity rather than as a child of honesty, but the correct hypernymy relation between these terms is unclear to the authors, even after referencing multiple dictionaries.
These examples point to the potential of augmenting or improving the relations listed in WORD-NET using semi-automatic methods.
5.6 Web-Retrieved Glosses Are Beneficial When They Contain Lexical Overlap