Seeking Clozure: Robust Hypernym Extraction from BERT with Anchored Prompts
Year: 2023

Abstract:
The automatic extraction of hypernym knowledge from large language models like BERT is an open problem, and it is unclear whether methods fail due to a lack of knowledge in the model or shortcomings of the extraction methods. In particular, methods fail on challenging cases which include rare or abstract concepts, and perform inconsistently under paraphrased prompts. In this study, we revisit the long line of work on pattern-based hypernym extraction, and use it as a diagnostic tool to thoroughly examine the hypernomy knowledge encoded in BERT and the limitations of hypernym extraction methods. We propose to construct prompts from established pattern structures: definitional (X is a Y); lexico-syntactic (Y such as X); and their anchored versions (Y such as X or Z). We devise an automatic method for anchor prediction, and compare different patterns in: (i) their effectiveness for hypernym retrieval from BERT across six English data sets; (ii) on challenge sets of rare and abstract concepts; and (iii) on consistency under paraphrasing. We show that anchoring is particularly useful for abstract concepts and in enhancing consistency across paraphrases, demonstrating how established methods in the field can inform prompt engineering.1

heading: 1 Introduction
text: Semantic relations play a central role in knowledge representation (Miller, 1995) and taxonomy construction (Snow et al., 2006; Navigli et al., 2011). As the backbone of semantic relations, hyponymy/hypernymy relations express a hierarchical relation between a specific concept (the hyponym; e.g., dog) and a general one (the hypernym; e.g., mammal), and form the foundation of human concept understanding (Yu et al., 2015) and relation reasoning (Lyons, 1977; Green et al.,
∗Now at Google DeepMind. 1Code and test sets are available at https://github.
com/ChunhuaLiu596/AnchoredPrompts
2002). Given its fundamental role, the automatic extraction of hypernym knowledge from large texts (Hearst, 1992; Roller et al., 2018) or pre-trained language models (PLMs) (Takeoka et al., 2021; Jain and Espinosa Anke, 2022), and its injection into NLP methods are active areas of research (Peters et al., 2019).
The unsupervised extraction of hypernyms from PLMs by prompting has attracted recent attention, e.g., using patterns like A dog is a type of [MASK] and retrieving the most likely filler words from the model (Ettinger, 2020; Weir et al., 2020; Jain and Espinosa Anke, 2022). Results were mixed: while PLMs can reliably predict hypernyms of concrete and frequent hyponyms (Ettinger, 2020; Weir et al., 2020), experiments on more challenging data sets show a quick deterioration in the face of rare concepts (Schick and Schütze, 2019), and a lack of response consistency across paraphrased prompts (Ravichander et al., 2020; Elazar et al., 2021). How to alleviate these issues and extract more reliable hypernyms from PLMs remain open questions.
In this paper, we draw connections between prompting for hypernyms and pattern-based hypernym extraction (Hearst, 1992; Snow et al., 2004) (see Figure 1 and Table 1). We systematically investigate the utility of different styles of patterns as BERT prompts, and use them as diagnostic tools
193
to better understand the conditions under which probing for hypernyms is effective and consistent.
Pattern-based hypernym extraction from raw text has a long history, starting from Hearst (1992)’s seminal work which promotes lexico-syntactic patterns (Y such as X)2 as more effective than definitional patterns (X is a type of Y). Follow-up work (Hovy et al., 2009) incorporated a co-hyponym, a concept that shares a hypernym with X, into the pattern (Y such as X and Z) to provide additional context signals. Figure 1 illustrates this, where the anchor parrot provides additional information to facilitate the prediction of the correct hypernym of kea. This method of ‘anchoring’ has been shown to improve the quality of automatically extracted hypernym knowledge. We apply these established patterns from the hypernym extraction literature in the context of language model prompting, and systematically study the existence and gaps of hyponym/hypernym knowledge in BERT. We conduct experiments on six English data sets and address three questions:
How to effectively construct anchored prompts? We devise a scalable method to automatically retrieve high-quality anchors (co-hyponyms) to construct anchored prompts. Anchors are mined from PLMs with established co-hyponym patterns (e.g., such as X and ) and evaluated with WordNet (Miller, 1995).
How do different pattern structures compare as prompts under different data conditions? We ground our prompts in hypernym patterns from which have been successfully used to mine hypernyms from raw corpora, and investigate their effectiveness for zero-shot PLM hypernym retrieval. We find strong, consistent benefits of anchored prompts, particularly for rare or abstract concepts.
Robust extraction of hypernym knowledge. Much recent work has shown that PLM prompting results are brittle under prompt paraphrases, calling into question whether prompting surfaces robust knowledge encoded in the PLMs or rather superficial associations. We compare the robustness of different patterns under paraphrasing, and find, again, a benefit of anchored prompts for retrieving more consistently correct hypernyms.
In summary, we contribute to the on-going research on hypernym extraction by unifying the long-standing work of pattern-based and prompt-
2We use Y to denote hypernyms, X for hyponyms and Z for the co-hyponym of X .
based approaches, demonstrating that anchoring prompts can unlock a wealth of hidden knowledge within BERT, and providing a framework of automatic construction of anchoring prompts.

heading: 2 Background
text: We introduce the two approaches for hypernym extraction on which we build in this paper: patternbased (§ 2.1) and prompting PLMs (§ 2.2).

heading: 2.1 Pattern-based Hypernym Extraction
text: The pattern-based approach applies hyponymhypernym patterns to large corpora to extract hypernyms. Two widely-used pattern structures have been identified: lexico-syntactic and definitional.
2.1.1 Lexico-Syntactic Patterns (LSP) Lexico-syntactic patterns (LSP; Table 1 bottom left) were first introduced by Hearst (1992) and have since been used to mine hyponym-hypernym pairs or build ontologies from large corpora (Pasca, 2004; Pantel and Pennacchiotti, 2006; Etzioni et al., 2005; Roller et al., 2018). The six LSP (1) all indicate the hyponym-hypernym relation with explicit signals (e.g., such as, especially), (2) frequently occur in text, and (3) are applicable to nouns or noun-phrases.
Anchored LSP (LSP+A) Hovy et al. (2009) proposed an ‘anchored’ version of LSP to mine hypernyms (LSP+A)3 which uses patterns like Y such as X and Z, where Z is an anchor which reduces ambiguity and assists the extraction of Y (Table 1, bottom right). A similar idea of using anchors to improve hypernym classifiers is used in Snow et al. (2004) and Bernier-Colborne and Barrière (2018). LSP+A has been shown to be effective at extracting reliable hypernyms from text corpora, however,
3LSP+A is referred to as DAP-1 in the original paper.
like all pattern-based approaches, it suffers from low recall because it needs X, Y and Z to co-occur. The sparsity issue can be potentially remedied by using embeddings from PLMs to represent X and Z when used as prompts. However, this hasn’t been studied in the context of extracting knowledge from PLMs. Inspired by this line of work and PLM prompting, we use LSP+A to mine hypernyms from PLMs and examine the benefit of anchors.
2.1.2 Definitional Patterns (DFP) In contrast to LSP that conveys the hypernym relation implicitly, definitional Patterns (DFP; Table 1 top left) explicitly define an Is-A relation between X and Y (Lyons, 1977). A common use of DFP is to mine sentences for definition extraction (Borg et al., 2009; Navigli et al., 2010) or ontology/dictionary building (Muresan and Klavans, 2002). Recently, DFP has been widely used in prompting studies (Schick and Schütze, 2020; Ettinger, 2020; Ravichander et al., 2020; Hanna and Mareček, 2021) to probe hypernym knowledge in PLMs.
Anchored DFP (DFP+A) Analogous to LSP+A, we augment DFP with anchors for disambiguation (Table 1 top right). To the best of our knowledge, Hanna and Mareček (2021) is the only work which uses anchored definitional patterns to prompt PLMs for hypernyms, described in more detail below.

heading: 2.2 Prompting-based Hypernym Extraction
text: With recent advances in PLMs, increasingly rich knowledge is captured language models. A stream of research aims at automatically extracting this knowledge, e.g., by probing PLMs for hypernym knowledge (Ettinger, 2020; Weir et al., 2020; Peng et al., 2022). Hanna and Mareček (2021) examined the effects of single hypernym patterns (e.g., ‘X is a Y’, ‘A Y such as X’) on prompting PLMs and
showed that performance varies with patterns. Similarly, Ravichander et al. (2020) found that PLMs fail to retrieve consistent knowledge over prompts paraphrased with singular vs plural hyponyms.
Most previous work on prompting was conducted under relatively simple conditions with one pattern structure and a single data set. We systematically investigate the effects of well-established patterns (LSP/LSP+A and DFP/DFP+A) on extracting hypernyms across six widely-used datasets and paint a more nuanced picture of hypernym knowledge in BERT by explicitly studying the challenging cases of rare or abstract concepts.

heading: 3 Anchored Prompts
text: We now introduce our framework of extracting hypernyms from a PLM by constructing sets of prompts given a hyponym X and a pattern type ∈ {DFP, DFP+A, LSP, LSP+A }. We illustrate the workflow in Figure 2, with LSP+A as an example.
Prompt Construction For each pattern type, we construct a set of prompts by instantiating each of its assigned patterns (cells in Table 1) with a concept in positions X and Z, and a [MASK] token in position Y. For DFP and LSP we can construct prompt sets directly given a hyponym X of interest. To construct prompts for LSP+A and DFP+A we need to additionally provide meaningful anchors Z. We next describe a way to effectively mine such anchors from language models (see Figure 2 (a)).
Anchor Extraction Given X, we use BERT to automatically extract a set of anchors, i.e., concepts Z that share a hypernym with X. To acquire such anchors, we again adopt a set of established lexico-syntactic patterns that indicate the fact that X and Z share a common hypernym (Hearst, 1992; Snow et al., 2004; Etzioni et al., 2005). Table 2 presents the full list of patterns we used to mine
anchors. Each pattern is converted into a prompt by filling in X and replacing Z with a [MASK] token, resulting in a set of co-hyponyms prompts C. We retrieve the 10 most likely filler words according to language model probability for each pattern Ci ∈ C. We score candidates z by their average probability across the patterns that contained z among the top 10 fillers:
sLM (z|x, C) = 1
|Cz|
|C|∑
i=1
PLM (z|x, Ci), (1)
where PLM (z|x, C) is the probability of z in the ith pattern instantiated with x and |Cz| is the number of patterns that predicted z. We finally keep the M highest scoring concepts as anchors, and instantiate M copies of LSP+A and DFP+A with the different anchors, respectively.
Hypernym Extraction Being able to construct sets of prompts for vanilla (PDFP,PLSP) and anchored prompts (PDFP+A ,PLSP+A), we are now in a position to prompt PLMs for hypernyms. Separately for each prompt set P ,4 we score hypernym candidates y by their average probability across patterns Pi ∈ P:
sLM (y|x,P) = 1
|P|
|P|∑
i=1
logPLM (y|x,Pi), (2)
where P = {PLSP,PDFP,PLSP+A ,PDFP+A}. The hypernyms ranked by sLM (y|x,P) and the top K are retained as hypernym candidates.

heading: 4 Experimental Setup
text: Datasets We conduct experiments on six English datasets. CLSB (Devereux et al., 2014) and DIAG (Ravichander et al., 2020) have been recently used
4We drop subscripts to avoid clutter.
to probe for hypernym knowledge in PLMs (Devlin et al., 2019). The remaining four data sets are widely-used test sets for hypernym extraction more generally (Shwartz et al., 2017; Roller et al., 2018), namely BLESS (Baroni and Lenci, 2011), EVAL (Santus et al., 2015), LEDS (Baroni et al., 2012), and SHWARTZ (Shwartz et al., 2017). We only consider NOUN-NOUN hyponym-hypernym pairs from the datasets. Dataset statistics are reported in Table 9 in the Appendix. Data sets vary widely in terms of their corpus size, the ratio of abstractness and concreteness, concept frequency and their construction methods, and hence underlying knowledge sources. While most data sets are based on WordNet, SHWARTZ builds on a wider set of resources, including ConceptNet and Wikipedia and hence includes more obscure concepts. EVAL stands out with a relatively high proportion of abstract concepts, unlike the other data sets which are predominantly concrete. Section 6 explores performance using these data conditions.
Model All our experiments are based on BERTlarge-uncased (Devlin et al., 2019) from Huggingface5 and use a zero-shot approach to probe the model. To allow for comparability of results across data sets, we adopt an open vocabulary approach throughout, considering the whole BERT vocabulary as hypernym candidates.6 We remove test instances where the hypernym is not in the BERT vocabulary.7 We set the number of anchors in anchored prompts to M = 5.8
Evaluation Metrics Following previous work (Petroni et al., 2019; Qin and Eisner, 2021), we retain the K=10 hypernym candidates and report Precision at 10 (P@10) as the extent to which correct hypernyms are included in the top 10 model predictions ranked by Equation 2. We also report mean reciprocal rank (MRR) of the true label. We evaluate model predictions at the concept level, normalizing predictions into their canonical form, i.e., accepting any inflection of the correct hypernym,9 and exclude punctuation, stop words, numbers and the hyponym x from the predictions.
5https://huggingface.co/ bert-large-uncased
6Prior work (Ravichander et al., 2020) adopted a closedvocabulary approach, limiting the set of candidate y to hypernyms in a particular data set.
7Note that there is no such restriction on hyponyms so that results in § 6.2 are not biased.
8This number was optimized on BLESS. 9We used pyinflect 0.5.1.
We measure the significance of differences with paired t-tests at p<0.05 after Holm-Bonferroni correction for multiple comparisons to adjust for comparisons across six data sets (Dror et al., 2017).
Analyses In addition to the main results, we aim to understand underlying factors that might affect the performance of hypernym extraction. We analyse the performance of pattern types on different types of concepts. We distinguish sets of hyponyms and hypernyms in terms of their frequency and abstractness and test consistency of predictions across prompt paraphrases.

heading: 5 Anchor Validation
text: How accurate are the automatically mined anchors? We qualitatively and quantitatively inspect retrieved anchor concepts. We use WordNet for this purpose, and follow Schick and Schütze (2020) to consider a candidate z to be a valid anchor of x if they share a common ancestor, within two levels above x and four levels above z. We exclude hyponyms that are not in WordNet in this analysis.
Table 3 reports the results across six datasets. For three of the data sets (BLESS, CLSB, LEDS), a correct anchor is predicted as top 1 result more than 33% of the time, and contained among the top 10 predictions we consider close to 70% of the time. The other data sets are overall challenging due to diversity and/or low frequency of concepts.
Qualitative inspection reveals that retrieved anchors that are not WordNet siblings according to our definition above are often reasonable, see Table 4. As we shall see in Section 6 the utility of anchors does not seem to hinge on them being actual co-hyponyms, and that the topically related anchors as produced by our method effectively improve hypernym extraction.

heading: 6 Hypernym Evaluation
text: We first examine the effectiveness of LSP vs DFP and the added value of anchoring on our six data sets overall (§ 6.1). Afterwards, we inspect specifically rare (§ 6.2) and abstract (§ 6.3) concepts as well as the well-known issue of inconsistency of responses in the face of prompt paraphrases (§ 6.4), explore different patterns in these contexts and end with an error analysis (§ 6.5).

heading: 6.1 Main Results
text: Table 5 presents the main results. Performance over datasets varies widely, with SCHWARTZ standing out with particularly low performance. SHWARTZ is dominated by proper noun hyponyms (e.g., city/person names), and includes a very broad range of hypernyms (1.1K). Performance on the other data sets are more comparable.
Do LSP and DFP differ? Comparing row one (DFP) and three (LSP) in Table 5, we see no consistent trend. While performance is often comparable, on BLESS LSP outperforms DFP. The reverse is true for EVAL. BLESS contains frequent and largely unambiguous hyponyms which are presumably more frequently discussed in natural patterns as comprised by LSP. EVAL is dominated by ambiguous and abstract concepts, which are perhaps more commonly described by formal, definitionstyle language.
Do anchors help retrieve more accurate hypernym knowledge? Table 5 reveals a consistent improvement of adding anchors for DFP (row 1 vs. 2) but not for LSP (row 3 vs. 4): definitional patterns benefit from anchoring via co-hyponyms
while lexico-syntactic patterns don’t.10
Next, in §6.2-§6.4 we disentangle the main results, considering a range of conditions which have been identified as challenging in prior work, and examine whether different patterns and/or anchoring can improve hypernym retrieval from BERT in these contexts.

heading: 6.2 The Impact of Frequency
text: Previous work (Ravichander et al., 2020; Hanna and Mareček, 2021; Schick and Schütze, 2020) found that BERT often fails to predict hypernyms for uncommon hyponyms. Here, we examine whether incorporating anchors can alleviate this issue. This is driven by the intuition that humans often draw on surrounding context signals to help understand the relationship between concepts. For example, even if we are unfamiliar with the concept
10We estimate the upper-bound of anchoring prompts with oracle anchors from WordNet, finding that better anchors can bring more benefits (see Table 10 in the Appendix).
of kea, knowing an anchor like parrot can help us infer that bird is one of the hypernyms. We expect that anchors can provide more linking context to the hypernym and improve the hypernym extraction performance when the hyponyms are rare. To verify this, we look into two aspects that reflect frequency: (a) existence in the BERT vocabulary - hyponyms that are included as single-tokens are frequent; (b) frequency in large corpora. We obtain term frequency from WorldLex (Gimenes and New, 2016) and categorize frequency into four levels based on absolute count: High (> 100), Medium (10-100), Low (1-10), and Unseen (0). For this analysis, we aggregate instances from all datasets to increase statistical power.
Figure 3 presents experimental results. We find that rare hyponyms have lower performance in general, aligning with previous work (Ravichander et al., 2020; Hanna and Mareček, 2021). More in-
terestingly, unlike in the main results, LSP exhibits a significant advantage over DFP on unseen and low frequency hyponyms (solid bars in UNSEEN and LOW blocks in Figure 3). Moreover, on the same blocks, we see that incorporating anchors into DFP significantly improves the performance on low frequent hyponyms (solid gray vs dashed gray). This confirms our hypothesis that anchors are beneficial for uncommon hyponyms by guiding BERT to predict hypernyms (see examples in Table 6). This is of practical relevance as it demonstrates that anchored prompts help for uncommon hyponyms, which can inform hypernym extraction in domain-specific or low-resources situations.

heading: 6.3 The Impact of Concreteness
text: Previous work on distributional semantics has shown that abstract words have higher contextual variability and are more difficult to predict than concrete concepts (Naumann et al., 2018). Here, we examine specifically whether the degree of concept abstractness affects hypernym extraction accuracy, as well as the impact of different patterns and anchoring in this context. To obtain the concept
concreteness level, we use the Brysbaert dataset (Brysbaert et al., 2014),11 which covers abstractness ratings for 40K common English concepts. Each concept was scored by at least 25 human annotators on a scale from 1 (most abstract) to 5 (most concrete). We use the median score to represent the abstractness of each word and bin them into Abstract (< 3) and Concrete (≥ 3). We inspect all four possible combinations of {concrete, abstract} × {hypernym, hyponym}, and again aggregate instances across data sets.
Figure 4 shows that hypernyms of hyponyms at same abstraction levels (e.g., Conc–Conc) are predicted with higher accuracy than those under different levels (e.g., Abs-Conc). This result is intuitive as words in same abstraction level tend to co-occur more (Bhaskar et al., 2017; Frassinelli et al., 2017). Overall, concrete hyponym-hypernym pairs are predicted with higher accuracy than pairs involving an abstract concept, indicating that abstract knowledge is more difficult to retrieve from BERT. More interestingly, we find that DFP+A brings remarkable improvements on abstract hypernyms, effectively reducing the gap between abstract and concrete hypernyms. A closer look at abstract hypernyms that failed with DFP but succeed on anchored prompts reveals failure on abstract hypernyms such as {emotion, organization, language, event}. For example, for the prompt excitement is a BERT predicts {thrill, fear, rush}. However, by incorporating anchors like surprise or anxiety, BERT predicts the
11We exclude hyponyms and hypernyms that are not in the Brysbaert dataset.
correct hypernym emotion. This finding is encouraging because it points to the weakness of using hyponyms alone to prompt PLMs for abstract hypernyms and can potentially inform future work on prompt design for retrieving specific types of knowledge (e.g., concrete or abstract) and building ontologies.

heading: 6.4 Consistency
text: Despite the success of prompting, a persistent challenge is an inconsistency of responses under slight rephrasing of the prompt (Elazar et al., 2021). In the context of hypernomy prediction, Ravichander et al. (2020) showed that compared to singular prompts (a car is a .), plural versions (cars are
.) returned different and worse results. We study consistency more systematically by including different paraphrases, and exploring the utility of anchoring on the robustness of results. We investigate: (a) consistency across prompts paraphrased with singular and plural hyponyms; and (b) consistency over prompts paraphrased with pattern type instantiations (cells in Table 1). We only score the prediction for a test instance as correct, if it was correctly predicted by all prompt paraphrases.
Pairwise Number Consistency Following Ravichander et al. (2020), we construct pairwise probes for singular and plural hyponyms, obtaining one representative pair for each of our four pattern types as listed in Table 7 (left). The results in Table 7 show that consistency strongly correlates with the choice of patterns: DFP prompts (row 1) produce inconsistent results, while LSP (row 3) shows strong potential for retrieving consistent knowledge. One reason is ambiguity in the plural DFP: the prompt Xs are [MASK] tends to return verbs and adjectives as candidates (e.g., carrots are {grown, eaten, orange}.), as plausible completions. In contrast, LSP contexts are more specific. Moreover, the consistency improves significantly for all but one data set when incorporating the anchors into LSP.12 This finding is important as it identifies a promising means of retrieving consistent knowledge from PLMs.
Group Consistency Our sets of pattern-type specific prompts suggest a natural, stricter consistency evaluation, namely to test whether BERT reliably predicts the same, true hypernym for all prompts
12Indeed, when comparing against the less strict evaluation in Table 5, LSP+A incurs the smallest performance drop.
associated with a pattern type (i.e, each of the cells of Table 1). Table 8 presents the results. What stands out in the table is that anchored prompts significantly improve group consistency, which aligns with our observation in the pairwise number consistency tests above. In summary, our results show that anchors, in particular LSP+A, can help retrieve more robust and consistent hypernyms from PLMs. This is not only important for downstream tasks which rely on (automatic) high-quality hypernym knowledge, such as taxonomy creation, but could also inform strategies to probe BERT for genuine, systematic knowledge, rather than superficial associations.

heading: 6.5 Error Analysis
text: When does anchoring hurt? Beyond benefits from anchors, we also observed that incorporating anchors at times degrades performance. Closer inspection identified sense ambiguity as a prevalent reason, especially for polysemous hyponyms, which have multiple hypernyms of different senses (e.g., fan is a person or an appliance.) With anchors, BERT predictions are skewed to a specific sense as selected by the anchor, which can be different from the true hypernym. Another situation is noisy anchors, including generic and irrelevant anchors (e.g., actually), or topically related anchors that are not co-hyponyms (e.g., wood and lake).
How do anchors improve consistency? We analyse hypernyms that are not consistently predicted correctly without anchors but are correct with anchors. There are three reasons for the inconsistency: (a) overly generic predictions from non-anchored patterns, e.g., Y, especially X often produces hypernyms like things or items; (b) predictions of co-hyponyms instead of hypernyms without anchors (e.g., a dog is a cat.), which is especially common with pattern A X is a Y, for which 30% of its predictions contain co-hyponyms from WordNet; (c) hypernyms in the intermediate levels of the WordNet taxonomy (e.g., garment, jewelry, sweet) are less consistent for patterns without anchors, e.g., anchors improve consistency by 11% for hypernyms whose minimum taxonomy depth is 7.13 This suggests that anchors can improve the consistency of mining new intermediate hypernyms from PLMs, aligning with prior work of using anchors to mine intermediate hypernyms from corpora (Hovy
13Table 12 in the Appendix lists the consistency of all depths.
et al., 2009).

heading: 7 Conclusion
text: In this work, we bridge two powerful techniques in hypernym extraction: the pattern-based and prompt-based approach and use them as a diagnose tool to probe knowledge in BERT. We provide a thorough study of how patterns from the corpus-mining literature can be used to probe neural models. We find that LSP and DFP exhibit similar capacities, while anchored patterns bring consistent and significant benefits, suggesting a way to overcome challenging scenarios. In particular, we demonstrated clear benefits for rare hyponyms and abstract hypernyms, and an increase in the reliability of retrieved hypernyms under paraphrased prompts. This finding can direct future work on prompt design to extract robust and consistent hypernyms knowledge. The idea of anchoring prompts can be extended to other semantic relations such as part-of and synonyms to advance taxonomy induction and knowledge graph construction.

heading: 8 Limitations
text: Effectiveness beyond noun-noun concepts: we apply our method to hyponym-hypernym pairs over nouns in the general domain. This idea of anchored prompts can also be extended mine hypernyms for other parts-of-speech using patterns developed for text corpora (Chklovski and Pantel, 2004; Kozareva, 2014), as well as semantic relations beyond hyponyms-hyponyms, e.g., Part-Whole (Girju et al., 2003). We leave this exploration for future work.
Time efficiency vs performance boost: incorporating anchors boost the performance for hypernym extraction, however, we also need to consider that the performance improvements comes with additional time cost. Querying with anchored prompts require more computation when multiple anchors are used, although runtimes for the experiments in the paper are all very low.
Hypernym diversity: current work on extracting hypernyms with BERT predominantly considers single-word hypernyms and does not consider multi-word hypernyms or hypernyms that are not in the BERT vocabulary. Our work is no exception.
Language diversity: Most work in both hypernymy retrieval as well as language model prompting focuses on English, and as a consequence there is a lack of data sets in other languages. The ex-
tension of technologies to less well-resourced languages is a pressing direction for future research.
Scale of Language Models We focus on comparing different pattern structures with a single model, BERT-large. The behaviours of patterns under larger language models such as GPT3 (Brown et al., 2020) remains to be examined (Wei et al., 2022).

heading: A Hypernym Evaluation
text: Dataset Statistics Table 9 presents the statistics on all datasets we used for experiments. We exclude hypernyms that are not included as single tokens in BERT vocabulary. The ratio of discarded (x, y) pairs is lower than 1% for most datasets, except for BLESS (30% is discarded) and CLSB (17% is discarded).
Comparison with oracle anchors To estimate the upper bound of anchored prompts, we treat siblings from WordNet (Miller, 1995) as oracle anchors and evaluate their effects on hypernym extraction. We select top five siblings with the highest rank of their path similarities calculated from WordNet, i.e., 1p(x,z)+1 , where p is the length of the shortest path between the x and z among their top two synsets. We use random sampling among siblings with the same score to select up to five anchors. The experimental results are presented in Table 10. We observe that using WordNet anchors can indeed lead to significant improvements in performance on datasets directly built from WordNet. For example, we observed large improvements for DIAG and LEDS when using WordNet anchors in combination with DFPA patterns. However, for other datasets, BERT anchors produce similar results as WorNet anchors. This highlights that with the improvement of anchor quality, anchoring prompts can unlock more hidden knowledge within BERT.
Computational Resources All experiments are conducted on single NVIDIA V100 GPU. A single run on each data set takes less than 2 hours, except for the large-scale dataset SHWARTZ, which takes nearly 24 hours on anchored prompts.

heading: B Consistency
text: B.1 Pairwise consistency on close vocab To compare our work with Ravichander et al. (2020) on pairwise probes using close vocab (nine hypernyms), we conduct the same experiments on DIAG dataset. Table 11 presents the results. The conclusion aligns with the open vocab set up: anchored patterns improve the consistency largely.
B.2 Group consistency over different depths of hypernyms
Table 12 reports the group consistency across different depths of hypernyms.

heading: Future Work
text: We leave this exploration for future work. Time efficiency vs performance boost: incorporating anchors boost the performance for hypernym extraction, however, we also need to consider that the performance improvements comes with additional time cost. Querying with anchored prompts require more computation when multiple anchors are used, although runtimes for the experiments in the paper are all very low. Hypernym diversity: current work on extracting hypernyms with BERT predominantly considers single-word hypernyms and does not consider multi-word hypernyms or hypernyms that are not in the BERT vocabulary. Our work is no exception. Language diversity: Most work in both hypernymy retrieval as well as language model prompting focuses on English, and as a consequence there is a lack of data sets in other languages. The ex-
tension of technologies to less well-resourced languages is a pressing direction for future research. Scale of Language Models We focus on comparing different pattern structures with a single model, BERT-large. The behaviours of patterns under larger language models such as GPT3 (Brown et al. , 2020) remains to be examined (Wei et al. , 2022)