TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks (2024)

Abstract: In this paper, we explore the capabilities of LLMs in capturing lexical-semantic knowledge from WordNet on the example of the LLaMA-2-7b model and test it on multiple lexical semantic tasks. As the outcome of our experiments, we present TaxoLLaMA, the “all-in-one” model for taxonomy-related tasks, lightweight due to 4-bit quantization and LoRA. TaxoLLaMA achieves 11 SOTA results, and 4 top-2 results out of 16 tasks on the Taxonomy Enrichment, Hypernym Discovery, Taxonomy Construction, and Lexical Entailment tasks. Moreover, it demonstrates a very strong zeroshot performance on Lexical Entailment and Taxonomy Construction with no fine-tuning. We also explore its hidden multilingual and domain adaptation capabilities with a little tuning or few-shot learning. All datasets, code, and pre-trained models are available online.1

heading: 1 Introduction
text: Recent studies in Natural Language Processing widely utilize Large Language Models (LLMs) for their capability to store extensive knowledge (Sun et al., 2023; Kauf et al., 2023; Tang et al., 2023) and to adapt quickly to different tasks via incontext learning without backpropagation (Dong et al., 2023). However, the application of LLMs to the classical lexical semantic tasks still remains understudied: for instance, no recent experiments with LLMs have been performed for the Hypernym Discovery task (Camacho-Collados et al., 2018) for different domains and languages. In Taxonomy Enrichment, LLMs are mostly used to extract vector representations which are further processed with a complex pipeline (Jiang et al., 2022).
Our work aims to investigate the capabilities of LLMs in addressing four tasks requiring taxonomic knowledge: Hypernym Discovery, Taxonomy En-
1https://github.com/VityaVitalich/ TaxoLLaMA
richment, Lexical Entailment, and Taxonomy Construction. We hypothesize that the model finetuned with hypernym (IS-A relationships) would be useful for solving taxonomy-related tasks. To verify this hypothesis, we develop a method inspired by (Moskvoretskii et al., 2024) to compile a taxonomyfocused instruction tuning dataset, sourced from English WordNet (Miller, 1998), to bring the implicit word knowledge of an LLM to the forefront when addressing lexical semantic tasks.
Having trained our model in this specialized setting, we are releasing the TaxoLLaMA — the finetuned version of the LLaMA-2-7b model (Touvron et al., 2023) — that is capable of solving tasks requiring taxonomic knowledge. Figure 1 presents the main idea of the model finetuning process. TaxoLLaMA operates effectively in a zero-shot setting, surpassing SOTA results in Lexical Entailment and Taxonomy Construction. With additional tuning, it also achieves SOTA performance in the Hypernym Discovery task across several languages and in half of the Taxonomy Enrichment tasks. Furthermore, we have optimized TaxoLLaMA to be
2331
lightweight through 4-bit quantization (Dettmers et al., 2023) and the application of LoRA (Hu et al., 2022), making it feasible to run on GPU devices with only 4.8Gb of GPU for forward pass and 5.5Gb for fine-tuning, ensuring its accessibility for widespread use, e.g. using Colab2.
The contributions of the paper are as follows:
• We introduce the use of LLMs across various lexical semantic tasks via hypernym prediction and propose an appropriate taxonomy instruction tuning method that exploits WordNet for dataset sampling.
• We present TaxoLLaMA – a unified model designed to address a spectrum of lexicalsematic tasks achieving state-of-the-art (SOTA) results in 11 out of 16 tasks and securing the second rank in 4 tasks.
• We present an instructive dataset based on English WordNet-3.0 only for training a taxonomy-based LLM and collected definitions for input words in the Taxonomy Enrichment datasets and the Lexical Entailment datasets using Wikidata3 and ChatGPT4.
• We perform a detailed error analysis for all tasks using both manual and automatic approaches: e.g. we evaluate error patterns and model quality using ChatGPT.

heading: 2 Related Work
text: In this section, we briefly describe previous approaches to the lexical semantics tasks that we are experimenting with in the paper.
Hypernym Discovery The Hypernym Discovery task involves predicting a list of hypernyms for a given hyponym (see example in Figure 2a). The recent study introduces a taxonomy-adapted, fine-tuned T5 model (Nikishina et al., 2023). Earlier models include the Recurrent Mapping Model (RMM) (Bai et al., 2021), which employs an Multilayer Perceptron (MLP) with residual connections and a contrastive-like loss. CRIM (BernierColborne and Barrière, 2018), distinguished as the best in SemEval, utilizes a similar MLP structure with a contrastive loss. The Hybrid model (Held
2https://colab.research.google.com 3http://wikidata.org 4https://chat.openai.com
and Habash, 2019) combines the k-Nearest Neighbor approach with Hearst patterns, while the 300- sparsans method (Berend et al., 2018) is an enhancement to the traditional word2vec approach.
Taxonomy Enrichment This task is addressed in SemEval-2016 Task 14 (Jurgens and Pilehvar, 2016), aiming to add a new word to the correct hypernym (node) in the given taxonomy. Numerous different architectures have been proposed to solve the task in recent years. TMN (Zhang et al., 2021) exploits multiple scorers to find 〈hypernym, hyponym〉 pairs for a given query concept. The TaxoEnrich (Jiang et al., 2022) employs two LSTMs (Staudemeyer and Morris, 2019) to encode ancestors and descendants information. In addition, the TaxoExpan (Shen et al., 2020) uses Graph Neural Network (GNN) (Scarselli et al., 2008) to predict whether the query is a child of an anchor concept.
Taxonomy Construction The taxonomy construction task aims to extract hypernym-hyponym relations between a given list of domain-specific terms and then construct a domain taxonomy based on them. The models for this task include Graph2Taxo (Shang et al., 2020), which employs a sophisticated GNN architecture, LMScorer. RestrictMLM (Jain and Espinosa Anke, 2022) uses zero-shot RoBERTa or GPT2 for pair relationship scoring, differing in their use of MASK or next token probabilities. TAXI+ (Aly et al., 2019) combines Hearst patterns with Poincaré embeddings for refinement of the existing approaches.
Lexical Entailment Lexical entailment is a classification task that identifies semantic relationships between phrase pairs. An example of the lexical entailment might be a hyponym “cat” which entails the existence of a hypernym “animal”.
One of the recent lexical entailment models is LEAR (Vulić and Mrkšić, 2018) a fine-tuning method of transforming Euclidean space so that it reflects hyponymy-hypernymy relations. In SeVeN (Espinosa-Anke and Schockaert, 2018) relations between words are encoded. Pair2Vec (Joshi et al., 2019) and variant of GloVe introduced in (Jameel et al., 2018) use words’ co-occurrence vectors and Pointwise Mutual Information. GBL (“Global” Entailment Graph) (Hosseini et al., 2018) is GNN that utilizes “local” learning and CTX (“Contextual” Entailment Graph) (Hosseini et al., 2021) is the improvement of GBL with contextual linkprediction. McKenna et al. (2023) proposes an
etailment smoothing technique to the CTX model resulting in SOTA for the task.

heading: 3 Methodology
text: This section outlines the process of data collection and the subsequent training of the model.

heading: 3.1 Data Collection
text: To create the dataset, we apply the algorithm presented by Moskvoretskii et al. (2024), focusing on hyponym-hypernym relationships only. We sample both nouns and verbs from the WordNet-3.0 graph. To prepare our training and validation sets, we randomly pick edges to form pairs of hyponymhypernym, the motivation for precise choice is given in Section E. If a child node links to more than one hypernym, we count each link as a separate pair. Additionally, we incorporate definitions for child nodes from WordNet to disambiguate the sense of the input word. As definitions may not be provided for some subtasks during inference (Lexical Entailment, MAG PSY, and MAG CS from Taxonomy Enrichment), we additionally generate definitions with ChatGPT for test sets that lack pre-defined explanations or take them from Wikidata. We use the web interface of ChatGPT 3.5 from February 2024 and the “gpt-3.5-turbo” model from the same period to generate definitions. The prompts for such requests and the statistics of the generated definitions are presented in the Appendix A in Examples 4-5 and in Table 7. This step is highly required: the lack of definitions can
reduce the performance of the model, as shown in Moskvoretskii et al. (2024).
Below we show a training sample from our dataset used for instruction tuning of TaxoLLaMA. It comprises a system prompt describing the desired output (1) combined with an input word selected from WordNet, along with its definition (2), and the target (3), which is the true hypernym of this input word, also sourced from WordNet:
(1) 
 «SYS» You are a helpful assistant. List all the possible words divided with a comma. Your answer should not include anything except the words divided by a comma «/SYS»
(2) hyponym: tiger (large feline of forests in most of Asia having a tawny coat with black stripes) | hypernyms: 

(3) big cat,
The statistics of the generated datasets are provided in the next Subsection 3.2 along with the setups they were created for.

heading: 3.2 Training Details
text: We introduce two versions of our model: TaxoLLaMA, the model trained on the full WordNet-3.0 dataset for further community usage in lexical semantic tasks, and TaxoLLaMA-bench, designed for the benchmark tests. For this model, we make sure that the training set does not include any nodes from the test sets of those four tasks. The size of the training set for the first model is 44, 772
items, whereas the other model was finetuned with 36, 775 samples. The TaxoLLaMA-Verb model that we experiment with in Section 5.4 is fine-tuned exclusively on the verb sub-tree from WordNet, resulting in 7, 712 samples. The finetuning procedure of our models is depicted in Figure 1.
To train in this setup, we use the LLaMA-2 model with 7 billion parameters (Touvron et al., 2023). For better computational efficiency during training and inference, we quantize the model to 4 bits and fine-tune it using QLoRA (Dettmers et al., 2023), a full-precision LoRA adapter. During pretraining, we used a batch size of 32 and a learning rate of 3e−4, applying a cosine annealing scheduler. Any further fine-tuning for different domains or languages was done with a batch size of 2 and a learning rate of 3e−4, without using schedulers. Other details are described in Appendix F.

heading: 3.3 Task Adaptation
text: We propose two methods for adapting LLMs, finetuned with the WordNet instructive dataset. Different tasks interpret a taxonomy node in a different way and their understanding is reflected in Figure 2. For instance, Taxonomy Enrichment operates with synsets or synset names while Hypernym Discovery operates with lemmas (Figure 2a). In Figure 2b, there is no difference between “TRUE” and “two-node connected”. However, they are depicted in such a way as to represent the expected outputs for two distinct tasks: Taxonomy Construction and Lexical Entailment. Taxonomy Construction focuses on creating a taxonomy graph from a list of nodes, essentially predicting the connections between them. On the other hand, lexical entailment involves determining whether a connection exists between two nodes. We benefit from this task likeness because we can train one model that would be able to solve multiple lexical semantic tasks. Despite this, it’s important to note that the tasks differ in how they interpret these relations.
We assume that Taxonomy-related tasks can be solved within two approaches from our pipeline.
Generative approach involves directly applying the same procedure as used in training. Given a hyponym, we use the model to generate a list of corresponding hypernyms. We apply this approach to the Hypernym Discovery and Taxonomy Enrichment datasets.
Ranking approach involves evaluating the hypernymy relation using perplexity: a lower score
indicates a stronger relationship. Beyond assessing this relationship, we can also evaluate the hyponymy relation by simply reversing the hypernym and hyponym positions (this way we obtain reverse perplexity). The ratio between these two scores is a measure of confidence that we use for ranking. The lower the Confidence score, the higher the confidence of the model in the hypernymy relationship between the two constituents of a pair.
We apply this approach for the Taxonomy Construction and Lexical Entailment datasets with slight modifications that will be described in the respective sections 4.3 and 4.4 in more detail.

heading: 4 Experiments
text: In this section, we assess the proposed methodology and the finetuned models, TaxoLLaMA and TaxoLLaMA-bench, on four lexical semantic tasks: Hypernym Discovery, Taxonomy Enrichment, Lexical Entailment, and Taxonomy Construction. We evaluate models in a zero-shot setting and after fine-tuning on the provided train sets for each task.

heading: 4.1 Hypernym Discovery
text: We test our model on the Hypernym Discovery task from SemEval-2018 (Camacho-Collados et al., 2018) using our generative approach. This task features an English test set for general hypernyms and for two domain-specific “Music” and “Medical” sets, and general test sets for Italian and Spanish. Performance is measured using the Mean Reciprocal Rank (MRR) metric. We test a zero-shot approach, where the model is not tuned to the training datasets. The test set differs from WordNet and may involve multiple hops to hypernyms, and can also be applied to narrow domains.

heading: 4.2 Taxonomy Enrichment
text: Taxonomy Enrichment aims to identify the most appropriate placement for a missing node within a taxonomy. Continuing the approach of prior works (Zhang et al., 2021; Jiang et al., 2022), the goal is framed as ranking nodes from the graph based on their likelihood of being the hypernym, where successfully placing the node means ranking its correct hypernyms at the top. In our setup, we use the generative approach described in Section 3.3 and depicted in Figure 2b.
The Taxonomy Enrichment benchmark encompasses the WordNet Noun, WordNet Verb, MAGPSY, and MAG-CS datasets (Jiang et al., 2022;
Shen et al., 2020). To ensure consistency, 1000 nodes from each dataset were sampled to match the test set from TaxoExpan (Shen et al., 2020). Following Jiang et al. (2022), we consider scaled MRR (Ying et al., 2018) as the main metric, which is the regular MRR multiplied by 10 and averaged over all of a node’s correct hypernyms.

heading: 4.3 Taxonomy Construction
text: This task aims to assemble a taxonomy given a list of nodes and a root. We employ datasets from TexEval-2 (Bordea et al., 2016) with “Eurovoc science”, “Eurovoc environment” and “WordNet food” subtasks and the F1 measure for evaluation.
We evaluate our model with the ranking approach applied to all node pairs. Using this principle, we iteratively established a threshold below which pairs are considered to have a relationship. The threshold for the “Food” domain was set to 1.8, for “Environment” to 4.6, and for “Science” to 1.89. To further refine the graph, we eliminate cycles by deleting the edge inside a cycle with the highest perplexity. Additionally, we limit each node to a maximum of three hypernyms. For nodes associated with more than three hypernyms, only three with the lowest perplexity scores are retained.

heading: 4.4 Lexical Entailment
text: Lexical Entailment aims at identifying semantic relationships between phrase pairs. Given a pair of words, the relation of entailment holds if there are some contexts in which one word can be substituted by the other, such that the meaning of the original word can be inferred from the new one.
We utilized the ANT entailment subset (Guillou and de Vroe, 2023) (a detailed enhancement of the Levy/Holt dataset (Holt, 2019)) and Hyperlex benchmark (Vulić et al., 2017) for our experiments.
ANT Dataset This dataset contains pairs of sentences differing in one argument in syntactic structure (for example: “The audience apploaded the comedian” and “The audience observed the comedian”, from Table 2 in (Guillou and de Vroe, 2023)). For these pairs, one of the relations is determined: antonymy, synonymy, directional entailment, or non-directional (which is reversed directional entailment) entailment. We treat the differing elements of the sentences as hypernym-hyponym pairs if the sentences are in one of the entailment relationships. To evaluate the entailment relations, we utilize the ratio of hypernym and hyponym ranking score, normalized via the L2 norm to represent the probability of entailment. For instance, we
calculate the perplexity for “move” as a hypernym of “walk” (PPLm→w) and vice versa (PPLw→m). The ratio PPLm→wPPLw→m of these scores will thus indicate the model’s confidence.
HyperLex Dataset This dataset focuses on the entailment for verbs and nouns, evaluating on a scale from 0 to 10. A score of 0 indicates no entailment, while 10 means strong entailment. The goal is to achieve the highest correlation with the goldstandard scores. For Hyperlex, we consider the ranking approach with no additional processing.
Previous methods generate embeddings and train a simple SVM on the Hyperlex training set. Finetuned models like RoBERTa demand substantial computational efforts and are tailored to the Hyperlex dataset. Compared to those prior studies, our zero-shot model uses perplexities directly as the predictions without a need for training. Therefore, a direct comparison might overlook the unique methodologies and resource implications, suggesting that each approach should be evaluated within its specific context.

heading: 5 Results
text: This section describes the main results of generative and ranking setup experiments for all tasks.

heading: 5.1 Hypernym Discovery
text: The results for the English language in Table 1, indicate that both the fine-tuned TaxoLLaMA and TaxoLLaMA-bench outperform previous SOTA results by a large margin. While the zero-shot performance of our models may be lower than when fine-tuned, they still deliver comparable outcomes to previous results for general English tasks and do not fall far behind in domain-specific tasks, considering that previous approaches are all fine-tuned.
Multilingual Performance For Italian and Spanish, the fine-tuned model surpasses previous SOTA results. We might assume the model’s effectiveness in a multilingual setting, knowing that LLaMA-2 is initially multilingual and that previous finetuning was performed exclusively on English pairs. However, we observe that the zero-shot performance struggles to generate accurate hypernyms for languages other than English. It is worth mentioning that both Italian and Spanish data were not included in the instruction tuning dataset.
Zero-shot Performance To investigate this zeroshot underperformance, we analyzed the effects of fine-tuning on both domains and languages, as shown in Figure 3a. It’s clear that, except for task 2B, the model exceeds previous SOTA results with just 50 samples for fine-tuning. Additionally, the fluctuating scores highlight the model’s sensitivity to the quality and nature of the training data.
Few-shot Performance We also explored the few-shot learning approach for the Italian and Spanish languages to assess the model adaptability in an in-context learning environment, as shown in Figure 3b. The model surpassed previous SOTA benchmarks with a near-logarithmic pattern of improvement for the Italian language with 30 and 50 shots, yet not performing as well for Spanish. We attribute the suboptimal few-shot to the 4-bit quantization and its relatively small size. Smaller models typically exhibit lower performance across various benchmarks compared to their larger counterparts, as illustrated by the example of LLaMA-2 (Touvron et al., 2023). Additionally, the capacity of smaller models or quantized models is also inferior compared to larger models, a finding corroborated by previous research (Wang et al., 2022; Frantar et al., 2023; Lin et al., 2024; Egiazarian
et al., 2024). The advantage gained from few-shot learning scenarios is less pronounced in quantized models compared to full-precision models. This observation has been specifically documented in the paper of Lin et al. (2024).

heading: 5.2 Taxonomy Enrichment
text: The results presented in Table 2 show that our model surpasses all previous methods on the WordNet Noun and WordNet Verb datasets but does not perform as well as the current SOTA method on the more specialized MAG-CS and MAG-PSY taxonomies even after fine-tuning. We also notice that TaxoLLaMA-bench, having less data, unexpectedly performed better on the MAG datasets. To delve deeper into the reasons behind overall underperformance, we conducted a comprehensive error analysis, detailed in Section 6.1.

heading: 5.3 Taxonomy Construction
text: The results in Table 3 demonstrate that applying our method directly leads to SOTA performance on the “Environment” and “Food” datasets, and secures a second-place ranking for the “Science” dataset. Further analysis of the graphs generated through our modeling is provided in Section 6.2.

heading: 5.4 Lexical Entailment
text: The results of TaxoLLaMA on the Lexical Entailment datasets surpassed our expectations.
Results on the ANT Dataset From the results on the ANT dataset in Table 4, we benchmark our models against prior SOTA performances. A notable finding is the obvious difference in performance between TaxoLLaMA, which is trained on both nouns and verbs, and TaxoLLaMA-verb, which focuses solely on verbs.
TaxoLLaMA-verb outperformes TaxoLLaMA in Lexical Entailment, suggesting difficulties in
processing nouns and verbs simultaneously that might impede verb learning, possibly due to quantization and LORA adapter tuning constraints. This issue seems specific to the entailment task, as it does not emerge in other tasks, such as Taxonomy Enrichment, which also includes a verb dataset. This discrepancy could stem from metrics requiring precise normalized perplexity ranking.
Table 4 shows that TaxoLLaMA-verb achieves SOTA performance on Average Precision and is second by normalized AUC. The comparison with previous SOTA results is skewed, as the bestperforming models benefited from the use of additional Entailment Smoothing (McKenna et al., 2023) on top of the model. This technique has yet to be applied to our models, which might be a promising direction for future enhancements.
Results on the HyperLex Dataset Table 5 demonstrates the superiority of our model over the previous SOTA in a zero-shot context for the “Lexical” subset and a second-place ranking for the “Random” subset. Contrary to the common trend where other models score higher on the random subset, our method does not follow this pattern, suggesting that the larger training size of the random subset benefits other methods more. Despite the straightforward zero-shot approach of our model, it still achieves notably high results. Future work could explore using this score as a meta-feature in task-specific models or adapting our entire model more closely to this task.

heading: 6 Error Analysis
text: In this section, we analyze the errors made by the TaxoLLaMA model, explore the reasons behind these inaccuracies, and suggest potential strategies for mitigation of LLMs applied to taxonomies.

heading: 6.1 Hypernym Discovery and Taxonomy Enrichment
text: As we apply the same generative approach for both Hypernym Discovery and Taxonomy Enrichment we perform the joint error analysis. We split the process into four steps: (i) manual analysis to identify the most common errors; (ii) automatic analysis of errors using ChatGPT; (iii) comparing and merging the most common errors identified; (iv) classification of the errors using ChatGPT.
First, we take about 200 random examples from both Hypernym Discovery and Taxonomy Enrichment datasets and write explanations of why the model fails to generate the correct hypernym. The following four classes are identified: (i) predicted hypernyms are too broad; (ii) incorrect/irrelevant definition generated by ChatGPT; (iii) the model was unable to generate relevant candidates in the same semantic field; (iv) miscellaneous cases.
We also use the prompt in Example 6 to ask ChatGPT to generate error types. The output is provided in Example 7; Table 8 summarizes all the error types generated during several runs. Then, we merge automatically and manually identified error types into the following classes:
1. Overly Broad Predictions: The model often generates predictions encompassing a broader concept than the true hypernym. 2. Underly Broad Predictions: Conversely, some predictions are too narrow and fail to capture the broader concept represented by the true hypernym. 3. Inaccurate Predictions: The model may predict words that are very semantically close to the true hypernym but struggles with fitting into the exact wording 4. Conceptual Ambiguity: The model may
struggle with ambiguous input words or concepts, leading to incorrect predictions. 5. Incorrect definitions: The model gets confused with the incorrect/inaccurate definition retrieved from external sources.
We used the prompt 8 presented in Appendix A to classify incorrectly predicted instances. The results for each task and each dataset are presented in Appendix B in Table 9 and in Figure 4 for average distribution. We also provide Table 10 with an example for each error type. The most common issue (75% of cases) is overly broad predicted concepts. It can be explained by the model adaptation to domain datasets that are richer than WordNet, like the “Music” and “Medical” domains. For Italian and Spanish, significant inaccuracies were attributed to grammatical complexities, due to the dataset limitations, linguistic intricacies, and lack of pretraining data. Similarly, MAG datasets faced issues with specificity and ambiguity, which led to lower results of TaxoLLaMA compared with Wordnetbased ones, as shown in Table 2.
Manual examination of MAG taxonomies reveals misaligned instances, like “olfactory toxicity in fish” being classified as a hyponym of “neuroscience”. Furthermore, we assess the accuracy of the predicted hypernyms using ChatGPT, inspired by contemporary research (Rafailov et al., 2023). We provide the inputs, predicted nodes, and ground truth nodes to ChatGPT, asking for preference. As depicted in Figure 5, ChatGPT mostly prefers neither of the answers and ground truth hypernyms only slightly more frequently than the predicted ones. The example of the input query is presented
in Appendix A in Example 9. We also evaluate the overlap between MAG datasets with WordNet data, discovering minimal correspondence. Only 5% of the nodes from the entire graph are present in the WordNet graph, with just 2% of edges for CS and 4% for PSY matching. For 92% of these, there is no path in the WordNet graph. Among the remaining connections, we see that 28% in CS and 10% in PSY represent cases where nodes are hypernyms of themselves. This also partially explains the low results of TaxoLLaMA, as MAG datasets greatly differ from the data used for TaxoLLaMA training.
Finally, we visualize the embeddings, revealing a notable discrepancy between predictions and ground truth in the MAG subsets—which was not seen with WordNet. ADetailed observations of this analysis are documented in Appendix C.

heading: 6.2 Taxonomy Construction
text: Our analysis of the predicted graphs across various domain datasets, based on statistics from Table 6, reveals consistent patterns. Generally, the gold standard graphs feature more edges, except in the environment domain. The model often omits entire clusters of nodes rather than individual ones: about 30% of nodes in the graph constructed with TaxoLLaMA lack a path to their actual parents, indicating they reside in separate components.
Nevertheless, existing paths are of a rather high quality, suggesting the model is performing either very accurately or completely off-target. The model assigns high perplexity to certain paths which are further incorrectly excluded. This tendency indicates a particular challenge with concepts that are neither too specific nor too general but fall in the middle of the taxonomy.
The nature of perplexity as a relative metric contributes to this issue, as some edges may not be created due to surpassing the perplexity threshold. Adjusting the threshold introduces incorrect edges, urging us to consider alternative approaches like using LLMs as embedders.

heading: 6.3 Lexical Entailment
text: Our examination of the ANT dataset showed it has nearly 3000 test samples but only 589 unique verbs. This means that errors on one verb could be replicated throughout the dataset. However, examining the overlap with WordNet revealed only 7 verbs were found in the same form. Lemmatization increases the count to 338, but about 42% of unique verbs still are not found in WordNet. No paths for the verbs presented in WordNet are found, which might have influenced model performance on the task. Hyperlex demonstrates better statistics, with nearly half of the words being unique and 88% found in WordNet. Only 27% of pairs are presented in the taxonomy, and 99% lack a connecting path.
Perplexity-related errors show high values for polysemous pairs (e.g., “spade is a type of card”) and low values for synonyms or paraphrases, indicating a semantic closeness but no hypernymy relation. This points to the model’s struggle with lexical diversity and ambiguity, emphasizing the need for disambiguation abilities in entailment tasks. Additional analysis is available in Appendix D.

heading: 7 Conclusion
text: In this paper, we introduce TaxoLLaMA— an LLM finetuned on WordNet-3.0, capable of solving various lexical semantic tasks via hypernym prediction. It achieved SOTA results in 11 out of 16 tasks and securing the second position in 4 tasks.
Manual and ChatGPT-based error analysis shows that the most errors (75%) are overly broad predicted concepts, due to overfitting to the idiosyncratic WordNet structure and inability to adapt to the target datasets. Experiments showed that, definitions greatly contribute to the final scores for Taxonomy Enrichment, similarly to (Moskvoretskii et al., 2024), as they help to better disambiguate input words. Regarding error analysis, the most difficult datasets were MAGs (Jiang et al., 2022), as they greatly differ from the data used for training of our model.
Limitations
We find that the main limitations of our work are as following:
• Dozens of large pre-trained generative models exist and we report results only on LLaMA2. An alternative base LLM used could further improve the results. However, our experiments showed that LLaMA-2 showed decent performance on hypernymy prediction compared to other models. Moreover, our goal was also to provide a lightweight model that could be of further research with limited resources. Finally, the research focused on the LLM application and not on an exhaustive search of all LLM models.
• We did not apply the “Ranking” approach to the Taxonomy Enrichment dataset, which would be also possible, as finding the most appropriate node for the input word could also be seen as ranking. However, the first experiments showed lower results.
• Possible “hypernymy hallucination” may also be considered as a limitation: apart from the generalization capabilities the model may overpredict types, or even invent new words or semantic categories.
• Another specificity of our model is its possible excessive focus on a single word sense, which may result in the inability to generate a wider variety of options.
• We tried to be exhaustive, yet we possibly did not cover some taxonomy-related tasks.
Ethical Statement
In our research, we employ advanced neural models like LLaMA-2, which have been pre-trained on a diverse corpus, including user-generated content. Although the creators of these models have endeavored to remove harmful or biased data, it is important to recognize that some biases may still persist in the model outputs.
This acknowledgment does not undermine the validity of our methods. We have designed our techniques to be flexible, allowing them to be applied to alternative pre-trained models that have undergone more rigorous debiasing processes. To the best of our knowledge, aside from the challenge of mitigating inherent biases, our work does not raise any additional ethical concerns.

heading: Acknowledgements
text: This work was supported by the DFG through the project “ACQuA: Answering Comparative Questions with Arguments” (grants BI 1544/7- 1 and HA 5851/2- 1) as part of the priority program “RATIO: Robust Argumentation Machines” (SPP 1999).
The work of Viktor Moskvoretskii was supported by Analytical center under the RF Government (subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021).

heading: A Using ChatGPT for Definition Generation and Automatic Error Analysis
text: Here below are two different example prompts 4 and 5 for ChatGPT for definition generation. The MAG PSY and MAG CS datasets for Taxonomy Enrichment, as well as ANT and HyperLex datasets for Lexical Entailment, do not possess definitions. Therefore, we developed several prompts specifically for the two types of datasets. For the hypernym prediction we want to have definitions for one input word, whereas for Lexical Entailment we expect to generate definitions for two words simultaneously, as they might be helpful for disambiguation. Table 7 represents statistics for the generated definitions for the datasets.
(4)
Write a definition for the word/phrase in one sentence.
Example: Word: caddle Definition: act as a caddie and carry clubs for a player
Word: eszopiclone 3 mg Definition:
(5)
Write a definition for Word 1 and Word 2. Each definition should be in one sentence. If a word is ambiguous, use the other word to disambiguate it.
Example: Word 1: depression Word 2: melancholy Definition 1: a mental state characterized by a pessimistic sense of inadequacy and a despondent lack of activity Definition 2: a constitutional tendency to be gloomy and depressed
Word 1: conflict Word 2: disagreement
Here is an example of the input to ChatGPT to automatically detect error types for hypernym prediction (Example 6) and the model output (Example 7). Furthermore, Example 8 demonstrates an example prompt for automatic classification.
(6)
Here is the data in four columns separated with "\t":
topic: psychology input word: social construction of schizophrenia
true: psychiatry, predictions: psychosis, social, psychological, and philosophical aspects, social psychology, psych, schizophrenia, antipsychotic, psychotic, antipsychotic
topic: psychology input word: use heroin true: methadone maintenance, predictions: heroin, opium tincture, heroin, opium, tincture, heroin, opium, and morphine, man, heroin, drug addiction
...
topic: psychology input word: coordinated management of meaning true: social psychology, predictions: communication disorders, language and literacy, and social studies, language and literacy, metaphor, metonymy, and politics, metonymy, metaphor
There is a topic, input word, correct hypernym (true), and model prediction (predicted). Your task is to compare true answers and model predictions and summarize what are the main error types of the model.
(7)

heading: To summarize the main error types of the model’s predictions compared to the true hypernyms:

heading: 1. Overly Broad Predictions: The model often generates predictions that encompass a broader concept than the true hypernym.
text: For example: "medication effects" was predicted with terms like "drug effects", "drug interactions", and "medication", which are broader than the true hypernym "clinical psychology" and "psychiatry."

heading: 2. Underly Broad Predictions: Conversely, some predictions are too narrow and fail to capture the broader concept represented by the true hypernym.
text: For example: "schizoid personality" was predicted with terms like "personality disorders" and "personality theory," which are narrower than the true hypernym "psychosis."

heading: 3. Inclusion of Unrelated Terms: In some cases, the model includes terms in its predictions that are not directly related to the input word or true hypernym.
text: For example: "pirisudanol" was predicted with terms like "dopamine receptor d1" and "dopamine receptor d2", which are unrelated to the true hypernym "psychiatry".

heading: 4. Repetition of Terms: The model occasionally repeats terms in its predictions, which might indicate redundancy or lack of diversity in its output.
text: For example: "pirisudanol" was predicted with repeated terms like "piracetam" and "hydergine."

heading: 5. Inadequate Coverage of Concepts: Some input words and true hypernyms receive predictions that lack comprehensive coverage of related concepts.
text: For example: "result category" was predicted with terms like "item response theory" and "qualitative research," which might not fully encapsulate the scope of the true hypernym "mathematics education."
(8)
You are a helpful linguist. Your task is to do error classification for a dataset for hypernym prediction in a general domain. Here are possible classes:

heading: 1. Overly Broad Predictions: The model often generates predictions encompassing a broader concept than the true hypernym.

heading: 2. Underly Broad Predictions: Conversely, some predictions are too narrow and fail to capture the broader concept represented by the true hypernym.

heading: 3. Inaccurate Predictions: The model may predict words that are very semantically close to the true hypernym, but struggles with fitting into the exact wording

heading: 4. Conceptual Ambiguity: The model may struggle with ambiguous (polysemantic/multivalued) input words or concepts, leading to incorrect predictions.

heading: 5. Incorrect definitions: The model gets confused with the incorrect/ inaccurate definition retrieved from external sources
text: You will be given an input word/phrase, true hypernym, and candidate hypernyms. Please, return a Python dict of error classes {1: 1, 2: 5, 3: 1, ..., 100:3}) for all instances below:
id: 1, input word: parathyroid_hormone, true hypernym: hormone, predicted: hormonal agent, hormon, hematopoietic growth factor, growth factor of the blood, growth regulator, growth substance, growth ... id: 100, input word: proofreader, true hypernym: printer, predicted: reader, audience, audience member, spectator, viewer, listener, listener-in, hearer, recipient, witness, watcher, observer
Here is the prompt Example 9 for ChatGPT in order to automatically evaluate TaxoLLaMA results, as manual analysis has shown that the gold true answers from MAG PSY and MAG CS datasets might not be of a good quality either. Therefore, ChatGPT was required to choose between the true answer from the dataset and the predicted candidate.
(9)
Here are the words in the psychological domain. Your task is to choose hypernym which is more relevant given two options. Answer 1 / 2 / both / none
Example: social construction of schizophrenia option 1: psychosis option 2: psychiatry Answer: 2
abdominal air sac option 1: air sac option 2: trachea Answer:

heading: B Error Type Analysis
text: This section represents the error types distribution across different datasets for hypernym prediction: Hypernym Discovery and Taxonomy Enrichment in Table 9. Moreover, in Table 10 we provide an example
for each error type that was classified by ChatGPT.

heading: C Distribution Visualization for Taxonomy Enrichment
text: In this section, we delve into the distribution patterns of ground truth and model predictions within the embedding space of the SentenceBert model (Reimers and Gurevych, 2019). To achieve this, we initiated two separate model runs, each with a distinct seed, aiming to capture the model’s variability. Subsequently, we extracted the predicted candidates and the ground truth hypernyms, mapping them into the embedding space provided by SentenceBert. To facilitate a clearer visual analysis, we condensed the embedding
dimensions to 50 using Principal Component Analysis and then applied t-SNE to project these dimensions onto two principal components for visualization.
The findings, illustrated in Figure 6, reveal a distinct pattern between WordNet and the MAG subsets (MAG_CS and MAG_PSY). WordNet displays a notable overlap between the gold standard and predictions, despite a few outliers that are presumably lower-ranked candidates. Conversely, the MAG subsets exhibit different behavior, forming two slightly overlapping clusters in the embedding space, suggesting a divergence between predictions and ground truths. Additionally, these subsets contain more outliers, indicating instances where the model may have completely missed the accurate hypernym sense. It’s important to consider, however, that the SentenceBert model’s representations could contribute to these discrepancies, especially for concepts that are not well-represented in its training data.
D Hyperlex Correlation Analysis
We also examine correlations using traditional methods for both test sets (refer to Figure 7). By overlaying the linear regression trend on the observed data points, a distinct trend emerges. However, this
trend is notably impacted by outliers, particularly within the Random set. This observation aligns with findings from taxonomy construction, highlighting the model’s challenges in accurately handling middle nodes or pairs exhibiting moderate entailment strength.
When analyzing gold scores ranging from 2 to 8, the Random set displays a lack of discernible trend, underscoring the model’s inconsistency in this area. The Lexical set shows a slightly better trend in that area. However, with both sets pairs characterized by strong entailment or minimal entailment are more accurately categorized. This distinction crucially enhances the overall correlation, which leads to a promising correlation score.

heading: E Hypernym motivation
text: Inspired by recent advancements in semantic analysis, particularly the work (Nikishina et al., 2023) on hyponym prediction, our study shifts focus towards hypernym prediction for several compelling reasons. First, predicting hypernyms is crucial for tasks such as taxonomy enrichment and hypernym discovery. Second, the formulation of a loss component for hypernym prediction is more straightforward, as most entities typically have a single correct hypernym, unlike hyponyms, where multiple valid options exist. This necessitates either adjustments to the loss function or extensive dataset collection and analysis.
Furthermore, our experimentation with various prompts revealed that the most effective format is detailed in the main section. Alternate prompts, which adopted a more narrative style (e.g., “Given a hyponym ’tiger’, the hypernym for it is"), led to the model generating paragraphs instead of concise hypernym lists. Adjustments to the system prompt failed to rectify this. Notably, appending a comma to the end of the target sequence remarkably improved the model output, encouraging it to list hypernyms instead of producing narrative text.
In addressing the disambiguation challenge, we experimented with incorporating definitions or technical identifiers from WordNet into the prompts. Definitions proved more effective, likely owing to the model pre-training on textual data. Attempts to generate hypernyms with specific WordNet codes resulted in the model appending the same numerical identifier to each hypernym which also resulted in lower scores.

heading: F Hyperparameter motivation
text: Our analysis revealed the model acute sensitivity to the learning rate and scheduler settings. The feasibility of employing a high learning rate in the primary study was contingent upon the use of the LORA adapter, which modulates weights without significant alterations. However, during full model fine-tuning, we faced instabilities, manifesting as either overfitting or underfitting—highlighting the necessity for further technical exploration into optimal hyperparameter configurations. Additionally, the implementation of 4-bit quantization requires careful learning rate selection, as this process notably compresses the weight distribution, demanding strategies to effectively recover the model knowledge thereafter.
In the fine-tuning process, we deliberately chose a smaller batch size to better accommodate the model to datasets, which are often limited in sample size. Contrary to our expectations, increasing the learning rate and batch size did not yield improved performance; this outcome can primarily be attributed to the reduced number of steps the model takes toward adapting to the specific domain. This strategy, however, did not apply to WordNet pre-training, where we observed differing trends.
Apart from certain instruction tuning methodologies, our approach does not involve calculating loss including on the instruction itself. Instead, loss calculation is confined solely to the target tokens.
The experiments utilized Nvidia A100 or Quadro RTX 8000 GPUs. Pre-training for TaxoLLaMA and TaxoLLaMA-bench spanned 6 GPU hours, while TaxoLLaMA-verb required less than 1 hour. Finetuning for MAG subsets took 5 GPU hours, attributed to the lengthy definitions. Fine-tuning for other datasets was completed in under an hour.

heading: Future Work
text: This technique has yet to be applied to our models, which might be a promising direction for future enhancements. Results on the HyperLex Dataset Table 5 demonstrates the superiority of our model over the previous SOTA in a zero-shot context for the “Lexical” subset and a second-place ranking for the “Random” subset. Contrary to the common trend where other models score higher on the random subset, our method does not follow this pattern, suggesting that the larger training size of the random subset benefits other methods more. Despite the straightforward zero-shot approach of our model, it still achieves notably high results. Future work could explore using this score as a meta-feature in task-specific models or adapting our entire model more closely to this task