Description Boosting for Zero-Shot Entity and Relation Classification (2024)

Abstract: Zero-shot entity and relation classification models leverage available external information of unseen classes e.g., textual descriptions to annotate input text data. Thanks to the minimum data requirement, Zero-Shot Learning (ZSL) methods have high value in practice, especially in applications where labeled data is scarce. Even though recent research in ZSL has demonstrated significant results, our analysis reveals that those methods are sensitive to provided textual descriptions of entities (or relations). Even a minor modification of descriptions can lead to a change in the decision boundary between entity classes. In this paper we formally define the problem of identifying effective descriptions for zero shot inference, we propose a strategy for generating variations of an initial description, a heuristic for ranking them and an ensemble method capable of boosting the predictions of zero-shot models through description enhancement. Empirical results on four different entity and relation classification datasets show that our proposed method outperform existing approaches and achieve new SOTA results on these datasets under the ZSL settings. The source code of the proposed solutions and the evaluation framework are opensourced. 1

heading: 1 Introduction
text: Named Entity Recognition (NER) and Relation Extraction (RE) allow for the extraction and categorization of structured data from unstructured text, which in turn enables not only more accurate entity recognition and relationship extraction, but also getting data from several unstructured sources, helping to build knowledge graphs and the semantic web. However, these methods usually rely on labeled data (usually human-annotated data) for a good performance, usually requiring domain experts for data acquisition and labeling, which may
1https://github.com/IBM/zshot
incur in high costs. Thus, it is not surprisingly that there is often a lack of labeled data for new domains, limiting the performance of these methods.
Zero-shot learning (ZSL) is a classification task in machine learning where - at inference time - samples are classified into one of several classes which were not observed during training. In ZSL, the sets of training and test entity classes are disjoint. Therefore, the strategy employed by zero-shot models is to rely on prior general knowledge that could be transferred to unseen instances at inference time. Having a classifier that can generalize to new unseen classes is important for a variety of practical reasons. First, ZSL methods can be used to learn models that are more robust to labeled data shortages and distributional shifts. Moreover, they can be used to extend the reach of models to new domains.
ZSL approaches in the Natural Language Processing (NLP) domain have seen significant improvements in recent years thanks to the availability of large pre-trained Language Models (LMs). For example, it has been shown that models such as GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022) and FLAN (DBL) achieve strong performances on many NLP tasks, including translation, question-answering, and cloze tests without any gradient updates or fine-tuning.
For entity recognition - including classification and linking - and relation classification problems, recent ZSL methods (Aly et al., 2021; Ledell Wu, 2020; Chen and Li, 2021) rely on textual descriptions of entities or relations. Descriptions provide the required information about the semantics of entities (or relations), which help the models to identify entity mentions in texts without observing them during training. Works such as (Ledell Wu, 2020; De Cao et al., 2021) and (Aly et al., 2021) show how effective it is to use textual descriptions to perform entity recognition tasks in the zero-shot context. However, the quality of the descriptions has
9441
an impact on how effective the transfer of knowledge from observed to unseen entities (Aly et al., 2021). The same mechanism can also be applied in other contexts, such as relation classification (Chen and Li, 2021). From now on, we refer to entity as both named entities and named relations.
An example of named entity classification with ZSL is demonstrated in Figure 1. At inference time, a zero-shot model is given short textual descriptions of new entity classes such as Company or Fruits, it then identifies and annotates mentions of those entity classes in an input sentence. Although state-of-the-art ZSL methods such as SMXM (Aly et al., 2021) have demonstrated significant results in recent research works, this toy example shows how the quality of the provided descriptions influences the accuracy of these models. For example, in Figure 1 even with a small modification of the Company entity class description, the SMXM model changes its entity prediction. In practice, the sensitivity to entity descriptions is problematic because, for non-expert users, it is not a trivial task to choose a proper description for black-box zero-shot models, in particular in an unfamiliar domain.
In this paper, we study different methods for improving the descriptions in an unsupervised way. Specifically, we propose UDEBO (for Unsupervised DEscription BOosting), the first unsupervised method capable of automatically modifying/generating descriptions to improve entity predictions in the zero-shot settings. We present several strategies to alter descriptions, such as using a generative model, paraphrasing, and summarization combined with description ranking/ensemble methods to reduce model uncertainty and increase overall performance. We empirically evaluate the performance of UDEBO on 4 existing standard zero-shot datasets, spanning two tasks: (i) named entity classification and (ii) relation classification.
Our results show that for the zero-shot entity
classification tasks, UDEBO improved the results of state-of-the-art models by 7 and 1.3 percentage points in terms of Macro F1 Score in the OntoNotes and MedMentions datasets, respectively. For what concerns relation classification, we achieve a performance improvement of 6 and 3 percentage points (Macro F1 Score) on the FewRel and WikiZS datasets over our baseline models, respectively.
We organize the paper as follows. In Section 2 we formally define the problem we aim to solve in this paper, i.e. how to enhance entity or relation descriptions to improve the performance of zero-shot models. In Section 3 we describe the proposed approaches for description boosting while in Section 4 we describe our experimental setup and results. We provide a literature review, a discussion about Large Language Models (LLMs) and draw the conclusions of our work in Sections 5, 6 and 7, respectively.

heading: 2 Problem Definition
text: Given a set of entity classes E of interest with their textual descriptions D and a corpus of sentences S to annotate as input, we can define the problem of description enhancement as follows:
Problem 1 (Description enhancement) Denote ϕ(D,S) as the function estimating the accuracy of ZSL models when using a given entity description D for annotating an input text corpus S. Our goal is to generate a set of descriptions D∗ such that:
D∗ = argmax D ϕ(D,S) (1)
In Section 3.1 we describe different strategies to generate new entity descriptions D′ for the input set E, intending to improve the accuracy of the predictions by the ZSL models over that corpus. If the labeled data is known, it is possible to select the
best descriptions via a brute force search across different description reformulations by measuring the accuracy as a function of D and S. However, given the absence of labeled data in the zero-shot context, an unsupervised approach is needed for ranking the descriptions D that yield the highest classification accuracy. In Section 3.2 and Section 3.3, we will discuss methods for ranking or combining predictions from different description variations to achieve better results.

heading: 3 Methods
text: The UDEBO approach comprises 2 steps. First, the descriptions are generated or improved. Finally, the descriptions are ranked in order to select the best ones. As an optional step, we analyze the ensembling of descriptions for boosting performance.

heading: 3.1 Generating description variations
text: Improving the completeness or clarity of entity descriptions is a complicated problem without a formal definition of an objective function, as there is a large space of candidates to explore. To enhance entity descriptions, in a more controlled way, we propose the following strategies.
Extension with pre-trained LMs. We propose to use large pre-trained LMs for generating text using the given description as context. Large LMs, as shown in (Petroni et al., 2019), capture linguistic and relational knowledge that can be extracted trough generation to extend a given description. In Section 4 we analyse the use of GPT-2 (Radford et al., 2019) for generating descriptions variations.
Extension with a fine-tuned LM. We fine-tune a LM for description generation and expansion. The LM is fine-tuned on a large dataset containing about 5.3 million Wikidata instances, including the name and the first few sentences of the respective articles. The model is fine-tuned on extending a truncated sub-string of the textual description, using a sequence to sequence objective. In Section 4 we analyse the use of a T5 large (Raffel et al., 2020) fine-tuned model for generating descriptions variations.
Summarization. Text summarization can be used to generate a concise description with less noise compared to the original one. In the experimental results we analyse the effect of using a BERT2BERT (small) (Turc et al., 2019) model fine-
tuned on CNN/Dailymail for text summarization to enhance entities descriptions.
Paraphrasing. Paraphrasing a description can simplify its linguistic form, using more common and general terms. In the experimental results we analyse the effect of using a Pegasus (Zhang et al., 2019) model fine-tuned for paraphrasing.

heading: 3.2 Description ranking via entropy
text: To rank a description for an entity, we propose to use a zero-shot model to first compute the probabilities of classes for each mention in the input text with a candidate description. We then compute the information entropy H from this input. In information theory, entropy is the average level of "information" or "uncertainty" inherent to a variable’s possible outcomes. Our assumption is that the lower the entropy is, the higher the confidence of the prediction will be, so Problem 1 can be reformulated as:
D∗ = argmin D H(D,S) (2)
Where H is the entropy of a zero-shot model for a corpus S, using the description D to accomplish a certain classification task. This way we can rank different candidate descriptions and choose the best one without requiring any labeled data, which is ideal for the zero-shot setting.

heading: 3.3 Boosting performances with descriptions variations ensembling
text: Besides description ranking via entropy, we propose an ensemble method that combines predictions from multiple pipelines executed with different entity descriptions. The main idea behind this approach is to leverage the complementary information provided by the different definitions to make a more accurate prediction, reducing the variance and bias of an individual pipeline. Furthermore, using the methods described in section 3.1, the descriptions variations can provide additional information useful for correctly discriminating between unseen classes.
Entity description ensemble. Given a sentence, for each span s and an entity label e ∈ E, denote v(s, e) as the number of pipelines that predict s or a sub-sequence of s with entity label e. For instance, given a span s = London Bridge, assume that among ten pipelines, four pipelines predict the label of s as e1 = Facility, the other four pipelines
predict the label of London as e2 = Location and the rest of the pipelines predict Bridge as Facility. Therefore, the accumulated number of votes for the span London Bridge are v(s, e1) = 6 and v(s, e2) = 4. Considering the majority of the votes, the final predicted label for the span London Bridge is Facility. Once the span London Bridge has been assigned a label, all of its sub-spans become redundant and thus are removed from consideration.

heading: 4 Experiments and Results
text: This section discusses experimental settings, baseline methods, and empirical results for both entity and relation classification tasks.

heading: 4.1 Datasets and experimental settings
text: We use two different settings: one for the Entity Classification (EC) task and one for the Relation Classification (RC) one.
Entity Classification setting. We use the pretrained SMXM model (Aly et al., 2021) with the checkpoints available in the official GitHub repository. 2 We refer the reader to the original paper (Aly et al., 2021) to see the details of the implementation, the training parameters, and the datasets used for fine-tuning the model. There are two different checkpoints, one for each one of the datasets used, OntoNotes (Pradhan et al., 2013) and MedMentions (Mohan and Li, 2019). Both datasets have been processed as in the respective official GitHub repositories. Table 1 shows the number of rows and the entities of each dataset. Note that the number of rows reported in Table 1 refers to the zero-shot version of the dataset, containing only sentences with entities. See Appendix A for more
2https://github.com/Raldir/ Zero-shot-NERC/
information on this process and the datasets. The results reported are all based on the test split of the datasets.
Relation Classification setting. For RC, we use ZS-BERT 3 (Chen and Li, 2021), a multitask learning model, based on BERT, to directly predict unseen relations. We trained our checkpoint using the official implementation of the model and following the steps of the official repository. 3 The datasets we use are FewRel (Han et al., 2018) and WikiZS (Sorokin and Gurevych, 2017). The results reported are all based on the test split of the datasets.
Description alteration settings. The language models used for the description alteration strategies: summarization, paraphrasing and pre-trained were obtained from the checkpoints available on Huggingface, while for the latter strategy we have fine-tuned a pre-trained T5-large model. We report detailed hyper-parameters of description alteration methods in section B of the appendix.

heading: 4.2 Empirical results
text: This section discusses the results of entity classification using methods for description enhancement.

heading: 4.2.1 Entity classification
text: Table 2 shows the results of the ensemble method (UDEBO) with ten descriptions generated by each of the description enhancing strategies, including pre-trained, finetuning, summarization and paraphrasing. For each enhancing strategy, we report the results when the descriptions with the lowest entropy are chosen for each class. The Combined strategy shows the results with the lowest entropy among all description-enhancing strategies.
We can see that the ensemble method (UDEBO) outperforms the SMXM baseline using the original descriptions provided on the OntoNotesZS dataset with a significant margin of 7 percentage points in terms of Macro F1 Score. On the MedMentionZS dataset, the improvement is 1.3 percentage points on the same reference performance measure (Macro F1 Score). Description ranking based on entropy works well with the pre-trained strategy on OntoNotesZS. However, the entropy does not seem to be a reliable score of model uncertainty on the MedMentionsZS dataset. Finding an alternative uncertainty score to entropy could be considered
3https://github.com/dinobby/ZS-BERT
as future work. Overall, these results confirm our hypothesis - discussed in Section 1 - that zero-shot methods are sensitive to provided descriptions and that an ensemble of description enhancement methods is needed to obtain more robust results.

heading: 4.2.2 Relation classification
text: In Table 3, we report our evaluation of the proposed approaches on the RC task. The results we observe here are similar to what we described for entity classification where the proposed ensembling method (UDEBO) achieves a higher performance across different measures compared to the baseline ZS-BERT model that does not rely on any relation description reformulation approach. We also observe on the FewRel dataset a higher Macro F1
Score associated with most of the description enhancement variants when employed independently from each other. These results further validate the strength of the proposed approach to enhance relation descriptions employed by ZSL models to improve their performance.

heading: 4.2.3 Descriptions enhancement strategies comparison and limitations
text: Generating variations of descriptions is relatively simple, as described in Subsection 3.1, several strategies allow to generate plausible extensions or variations of a text. Considering the results of ranking the descriptions using entropy in Section 4, we analyze and discuss here the correlation between Macro F1 Score and entropy measures and
the limitations of the proposed approach. Figure 2 and Figure 3 show the distributions of the Macro F1 Score on the test split of the OntoNotesZS and the MedmentionsZS dataset for each class, using the strategies described in Section 3.1 to generate 100 description variations for each class. None of the strategies is a clear champion over all the classes. The high variance of the performance explains the fact that the ensemble method makes a better prediction as observed in Table 2 and Table 3 thanks to successfully combining the strength of individual description alteration strategies. Figure 4 shows the correlations between Macro F1 Score and entropy for each unseen class on the OntoNotesZS test split with 100 description variations. Although there appears to be a significant statistical correlation using a sign test with (p-value = 0.03) between Macro F1 Score and entropy measures on the OntoNotesZS test set, the correlation does not appear to be statistically significant in the MedMentionsZS dataset. Also, as evidenced by the results in Table 2 and 3, using the descriptions with minimum entropy does not seem like a good strategy for selecting descriptions.
This phenomenon may be due to several factors like the change in the style of generated descriptions compared to the ones observed during training. Although a new description might seem more relevant, it could make the model more uncertain. See an example in Appendix C.2. The importance of this problem motivates the future study of alternative heuristics with more significant correlations,
indirectly unveiling the mechanism behind zeroshot predictions.

heading: 5 Related work
text: Zero-shot entity recognition and linking. Zeroshot end-to-end entity linking refers to the task of detecting and disambiguating entity mentions by linking them to an entity in a Knowledge Base (KB), without requiring new labeled data. KBs are inherently incomplete and evolve over time with the addition of new entities and relations. Zero-shot entity linking usually relies on available textual information, or other set of relations in the KB, to generalise to entity sets unseen in the training data.
BLINK (Wu et al., 2020) is a BERT-based solution for Zero-shot linking of textual mentions - extracted for example using FLAIR (Akbik et al., 2018) - to entities in Wikipedia. It follows a biencoder architecture, each mention is encoded in a dense space, together with its context (left and right part of the input sentence). Independently, each entity in the KB is encoded in the same dense space together with its context e.g., entity description. Mentions are linked to entities in the dense space using a nearest neighbour search. To improve accuracy, candidate entities are ranked by passing each concatenated mention, its context and entity description to a more expensive cross-encoder.
GENRE (De Cao et al., 2021) is a BART based model fine-tuned using a sequence to sequence objective, which claims to outperform BLINK. It is an autoregressive end-to-end entity linker, it detects
and retrieves mentions and the respective entities in a KB by generating their unique textual name - left to right, token-by-token. To do so, it uses a constrained decoding strategy that forces the generated name to be in a predefined candidate set. Compared to multi-class classification models such as BLINK, GENRE has a lower memory footprint to store dense vectors for large KBs, scaling linearly with vocabulary size, not entity count, and does not need to subsample negative data during training.
Zero-shot entity classification. Entity classification consists in predicting a probability for each semantic type of an entity mention, given a set of types (e.g, organisation, organic compound). The most straightforward feature used to generalise to unseen types is the textual descriptions. For example, SMXM (Aly et al., 2021) uses a cross-attention encoder to generate a vector representation for each type description and token in the input sentence and recognizes as entity types those representations that are closer to each other, including rarer classes unseen in training. It is evaluated using zero-shot adaptations of OntoNotes (Pradhan et al., 2013) and the domain specific biomedical dataset MedMentions (Mohan and Li, 2019), it also considers out-of-KB predictions i.e., nil predictions for mentions that do not have a valid gold entity.
ReFinED (Ayoola et al., 2022) is an end-to-end entity linking model optimised to perform mention detection, fine-grained entity typing (classification), and entity disambiguation in a single pass. Similar to BLINK, ReFinED uses a bi-encoder architecture modified to encode all mentions in a document simultaneously, which improves efficiency relatively to zero-shot models such as (Wu et al., 2020) that requires a forward-pass for each mention. Men-
tion embeddings and entity description embeddings are projected into a shared vector space to calculate their dot product as the entity score. A fast bi-encoder combined with a score for unseen entities, computed based on the scores for entity types and description, is enough for ReFinED to obtain state-of-the-art performance on entity linking and to scale the approach from Wikipedia (5.9M entities) to Wikidata (90M entities).
The analyses in (Aly et al., 2021) show that while Wikipedia descriptions work well on general entity types, they perform poorly on domain specific data, e.g. MedMentions. They also show the impact of using annotation guidelines for descriptions to improve the transfer of knowledge from observed to unseen entities. The adoption of this approach led to a better performance compared to using a class name itself or Wikipedia passages. In particular, description vagueness, noise and negations had a negative effect, while annotation guidelines, including explicit examples and syntactic and morphological cues, improved the performance.
Zero-shot relation classification. Textual descriptions have also been employed in the relation classification task to predict new relations that could not be observed at training time. For example, ZS-BERT (Chen and Li, 2021) learns two functions – one to project sentences and the other to project relation descriptions into an embedding space. The objective is first to jointly minimise the distance between the embedding vectors for an input sentence and the relation description for positive entity pairs and then to classify the relation (using a softmax layer to produce a classification probability). At inference time, the prediction of unseen relation classes can be achieved through
nearest neighbor search. Overall, using descriptions seems to improve existent zero-shot methods and expand their domains of application. Still, descriptions are not always good enough to get good predictions. Improving the accuracy of these approaches remains an open challenge. The better the separation between embedding of different relations, the more accurate the model predictions, however, as the number of unseen relations increases, it becomes more difficult to predict the right one (Chen and Li, 2021).
Existent ZSL methods usually rely on external knowledge from KGs, ranging from textual information, class attributes, hierarchy, domain and range constrains and relations to logic rules. There are relatively few studies evaluating their performance for unseen relations, a comparison using different external knowledge settings for zero-shot relation classification and KG completion can be seen in (Geng et al., 2021). To the best of our knowledge, we present the first approach to automatically predict and generate entity descriptions to improve the accuracy of entity recognition and relation classification models.

heading: 6 Discussion about Large Language Models
text: In the era of Large Language Models, all kinds of problems are being solved with LLMs, that achieve outstanding results in different tasks. However, LLMs also raise some concerns, being one of the most important, the green footprint of these models. Serving a single 175 billion LLM requires at least 350 GB GPU memory using specialized infrastructure, (Zheng et al., 2022). This makes it unfeasible for a lot of users to use LLMs, and even if it’s possible to use, there is a lot of concern about using them, specially for tasks that could be solved with smaller models. With UDEBO we try to push the research in a direction that improves the performance of small LMs to achieve results comparable to LLMs. However, to compare the performance of UDEBO against LLMs, we select 3 open-source LLMs available to the community and evaluate them. The results, discussion, and settings can be found in Appendix D.

heading: 7 Conclusion and future work
text: In this paper, we formally defined the problem of selecting descriptions to make predictions about unseen classes in the ZSL context. To the best of our
knowledge, this is the first time for entity/relation ZSL problems in which the impact of description variations on prediction performance is studied, and different methods for automatic creation of descriptions are considered. We empirically evaluated the sensitivity of two ZSL methods to description changes, and proposed 4 different strategies to enhance them using the implicit knowledge of pretrained language models. We also studied in detail the efficacy of the proposed entropy-based heuristic to rank different description formulations, analyzing its correlation with the performance (in terms of Macro F1 Score) of the model. We observed a negative correlation between the proposed heuristic and Macro F1 Score on two out of four of the considered datasets (OntoNotesZS and FewRel). The same assumption however was not valid for the other datasets (MedMentionsZS and WikiZS), thus motivating the need to develop more effective heuristics in the future. Finally, we described the UDEBO method, which combines the predictions obtained by the same model using different automatically generated variants of entity and relation descriptions. Our experimental results, on 4 different datasets, spanning across two different NLP tasks (Entity Classification and Relation Classification) showed how UDEBO outperforms the baselines by a significant margin and achieves new state-of-the-art results on these benchmarks under the zero-shot setting. Existing ensemble methods focus on ensembling different models trained on different data or models with different structures. Our work is orthogonal to these approaches, we proposed methods that consider entity/relation description variations as the hyper-parameters that need to vary. Most importantly, the description variations are not provided by the users but were generated from the initially provided descriptions. Therefore, this is a new way of creating ensembles, at least in the context of Zero-shot Entity/Relation extraction this is the first time a method for descriptions generation to diversify the pipelines and make an ensemble that improve the quality of the results is proposed.
Limitations
While our proposed method of boosting for zeroshot entity and relation classification shows promising results, there are several limitations that need to be acknowledged. Firstly, ensembling methods can be computationally intensive, which can limit their applicability to large-scale datasets. Our current implementation combining multiple model predictions, by varying the descriptions, which requires a significant amount of computational resources. Therefore, future research should explore alternative methods for ensembling that are more computationally efficient. Secondly, while we used entropy as a metric to identify helpful descriptions, it may not always be the most effective metric. Entropy measures the uncertainty or randomness of a distribution, but it may not necessarily capture the semantic relevance of a description. Therefore, there is a need for further research to develop better metrics or heuristics for identifying helpful descriptions. Finally, we have only experimented with a limited set of generation techniques. Future research should explore ways to improve the quality and coverage of the descriptions.
Supplemental Material Statement and Reproducibility
Source code availability the source code including the frameworks for pipeline ensemble and evaluation on the standard benchmark datasets are available as an open-source github repository: https://github.com/IBM/zshot.
Dataset availability Both datasets used in our evaluation are publicly available, also included in our open-source evaluation framework. We also release the enhanced descriptions generated by the generative models used to create the ensemble pipelines.

heading: 8 Appendix

heading: A Datasets
text: As mentioned in Section 4.1, we evaluate our approach on four different datasets, two for EC and two for RC. For EC, we use OntoNotes (Pradhan et al., 2013) and MedMentions (Mohan and Li, 2019). OntoNotes is a dataset that comprises various genres of text (news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, talk shows). We use the version available in Huggingface 4 and adapt it to perform zero-shot as explained in (Aly et al., 2021), removing all the entities that are out of the split - i.e., each split has a unique set of entities, so all the entities labeled with entities out of that set are removed - removing sentences without any entity labelled and using the same train/test/dev splits, so the pre-trained model has not seen the entities in the test set neither. The entity descriptions used for OntoNotesZS (the zeroshot version of OntoNotes) were provided by the authors of (Aly et al., 2021).
MedMentions is a corpus of Biomedical papers annotated with mentions of UMLS entities. We apply the same preprocessing steps we used for the MedMentions dataset, with the descriptions available in the official GitHub repository of (Aly et al., 2021). 2 The version of the MedMentionsZS dataset we use is also available on Huggingface. Both of them in their zero-shot version, as proposed in (Aly et al., 2021). To convert them to the zeroshot version, we follow the following steps:
1. Get the train/test/dev splits of the datasets;
2. Collect the entities in each split;
3. Remove entities out of the split i.e., if one entity e belongs to the train split, all mentions labelled as e in the test and dev splits will be replaced with the O label.
4https://huggingface.co/datasets/ conll2012_ontonotesv5
4. Remove sentences without labels. As the previous processing step (3) may remove all the entities of one sentence, the result dataset will have a lot of empty sentences. These sentences are removed in the final dataset.
Table 4 and Table 5 report the entities for each split in the dataset and the number of entities for MedMentionsZS and OntoNotesZS, respectively. As we can observe, both datasets are highly imbalanced, with some entities appearing 25007 times and some others only 89 in the case of MedMentionsZS, and 24163 and 65 times for OntoNotesZS. However, the most common entities are used only for training and the ones with fewer examples are used for validation and testing. As pointed in (Xian et al., 2019), real-world scenarios annotated data is likely to be available for the more common ones.
In Table 6 we report some statistics concerning the length of sentences on both MedMentionsZS and OntoNotesZS. In both datasets, there are sentences containing only 1 token and 1 entity. The maximum number of tokens also varies across
datasets and splits, with a maximum of 179 for MedMentionsZS and 210 for OntoNotesZS.
For RC, we use the FewRel(Han et al., 2018) and WikiZS (Sorokin and Gurevych, 2017) datasets. FewRel is a dataset for RC compiled by collecting entity-relation triplets with sentences from Wikipedia articles, and manually filtered to ensure the data quality and class balance. We use different relations for the train and the test split to ensure the zero-shot version of the dataset. The dataset is available in the Huggingface hub. 5 We use the train_wiki split in Huggingface as training split for the ZS-BERT model and the wiki_val as test split. Table 1 shows the total number of sentences in FewRel, and the number of different relations for each split. There are 700 samples for each relation in each split, thus the number of sentences reported in Table 1 is equal to the number of relations times the number of samples for each of them (e.g. train split: 44800 = 64 ∗ 700). Differently from FewRel, WikiZS was constructed using the Wikidata knowledge base. The dataset contains a total of 93431 sentences, each with an entity pair and a labelled relation between them. In this case, the number of instances per relation class is not balanced and we employ our own random splits containing different distinct sets of relations for the training (83 relations), validation (15 relations) and
5https://huggingface.co/datasets/few_ rel
testing (15 relations) of the ZS-BERT model. More information on the dataset is contained in Table 1.

heading: B Additional details on the models used for generating description variations
text: In this section, we report additional details on the methods used to generate description variations described in Section 3.1.
Extension with pre-trained LMs. An off-theshelf GPT-2 pre-trained model was used for generating the variations, using the checkpoint from the Huggingface Hub. 6 We used min_length = 80, max_length = 120, num_beams = 8, temperature = 1 and no_repeat_ngram_size = 2 for the generation.
Extension with a fine-tuned LM. A model based on T5 large (Raffel et al., 2020) and finetuned on the task of description generation and extension was used for generating the variations. As a starting point for the fine-tuning, the checkpoint from Huggingface Hub 7 was used. The Wikidata dataset, containing the name and the first few sentences of included Wikipedia articles where the model was fine-tuned on, was taken from Facebook Research’s BLINK project. 8 After cleaning the data i.e., removing instances with no or too short (less than 10 words) descriptions, about 5,310,000 samples were available for training the model to perform a new sequence to sequence task using learning_rate = 3e − 05 and epochs = 1. The objective was to complete the input description, starting from a sub-string containing the first ten words of it. For the generation task, just the name of the description was used. In the latter case, we set min_length = 80, max_length = 120, num_beams = 8, temperature = 1 and no_repeat_ngram_size = 2.
Summarization. A warm-started BERT2BERT (small) model fine-tuned on the CNN/Dailymail for document summarization was used for generating the descriptions variations, using the checkpoint from the Huggingface Hub. 9 We used min_length = 80, max_length = 512, num_beams = 8, temperature = 1 and
6https://huggingface.co/gpt2 7https://huggingface.co/t5-large 8http://dl.fbaipublicfiles.com/BLINK/
entity.jsonl 9https://huggingface.co/mrm8488/ bert-small2bert-small-finetuned-cnn_ daily_mail-summarization
no_repeat_ngram_size = 2 for this set of experiments.
Paraphrasing. A PEGASUS model fine-tuned for paraphrasing was used for generating the description variations, using the checkpoint from the Huggingface Hub. 10 We used min_length = 10, max_length = 60, num_beams = 8, temperature = 1 and no_repeat_ngram_size = 2 for the generation of text.

heading: C Additional experiments on Entropy and Macro F1 Score correlations
text: In this section we report additional insights on the correlation analysis discussed in the paper.
C.1 Correlations analysis of Macro F1 Score and entropy on MedmentionsZS
Figure 5 reports the correlations between Macro F1 Score and entropy on MedmentionsZS test-set. As discussed in the paper we did not observe any statistically significant correlations, with p-value = 0.50 .
C.2 Example of generated descriptions and entropy values
We analyze the variations for both entity and relation descriptions.
10https://huggingface.co/tuner007/ pegasus_paraphrase
Entity Tables 7, 8 and 9 show some examples of the description variations generated with the different methods. The original descriptions used are shown in Table 10. It can be seen that the models used for generation might generate wrong descriptions, e.g. Summarization variations in Table 7, in which the model, that has been fine-tuned over CNN/Dailymail, a news dataset, replicates the style of writing as if it was a reporter. Also, some hallucinations lead the models to generate URLs, that have been removed. This means that the base models used for generating the descriptions are crucial, and they should be chosen and fine-tuned carefully for the description generation task. We leave this as a task for future work.
Relations Given the relation Film Director described as:
"director(s) of film, TV-series, stageplay, video game or similar".
The fine-tuned approach for generating variations produces the alternative description:
The director(s) of a film, TV-series, stageplay, video game or similar is the person who directs the production of the film or television series. The term "director" is also used to describe an individual or group of people who are responsible for the creation, production, and/or
Enhancement
Description Entropy
directing of video games, films, television shows, or other forms of media..
Although the generated description seems more complete and containing relevant additional information, the entropy calculated with ZS-BERT is higher in this case than when using the original description. This means that the model is more uncertain of its prediction.

heading: D Discussion about Large Language Models
text: Recently, several large language models (LLMs) have been released demonstrating high capabilities for diverse NLP tasks including, but not limited to, text generation, question answering, text summarization and also NER and RE, (Workshop et al., 2023), (Chowdhery et al., 2022). In the zeroshot setting, the LLMs perform exceptionally well. However, there are some problems that might limit the usage of LLMs for NER and RE, like the need of prompt engineering and result parsing, the token
Enhancement
Description Entropy
limitation, the hallucionations and out of context generation or the efficiency.
• Prompt Engineering and result parsing. The performance of these models is highly dependent on the prompt used, and also the output of the model (Ding et al., 2021). Thus, depending on the use case, one prompt might or might not be adequate. Also, the output has to be processed to extract the actual entities, and depending on the prompt this process can be different.
• Token Limitation. These models are all based on text generation, having a minimum and a maximum number of tokens to generate. Depending on these hyperparameters, the result might not be complete or might lead to hallucinations or false positives.
• Hallucinations and out of context generation. LLMs often suffer from hallucination and out of context generation, which in the case of NER and RE might result into entities and relations extracted that are not present in the
text. Some approaches add a self-verification strategy to alleviate the hallucination issue, which requires further executions of the LLM (Wang et al., 2023). Moreover, common NER and RE use cases focus only on some specific entities or relations to be extracted, but these models can extract other entities and relations that may not be of interest to the user.
• Efficiency. Serving a single 175 billion LLM requires at least 350 GB GPU memory using specialized infrastructure, (Zheng et al., 2022). This makes it unfeasible for a lot of users to use LLMs, and even if it’s possible to use, there is a lot of concern of the green footprint of these models.
The efficiency problem is one of the most important problems of the LLMs, and thus some approaches try to reduce the size of the models or to train new models via knowledge distillation. Recent works approach the knowledge distillation process with human rationales to improve the performance of the distilled model (Hsieh et al., 2023). In this approach, the human rationale adds information to the
input, so the model can use it to perform the task. Similarly, the descriptions of the entities and relations are used to add information to the input. We leave to future work the usage of the descriptions for knowledge distillation. In either cases, UDEBO could be used to improve the descriptions or the human rationales to improve the performance of the models.
The focus of this work is the evaluation of the method UDEBO, and not the performance of the model itself, as a different size of the model, more pretraining, or even a different architecture could lead to changes in the results. However, we evaluate 3 LLMs, BLOOM (Workshop et al., 2023), FALCON (Almazrouei et al., 2023) and LLAMA 2 (Touvron et al., 2023). In Table 11 we evaluate the largest version of the models in zero-shot. In Table 12 we fine-tune a smaller version of the models (7B) using QLORA (Dettmers et al., 2023), with 0.06%, 0.03%, and 0.12% trained parameters for BLOOM, FALCON, LLAMA 2, respectively.
The fine-tuned version of the models benefit in the MedMentions dataset, which is specific, but they suffer in the generic domain (OntoNotes), as they extracted entities from the training set, and not the ones in the test set. We use the following prompt (example for OntoNotes):
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
From the input context below extract instances of the following labels: [‘LOCATION’, ‘WORK_OF_ART’, ‘BUILDING_NAME’]
### Input:

heading: Future Work
text: We leave to future work the usage of the descriptions for knowledge distillation. In either cases, UDEBO could be used to improve the descriptions or the human rationales to improve the performance of the models. The focus of this work is the evaluation of the method UDEBO, and not the performance of the model itself, as a different size of the model, more pretraining, or even a different architecture could lead to changes in the results. However, we evaluate 3 LLMs, BLOOM (Workshop et al. , 2023), FALCON (Almazrouei et al. , 2023) and LLAMA 2 (Touvron et al. , 2023). In Table 11 we evaluate the largest version of the models in zero-shot. In Table 12 we fine-tune a smaller version of the models (7B) using QLORA (Dettmers et al. , 2023), with 0. 06%, 0. 03%, and 0. 12% trained parameters for BLOOM, FALCON, LLAMA 2, respectively. The fine-tuned version of the models benefit in the MedMentions dataset, which is specific, but they suffer in the generic domain (OntoNotes), as they extracted entities from the training set, and not the ones in the test set. We use the following prompt (example for OntoNotes):
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction:
From the input context below extract instances of the following labels: [‘LOCATION’, ‘WORK_OF_ART’, ‘BUILDING_NAME’]
### Input: