Modelling Complex Semantics Relation with Contrastively Fine-Tuned Relational Encoders
Authors: Naïm Es-sebbani, Esteban Marquer, Zied Bouraoui
Accepted at: acl25

Modeling Complex Semantics Relations with Contrastively Fine-Tuned Relational Encoders

Naïm Es-sebbani1,2 and Esteban Marquer1 and Zied Bouraoui1
1 CRIL, Univ. Artois & CNRS, France      2 GRAMMATICA UR 4521, Univ. Artois, France
{essebbani,marquer,bouraoui}@cril.fr

Abstract
Modeling relationships between concepts and entities is essential for many applications. While Large Language Models (LLMs) capture relational and commonsense knowledge effectively, they are computationally expensive and often underperform in tasks requiring efficient relational encoding, such as relation induction, extraction, and information retrieval. Despite advancements in learning relational embeddings, existing methods often fail to capture nuanced representations and the rich semantics needed for high-quality embeddings. In this work, we propose different relational encoders designed to capture diverse relational aspects and semantic properties of entity pairs. Although several datasets exist for training such encoders, they often rely on structured knowledge bases or predefined schemas, which primarily encode simple and static relations. To overcome this limitation, we also introduce a novel dataset generation method leveraging LLMs to create a diverse spectrum of relationships. Our experiments demonstrate the effectiveness of our proposed encoders and the benefits of our generated dataset.

1 Introduction
Understanding semantic relationships between entities is fundamental to advanced natural language understanding and reasoning. For example, analogy-making requires precise comprehension of the relations between entities (Gentner and Markman, 1997; Turney, 2012; Kumar and Schockaert, 2023; Srikumar and Roth, 2013). The presence of distractors that share only surface-level similarities with a query relation \( A : A' \) has been shown to cause performance drops in automatic systems attempting to identify analogous relationships \( B : B' \) (Bitton et al., 2023).

Modeling relational knowledge is essential for tasks such as relation induction (Bouraoui et al., 2018; Sun et al., 2022a), relation extraction (Zhong and Chen, 2021; Zhou and Chen, 2022), and question answering (Yasunaga et al., 2021; Sun et al., 2022c; Jiang et al., 2022). Knowledge Graphs (KGs), which represent relations as triplets \((h, r, t)\) between a head entity \(h\) and a tail entity \(t\), offer a structured means of encoding relational information. Likewise, Large Language Models (LLMs) have shown strong capabilities in capturing commonsense and relational knowledge (Bouraoui et al., 2020). However, both KGs and LLMs face limitations. KGs are constrained by predefined schemas, which restrict the expressiveness and granularity of relations they can represent (Yasunaga et al., 2021; Sun et al., 2022c; Jiang et al., 2022; Wang et al., 2018). On the other hand, while LLMs are powerful, their high computational cost makes them impractical for large-scale applications involving extensive corpora such as news or scientific texts. In contrast, smaller language models such as BERT offer a more scalable and efficient alternative for relation extraction (Ushio et al., 2023). In this context, relational embeddings provide a compact and cost-effective means of encoding entity relationships without requiring explicit relation labels or rigid schemas. These embeddings leverage the internal knowledge of LMs to capture relational semantics between head and tail entities (Baldini Soares et al., 2019; Hao et al., 2023; Cohen et al., 2023). Recent approaches that employ relational embeddings have achieved strong performance, particularly in relation extraction (Zhong and Chen, 2021; Hao et al., 2023; Cohen et al., 2023; Jiang et al., 2022). Nevertheless, designing efficient and semantically rich relation embeddings remains an open challenge (Ushio et al., 2023).

Although current strategies have shown encouraging outcomes, they may still be less than ideal. First, most relational encoding models, to the best of our knowledge, are fine-tuned on triplets predominantly extracted from KGs. They inherit structural limitations inherent to KG schema. Specif-
ically, KGs typically encode simple relations, as complex relationships are conventionally decomposed into sequences of simpler ones. For instance, the relationship between "Paris" and "France" in KGs might only encode simple attributes like "capital of," omitting richer semantic contexts such as "tourist hub" or "cultural center." The reliance on a predefined and closed set of relations constrains the variety and expressiveness of relational contexts, reducing their ability to generalize beyond the training schema. Second, current models often fail to capture the full diversity of semantic properties and the complexity of inter-entity relationships. Their limited capacity for representing nuanced and multifaceted relations hinders performance in tasks that require fine-grained relational reasoning (Kumar and Schockaert, 2023).

To address these concerns, we first introduce a new dataset of over 80,000 triplets that leverages LLMs to capture a wide spectrum of relational knowledge. Our dataset generation process is flexible and can be extended to incorporate new categories of relations, ensuring adaptability to diverse tasks. When used to train relation encoders, this dataset consistently improves performance across existing models. In addition to the dataset, we propose three contrastive relational encoders designed to capture diverse aspects of relational semantics. Each encoder is fine-tuned using carefully crafted prompts, either to model relational semantics directly or to independently encode the head and tail entities before combining them into a compact relationship representation. To evaluate the effectiveness of these models, we focus on analogy questions and lexical relation classification tasks. The results show that our proposed relational encoders outperform state-of-the-art models in capturing and representing relational semantics.

2 Related Work

Learning Relational Encoders Since their introduction, pre-trained language models (LMs) have been extensively studied to evaluate their capacity for capturing commonsense knowledge (Forbes et al., 2019; Zhou et al., 2020; Roberts et al., 2020) and their potential for modeling relational knowledge (Petroni et al., 2019; Bouraoui et al., 2020; Sun et al., 2022b). To model relational knowledge, (Petroni et al., 2019) proposed a BERT-based model that utilizes manually defined prompt sentences, where the tail entity of the relation to predict is masked. Predictions for the masked token are used for link prediction, completing knowledge graph triples. Similarly, (Bosselut et al., 2019) introduced a fine-tuned GPT-based model for commonsense knowledge graph completion tasks, such as ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019). Several studies have since moved beyond hand-crafted templates, exploring automated methods for template generation. For instance, (Bouraoui et al., 2020) mined text corpora to identify effective templates for specific relations, using these templates with a BERT encoder to evaluate whether a word pair, instantiated within the template, forms a natural sentence. Other works, such as (Jiang et al., 2020; Haviv et al., 2021), developed paraphrasing-based strategies to enhance lexical diversity in templates. AutoPrompt (Shin et al., 2020; Liu et al., 2019) introduced a gradient-based search strategy to identify trigger tokens for creating prompts automatically. Additionally, (Jiang et al., 2020) combined mining-based, paraphrasing-based, and ensemble methods to generate diverse discrete textual prompts for relational prediction.

Despite recent progress, both manually crafted and discrete textual prompts have notable limitations. Hand-written prompts often fail to fully leverage the capabilities of LMs in encoding factual and relational knowledge. Approaches such as RelBERT (Ushio et al., 2023) aim to address this by encoding word pairs into relational embeddings using predefined or automatically generated prompts (e.g., “Paris is the [MASK] of France”) and fine-tuning LMs on datasets like SemEval 2012 Task 2. RelBERT employs loss functions such as triplet loss to optimize relational similarity and constructs relation embeddings by averaging contextualized LM outputs. However, this method may miss important semantic nuances of the relation between entities, due to inherent biases in the prompts and the limited richness of the training datasets.

Relational Reasoning with LLMs. LLMs face challenges in relational reasoning and information extraction, particularly due to difficulties in domain adaptation, implicit relation identification, and the need for specification-heavy instructions. Fine-tuning has proven essential to mitigate these limitations. Peng et al. (2023) highlighted that LLMs often struggle with tasks requiring detailed guidelines. Although LLMs are adept at capturing commonsense knowledge, this strength does
not always extend to tasks such as relation extraction, induction, or analogy reasoning, especially in domain-specific or highly constrained contexts. To address such issues, Engelbach et al. (2023) tackled hallucinations in extraction tasks using fine-tuned question-answering models, while Shi and Luo (2024) proposed methods to enhance logical consistency and precision. Despite their broad knowledge, LLMs still fall short in delivering high-quality relational embeddings, which remain critical for robust performance in these tasks (Bitton et al., 2023). Nonetheless, LLMs offer a valuable tool for generating diverse relational triplets that can compensate for the limitations of static, schema-bound datasets. By employing carefully designed prompts, we ensure our dataset encompasses a broad spectrum of relational types, resulting in a rich and nuanced resource for training relational encoders.

3 Deriving Relational Knowledge

Several datasets exist for training relational encoders, and we will revisit them in detail in Section 3.1. However, these datasets often have notable limitations, as they are predominantly derived from structured knowledge bases or predefined schemas that focus on encoding simple and static relationships. Consequently, they struggle to capture the complexity and richness of semantic relationships, particularly those grounded in real-world contexts or commonsense reasoning. To address these shortcomings, we propose a new approach in Section 3.2 that leverages the capabilities of LLMs to capture diverse commonsense knowledge.

3.1 Existing Datasets

Four primary sources are commonly used for training relational encoders, as outlined below. These datasets were meticulously curated in the work of (Ushio et al., 2023) and combined to train the RelBERT model1.

RelSim This dataset is derived from SemEval 2012 Task 2 and includes crowdsourced annotations on 79 fine-grained semantic relations organized into 10 major categories: Class Inclusion, Part-Whole, Similar, Contrast, Attribute, Non-Attribute, Case Relation, Cause-Purpose, Space-Time, and Representation.

ConceptNet This dataset originates from a filtered version of ConceptNet (Li et al., 2016) where certain triples with low confidence scores are excluded during the filtering process to ensure quality.

NELL-One NELL-One (Xiong et al., 2018) is a curated knowledge graph tailored for "one-shot" relational learning and is a cleaned version of the original NELL dataset (Mitchell et al., 2018).

T-REX Constructed from Wikipedia and Wiki-data, T-REX (Elsahar et al., 2018), the dataset is reduced to 839 distinct relations, creating a more focused resource for relational learning.

In the following, we will refer to these datasets as Relational Knowledge Bases (RelKB)

3.2 LLM Generated Data

We leverage LLMs as a source of common-sense knowledge to create a semantically diverse dataset for training relational encoders. Using Llama-3.1-8B-Instruct2, we generate triplets (head, relation, tail) that capture a wide range of semantic relationships, including causality, spatial relations. This allows to enrich and complement the existing datasets reported above, offering more varied examples of relations.

Relation categories and prompts To generate more detailed and accurate triplets, some of the RelSim relations were first refined or split into corresponding subcategories. For instance, we distinguished Spatial and Temporal relations from the original Space-Time category in RelSim. Similarly, Hypernym relations were complemented by Hyponymy to account for the LLMs sensitivity to the directionality of relations in prompts. Moreover, Meronymy and Synonymy relations were expanded into Member-Collection and Similarity, respectively, using more descriptive prompts that avoid reliance on the terms “meronym” and “synonym.” This adjustment aims to capture a broader range of related concepts, including triplets that do not strictly align with traditional definitions of Meronymy and Synonymy. Additionally, the Entailment relation was introduced to reflect a more logic-driven notion compared to Causal relations, enabling the generation of distinct triplets. We also added Commonsense, Functional, Collocation, and Troponymy (manner of an action) relations, which were not included in the original RelSim dataset.

1All datasets are available at https://huggingface.co/datasets/relbert/
2https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
Finally, we excluded Non-Attribute relations, as these describe the absence of attributes and may not be well-suited for an open-world setting relying on the commonsense knowledge encoded in the LLM. As illustration, we use such prompt for leveraging commonsense relation: “I am interested in knowing knowledge triplets that describe commonsense relation. For example : * glass, falls, breaks, * day, follows, night, * fish, live in, water, * wheel, part of, car, * knife, used for, cutting. Generate a bullet list of 100 different examples.” For other relation types, we use the same template, replacing the examples and “commonsense relation” by examples and a description fitting the kind of considered relation. The set of the relations we consider and corresponding few-shot examples for the prompts of the LLM generated data are provided Tab. 6 in the Appendix.

As shown in the example above, we used five carefully selected examples to guide the LLM’s generation process. This assumes that a small set of highly informative examples can significantly improve the diversity and accuracy of generated triplets. To further enhance diversity and avoid linear relational patterns common in sequence-based generative tasks, we formatted the output as a bullet list. This structure encouraged the LLM to treat each triplet as an independent instance, rather than as sequentially related to others. The small-scale generation strategy proved effective in maintaining the quality of the relational data. By limiting the number of triplets generated per prompt, we minimized the risk of reducing semantic precision.

SemRelLM Dataset. To construct our SemRelLM dataset, we iteratively queried the language model until we obtained at least 5,000 valid triplets for each targeted relation type. Statistics for the generated triplets across relation categories are provided in Tab. 1. We retained only successfully parsed triplets, without applying additional filtering. A qualitative analysis of the resulting data is presented in App. A.

<table>
  <tr>
    <th>Relation category</th>
    <th>Unique triplet tokens</th>
    <th>Unique triplet lemmas</th>
    <th>Unique relation tokens</th>
    <th>Unique relation lemmas</th>
  </tr>
  <tr>
    <td>Antonymy</td>
    <td>5003</td>
    <td>4861</td>
    <td>44</td>
    <td>44</td>
  </tr>
  <tr>
    <td>Attribute</td>
    <td>5015</td>
    <td>4885</td>
    <td>5</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Commonsense</td>
    <td>5019</td>
    <td>3902</td>
    <td>1303</td>
    <td>527</td>
  </tr>
  <tr>
    <td>Functional</td>
    <td>5026</td>
    <td>4604</td>
    <td>732</td>
    <td>661</td>
  </tr>
  <tr>
    <td>Collocation</td>
    <td>5029</td>
    <td>4779</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Troponymy</td>
    <td>5014</td>
    <td>4883</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Causal</td>
    <td>5017</td>
    <td>4358</td>
    <td>486</td>
    <td>317</td>
  </tr>
  <tr>
    <td>Entailment</td>
    <td>5006</td>
    <td>4178</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Spatial</td>
    <td>5000</td>
    <td>4892</td>
    <td>108</td>
    <td>106</td>
  </tr>
  <tr>
    <td>Temporal</td>
    <td>5003</td>
    <td>4762</td>
    <td>155</td>
    <td>133</td>
  </tr>
  <tr>
    <td>Hypernymy</td>
    <td>5019</td>
    <td>4971</td>
    <td>218</td>
    <td>205</td>
  </tr>
  <tr>
    <td>Hyponymy</td>
    <td>5001</td>
    <td>4935</td>
    <td>114</td>
    <td>98</td>
  </tr>
  <tr>
    <td>Meronymy</td>
    <td>5015</td>
    <td>4347</td>
    <td>17</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Member-collection</td>
    <td>5019</td>
    <td>4986</td>
    <td>12</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Synonymy</td>
    <td>5014</td>
    <td>4961</td>
    <td>626</td>
    <td>620</td>
  </tr>
  <tr>
    <td>Similarity</td>
    <td>5039</td>
    <td>4852</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <th>All</th>
    <th>80239</th>
    <th>74576</th>
    <th>3021</th>
    <th>1811</th>
  </tr>
</table>

Table 1: Number of unique triplets and relations in the LLM generated data. Tokens correspond to the output of the LLM, while lemmas are obtained using SpaCy to lemmatize and remove stop words. Relation categories that form complementary pairs are grouped together.

In the following, we propose three encoders aiming to capture rich semantics of relations, illustrated in App. Fig. 4.

4 Contrastive Relation Encoders

In this section, we describe our relational encoder models. We assume that we are given a set of triples \( \{(h_1, r_1, t_1), ..., (h_n, r_n, t_n)\} \) as training data where \( h_i = i \) and \( t_i \) are respectively head and tail entities and \( r_i \) the relation. The model needs to learn a relation embedding of each relation between a given pair \( (h, t) \) of head and tail entities.

\[
-\sum_r \sum_{a, a^+ \in Pos_r} \log \frac{\exp \left( \frac{\cos(f(a), f(a^+))}{\tau} \right)}{\sum_{a^- \in Neg_r} \exp \left( \frac{\cos(f(a), f(a^-))}{\tau} \right)}
\]

where \( f(\cdot) \) is the relation encoder, such that \( f(a) \) is the embedding of relation \( a \), \( Pos_r \) is the set of positive head-tail pairs \( \{(h_1, t_1), \ldots \} \) associated
with relation \( r \), meaning that \( a = (h_a, t_a) \) and \( a^+ = (h_{a^+}, t_{a^+}) \) share the relation \( r \). \( Neg_r \) is the set of negative head-tail pairs for relation \( r \), meaning there is no valid triplet where the head and tail are related by \( r \). \( \tau \) is a temperature parameter that controls the sharpness of the similarity distribution.

A common approach for training the encoder \( f(\cdot) \) is to use the prompt “The relationship between [HEAD] and [TAIL] is [MASK],” where the embedding of the [MASK] token serves as the relation representation. As shown in Tab. 10, the choice of prompt has a significant impact on relational encoding performance across multiple analogy datasets. These prompts are crafted to capture different aspects of relational meaning between [HEAD] and [TAIL] pairs, varying in specificity and contextual framing. Among them, the prompt “One property of [HEAD] is to be the [MASK] of [TAIL].” yielded the highest average performance. This suggests that embedding relational properties within a structured and interpretable context enhances the model’s ability to capture diverse characteristics of a relation.

However, the performance of a UniPRE is highly domain-dependent. For instance, prompts optimized for part-whole relations perform well on tasks involving compositional relationships but fail to generalize to other types of relations, such as temporal relations. This limitation arises because each prompt inherently biases the encoder toward a specific relational meaning, leaving other relational dimensions underrepresented or poorly encoded.

4.2 MultiPrompts Encoder

Another strategy we follow is to integrate multiple LM encoders, each using a distinct prompt tailored to capture specific relational domains, such as temporal relations. By aggregating embeddings from these diverse prompts, the MultiPRE model generates richer and more balanced relational representations, overcoming the domain-specific biases of UniPRE approaches. Let \( f_i \) denote the \( i \)-th LM encoder with its own prompt, and \( (h, t) \) represent a head-tail pair. To aggregate embeddings, we apply a learnable weighting mechanism, scaling each embedding by a weight \( w_i \). The final embedding, is obtained by concatenating the weighted embeddings, followed by normalization:

\[
\text{embedding}(h, t) =
\text{Concat}\left(w_1 f_1(h, t), \ldots, w_N f_N(h, t)\right)
\]

where \( N \) is the number of LM encoders. We apply L2 normalization on \( \text{embedding}(h, t) \). We use the same loss Eq. (1) where \( f \) is replaced by the final embedding from this model. The prompt combinations used are presented as prompt 5 and 6 in App. Tab. 10. This approach allows to capture a rich, multi-dimensional representation of relationships by leveraging the diverse relational perspectives offered by multiple encoders. By combining these embeddings through learnable weights, the model dynamically optimizes its understanding of semantic relationships.

4.3 Semantic Properties Encoder

While the multi-prompt encoder effectively captures diverse relational aspects, it primarily focuses on the relationship itself, abstracting away the intrinsic properties of the entities. For instance, it performs well on pairs like "coffee" and "black," where the relation emerges from their combination. However, in cases like "fast" and "food," understanding the relationship requires modeling the semantic attributes of each entity. We propose the Semantic Properties Encoder (SemPRE), a bi-encoder architecture with separate LM encoders for the HEAD and TAIL entities. Using prompts such as “[HEAD] means [MASK].” and “[TAIL] means [MASK].” as suggested in (Kteich et al., 2024), it extracts rich semantic embeddings that capture the core meanings of each entity. These embeddings are then combined using the Hadamard product:

\[
\text{embedding}(h, t) = f_{\text{HEAD}}(h) \odot f_{\text{TAIL}}(t)
\]

The model is also trained using an InfoNCE loss.

5 Experiments and Results

We evaluate our relation encoders through intrinsic tasks, including analogy questions (Sec. 5.2) and lexical relation classification (Sec. 5.3), to assess their ability to capture relational semantics and generalize to unseen relations. The experiments compare our models with SOTA baselines and analyze scalability across different BERT family models (e.g., BERT, RoBERTa, DeBERTa). Additionally, we also conduct experiments with LLMs, such as LLama3.3-70B, to compare performance with our encoders\(^3\). The performance of LLMs are discussed in App. C.

\(^3\)Our datasets and implementation are available at https://github.com/essebaninain/encoder-acl, and hyperparameters are detailed in App. D.2
<table>
  <tr>
    <th>Model</th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>Google</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>RelBERT-large</td>
    <td>67.0</td>
    <td>63.0</td>
    <td>80.0</td>
    <td><b>95.0</b></td>
    <td><b>27.0</b></td>
    <td>65.0</td>
    <td>63.0</td>
    <td>47.0</td>
    <td>63.37</td>
  </tr>
  <tr>
    <td colspan="10">RelKB(RelBERT finetuning data)</td>
  </tr>
  <tr>
    <td>UniPRE-large</td>
    <td>61.84</td>
    <td>60.65</td>
    <td>79.66</td>
    <td>92.8</td>
    <td>19.74</td>
    <td><b>77.17</b></td>
    <td>68.85</td>
    <td>47.9</td>
    <td>63.58</td>
  </tr>
  <tr>
    <td>MultiPRE-large</td>
    <td>61.84</td>
    <td>67.36</td>
    <td>82.32</td>
    <td>93.6</td>
    <td>20.3</td>
    <td>76.33</td>
    <td>70.49</td>
    <td><b>53.52</b></td>
    <td>65.72</td>
  </tr>
  <tr>
    <td>SemPRE-large</td>
    <td>31.58</td>
    <td>32.64</td>
    <td>49.58</td>
    <td>53.6</td>
    <td>11.51</td>
    <td>74.67</td>
    <td>56.28</td>
    <td>26.85</td>
    <td>42.09</td>
  </tr>
  <tr>
    <td colspan="10">SemRelILM(LLM-generated data)</td>
  </tr>
  <tr>
    <td>UniPRE-large</td>
    <td><b>73.68</b></td>
    <td><b>75.0</b></td>
    <td>79.1</td>
    <td>92.0</td>
    <td>20.36</td>
    <td>56.17</td>
    <td>69.95</td>
    <td>37.92</td>
    <td>63.02</td>
  </tr>
  <tr>
    <td>MultiPRE-large</td>
    <td>70.61</td>
    <td>74.31</td>
    <td>80.77</td>
    <td>92.2</td>
    <td>23.21</td>
    <td>62.67</td>
    <td>64.48</td>
    <td>37.16</td>
    <td>63.18</td>
  </tr>
  <tr>
    <td>SemPRE-large</td>
    <td>37.72</td>
    <td>40.51</td>
    <td>49.58</td>
    <td>54.8</td>
    <td>15.53</td>
    <td>54.0</td>
    <td>16.39</td>
    <td>23.32</td>
    <td>36.48</td>
  </tr>
  <tr>
    <td colspan="10">RelKB+SemRelILM</td>
  </tr>
  <tr>
    <td>UniPRE-large</td>
    <td>72.81</td>
    <td>74.54</td>
    <td>82.66</td>
    <td>93.6</td>
    <td>20.3</td>
    <td>73.83</td>
    <td><b>71.04</b></td>
    <td>40.69</td>
    <td><b>66.18</b></td>
  </tr>
  <tr>
    <td>MultiPRE-large</td>
    <td>71.05</td>
    <td>73.15</td>
    <td><b>83.77</b></td>
    <td>94.6</td>
    <td>25.56</td>
    <td>69.83</td>
    <td>65.57</td>
    <td>39.51</td>
    <td>65.38</td>
  </tr>
  <tr>
    <td>SemPRE-large</td>
    <td>38.6</td>
    <td>40.05</td>
    <td>57.53</td>
    <td>62.2</td>
    <td>16.96</td>
    <td>69.83</td>
    <td>38.25</td>
    <td>24.83</td>
    <td>43.53</td>
  </tr>
</table>

Fixed best results not being the ones in bold for some cases

Table 2: Accuracy (in %) on analogy questions. RelBERT results are from (Ushio et al., 2023).

5.1 Experimental Setup

Training data and details We utilized three datasets: (i) RelKB used for training RelBERT (Ushio et al., 2021) (ii) our newly generated dataset SemRelILM, and (iii) a combined dataset RelKB+SemRelILM. We fine-tuned three LMs encoder variants –BERT (Devlin et al., 2019), RoBERTa (Wolf et al., 2020), and DeBERTa (He et al., 2021)– in both small and large versions to evaluate the robustness of our encoders across architectures and sizes. In our main experiments, we report results using DeBERTa as it produces better results. Detailed comparisons across all models and configurations are provided in App. D.4 Tabs. 12 to 23 and App. D.5 Tabs. 25 to 36. We study in Sec. 5.4 the impact of the finetuning dataset.

Baseline models As a baseline, we use the RelBERT models from (Ushio et al., 2023). In particular, we consider two variants of the model, RelBERT-base and RelBERT-large, respectively with RoBERTa-base and RoBERTa-large (Wolf et al., 2020) as they present the best configurations.

While not entirely comparable as they do not provide relational embeddings, we also provide the performance of the latest (at the time of writing) publicly available LLMs, namely GPT-4o and Llama-3.3-70B-Instruct4. For GPT-4o, we consider only 100 samples for each subset of the benchmarks, due to cost limitations. As such, GPT-4o results should be interpreted with caution.

5.2 Analogy Questions

The first downstream task we consider is analogy questions dataset5. The task involves predicting which pair of entities, among a set of candidates, has the most similar relationship to a given query pair, framed as a multiple-choice problem. For instance, for the query pair strong:stronger, the candidate pairs are fortunate:fortunately, tight:young, tall:taller, newer:larger. In this example, the expected answer is tall:taller. In this setting, we encode both the query pair and the candidate pairs using our relational encoders. The candidate pair with the highest cosine similarity to the query pair is selected as the correct answer. This approach is unsupervised, relying solely on the quality of the relation embeddings.

Tab. 2 reports the results on analogy questions for large encoders (results for base encoders are in Tab. 11) for eight datasets (U2, U4, BATS, Google, SCAN, NELL, T-REX and ConceptNet), with accuracy summarized using an average score. RelBERT-large demonstrates strong performance across datasets, achieving an average accuracy of 63.37%, with particularly high scores on Google and NELL.

The proposed relational encoders UniPRE, MultiPRE, and SemPRE show varied performance de-

5all the datasets are available at: https://huggingface.co/datasets/relbert/analogy_questions
<table>
  <tr>
    <th>Model</th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>RelBERT-large</td>
    <td>81.25</td>
    <td>72.14</td>
    <td>53.45</td>
    <td>88.59</td>
    <td>75.67</td>
    <td>74.22</td>
  </tr>
  <tr>
    <td colspan="7">RelKB (RelBERT finetuning data)</td>
  </tr>
  <tr>
    <td>UniPRE-large</td>
    <td>88.14</td>
    <td>78.86</td>
    <td>67.24</td>
    <td>93.48</td>
    <td>83.67</td>
    <td>82.28</td>
  </tr>
  <tr>
    <td>MultiPRE-large</td>
    <td>90.54</td>
    <td>86.82</td>
    <td>64.94</td>
    <td>94.37</td>
    <td>83</td>
    <td>83.93</td>
  </tr>
  <tr>
    <td>SemPRE-large</td>
    <td>80.93</td>
    <td>69.4</td>
    <td>41.38</td>
    <td>92.74</td>
    <td>69.67</td>
    <td>70.82</td>
  </tr>
  <tr>
    <td colspan="7">SemReILM(LLM-generated data)</td>
  </tr>
  <tr>
    <td>UniPRE-large</td>
    <td>88.46</td>
    <td>78.86</td>
    <td>64.94</td>
    <td>93.41</td>
    <td>89.33</td>
    <td>83</td>
  </tr>
  <tr>
    <td>MultiPRE-large</td>
    <td><b>92.63</b></td>
    <td>85.07</td>
    <td>67.24</td>
    <td>94.37</td>
    <td>89.67</td>
    <td>85.8</td>
  </tr>
  <tr>
    <td>SemPRE-large</td>
    <td>79.33</td>
    <td>69.9</td>
    <td>47.13</td>
    <td>93.11</td>
    <td>78.0</td>
    <td>73.49</td>
  </tr>
  <tr>
    <td colspan="7">RelKB+SemReILM</td>
  </tr>
  <tr>
    <td>UniPRE-large</td>
    <td>88.94</td>
    <td>82.09</td>
    <td>63.79</td>
    <td>93.26</td>
    <td>89.0</td>
    <td>83.42</td>
  </tr>
  <tr>
    <td>MultiPRE-large</td>
    <td>91.67</td>
    <td><b>90.8</b></td>
    <td><b>67.82</b></td>
    <td><b>94.74</b></td>
    <td><b>91.0</b></td>
    <td><b>87.21</b></td>
  </tr>
  <tr>
    <td>SemPRE-large</td>
    <td>80.13</td>
    <td>72.14</td>
    <td>46.55</td>
    <td>93.26</td>
    <td>74.0</td>
    <td>73.22</td>
  </tr>
</table>

Table 3: Micro F1-score (in %) on lexical relation classification. RelBERT results are from (Ushio et al., 2023)

pending on the training data used. When trained on RelKB data, MultiPRE-large achieves the highest average accuracy (65.72%), excelling on datasets such as U4 and ConceptNet. UniPRE-large performs competitively with an average score of 63.58%, particularly strong in NELL and T-REX. However, SemPRE-large lags behind, with an average accuracy of 42.09%, due to its focus on entity-level semantic properties rather than relational nuances. Training on SemReILM data yields similar trends, with UniPRE-large and MultiPRE-large achieving comparable results (63.02% and 63.18%, respectively), while SemPRE-large underperforms again (36.48%). When combining RelKB and SemReILM data, UniPRE-large achieves the highest average score (66.18%), with notable improvements in T-REX and NELL, while MultiPRE-large performs slightly lower overall (65.38%).

These results demonstrate the effectiveness of the proposed models, particularly MultiPRE-large and UniPRE-large, in capturing diverse relational semantics. MultiPRE-large leverages multiple relational prompts effectively, while UniPRE-large performs exceptionally well when trained on combined datasets. SemPRE-large, designed to model entity-specific properties, struggles in comparison, highlighting its limitations for analogy questions tasks.

Overall, our model outperforms RelBERT-large, particularly excelling on challenging datasets like T-REX and NELL. For Google, we achieve near SOTA performance (ours: 94.6%; RelBERT: 95%), and on SCAN, the difference is less than 2%. SCAN is somewhat unique, as it focuses solely on two relation types: science and metaphor. The advantage of RelBERT on these datasets can likely be attributed to its better alignment with simpler relational structures.

5.3 Lexical Relation Classification

A second task we consider to evaluate our approach is the lexical relation classification task6. In this task, the goal is to classify word pairs into predefined relation categories. We trained a multi-layer perceptron (MLP) with one hidden layer of size 150 and a learning rate of 0.0001. The input to the MLP is the relation embedding of the word pair, while the relational encoder remains frozen during training to evaluate its performance independently. We also evaluate the performance of LLMs in both 0-shot and 5-shot settings. For an example of the prompt used, refer to App. Fig. 2.

Tab. 3 reports the results for large models (results for base model are in Tab. 24) in terms of F1-scores across five datasets (BLESS, CogALexV, EVALution, K&amp;H+N, and ROOT09). RelBERT-large achieves an average score of 74.22%, performing well on datasets like BLESS and K&amp;H+N but underperforming on EVALution highlighting its limitations in handling complex relational semantics.

The proposed encoders demonstrate significant

6The dataset is available at: https://huggingface.co/datasets/relbert/lexical_relation_classification
improvements, particularly MultiPRE-large, which achieves the highest scores across all training setups. When trained on ReKB data, MultiPRE-large achieves an average score of 83.93%, excelling in datasets such as CogALexV and K&H+N. UniPRE-large also performs well with an average of 82.28%, while SemPRE-large lags behind at 70.82%, struggling particularly with datasets like EVALution. Training on SemRelLM data further improves performance, with MultiPRE-large achieving an average score of 85.8% and notable gains on ROOT09 and BLESS. Combining ReKB and SemRelLM data yields the best results, with MultiPRE-large achieving a top average score of 87.21%, outperforming UniPRE-large (83.42%) and SemPRE-large (73.22%). MultiPRE-large excels across all datasets, particularly CogALexV, EVALution, and ROOT09, demonstrating its ability to leverage diverse relational prompts and high-quality training data.

Overall, MultiPRE-large consistently outperforms other models, achieving state-of-the-art results, particularly when trained on the combined ReKB and SemRelLM data. This highlights the effectiveness of leveraging diverse relational prompts and high-quality datasets for robust relational encoding.

5.4 Main Factors of Model Performance

Training datasets Considering the average performance for UniPRE and MultiPRE, fine-tuning with the SemRelLM dataset outperforms ReKB for lexical relation classification, particularly for complex tasks like ROOT09 and BLESS which require nuanced relational reasoning not fully captured by ReKB. For analogy questions, the performance of the two datasets is comparable. For both tasks, the best results are achieved by combining SemRelLM and ReKB, highlighting their complementary nature, as SemRelLM enriches relational diversity. Its larger size allows it to cover a broader spectrum, supporting better generalization. Additionally, SemRelLM addresses commonsense relations not fully represented in ReKB, which makes it an effective complement. This combination leads to significant performance improvements, up to +3% on analogy tasks and +4% on lexical relation classification.

Pre-trained model checkpoint Overall, for UniPRE and MultiPRE, fine-tuning DeBERTa outperforms RoBERTa and BERT for both base and large variants. However, for the SemPRE, BERT performs best, followed by DeBERTa and RoBERTa, likely due to BERT’s Wikipedia-based training, which better captures entity-dependent relational information. Detailed performance comparisons are provided in App. Tabs. 18 to 23 and 31 to 36 for UniPRE and MultiPRE, and Tabs. 12 to 17 and 25 to 30 for SemPRE.

LLM models As mentioned in Sec. 5.1, we performed experiments with LLaMA and GPT-4o reported in reported in App. Tabs. 8 and 9 and App. C for analogy questions and lexical relation classification respectively. The experimental settings for these LLMs are not aligned with those of our models or the ReKBERT baseline, making the results incomparable, and the results for GPT-4o should be taken with caution, as its evaluation was limited to only 100 examples due to cost constraints.

Overall, LLMs outperform our models on simpler analogy tasks (see App. Tab. 8) involving well-known relations like “capital of” or straightforward word pairs such as “simple” and “difficult.” However, their performance declines in more challenging settings like the NELL and CN datasets due to the “lost in the middle” phenomenon. In contrast, our models remain robust and consistent, benefiting from compact and focused relational embeddings. In lexical relation classification, LLMs demonstrate strong performance (see App. Tab. 9), highlighting their capacity to understand relational semantics. However, our encoders outperform them, likely due to the additional training of an MLP on top of the embeddings, which fine-tunes the models for task-specific nuances that LLMs may miss in zero-shot or few-shot settings.

6 Conclusion

This paper presents a novel approach to semantic relation encoding, leveraging diverse prompts and embedding fusion to enhance relational representations. Explicit, contextually rich prompts proved crucial for capturing nuanced relationships, while challenges such as factual accuracy and asymmetric relations highlight areas for improvement. Future work will explore automated prompt generation to refine relational encoding further. Moreover, LLMs do not naively produce embeddings, which are essential for applications like retrieval, classification, clustering, and other operations that require robust, semantically meaningful vector representations.
Limitations

Our analysis in this paper focused exclusively on the English language and the fully supervised setting for relational encoders. While we conducted experiments with LLMs, tasks such as analogy questions and lexical classification were limited to open-source LLMs due to the cost of using proprietary models like GPT-4o. For GPT-4o, we performed a limited experiment with only 100 samples to evaluate its capabilities. Our work also raises the question of whether dedicated relation embeddings are still necessary in the era of LLMs. Relation embeddings, however, offer distinct advantages, such as being more effective for modeling relational similarity and providing efficient, task-specific representations that LLMs may not inherently optimize for.

Acknowledgments

This work was supported by ANR-22-CE23-0002 ERIANA and was granted access to the HPC resources of IDRIS under the allocation 2025-A0171013338.

References

Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895–2905, Florence, Italy. Association for Computational Linguistics.

Yonatan Bitton, Ron Yosef, Eliyahu Strugo, Dafna Shafaf, Roy Schwartz, and Gabriel Stanovsky. 2023. VASr: Visual analogies of situation recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1):241–249.

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for automatic knowledge graph construction. In ACL, pages 4762–4779. Association for Computational Linguistics.

Zied Bouraoui, José Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7456–7463. AAAI Press.

Zied Bouraoui, Shoaiib Jameel, and Steven Schockaert. 2018. Relation induction in word embeddings revisited. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1627–1637, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge-base of language models. In EACL, pages 1856–1869. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.

Hady Elsahar, Pavlos Vougiouklis, Arsen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Matthias Engelbach, Dennis Klau, Felix Scheerer, Jens Drawehn, and Maximilien Kintz. 2023. Fine-tuning and aligning question answering models for complex information extraction tasks. arXiv preprint.

Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In CogSci, pages 1753–1759. cognitivesciencesociety.org.

Dedre Gentner and Arthur B. Markman. 1997. Structure mapping in analogy and similarity. American Psychologist, 52:45–56.

Shibo Hao, Bowen Tan, Kaiwen Tang, Bin Ni, Xiyan Shao, Hengzhe Zhang, Eric Xing, and Zhiting Hu. 2023. BertNet: Harvesting knowledge graphs with arbitrary relations from pretrained language models. In ACL, pages 5000–5015. Association for Computational Linguistics.

Adi Haviv, Jonathan Berant, and Amir Globerson. 2021. Bertese: Learning to speak to BERT. In EACL, pages 3618–3623. Association for Computational Linguistics.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.

Jinhao Jiang, Kun Zhou, Ji-Rong Wen, and Xin Zhao. 2022. great truths are always simple : a rather simple knowledge encoder for enhancing the commonsense reasoning capacity of pre-trained models. In NAACL, pages 1730–1741. Association for Computational Linguistics.
Pengcheng Jiang, Jiacheng Lin, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2024. Genres: Rethinking evaluation for generative relation extraction in the era of large language models. In NAACL, pages 2820–2837.

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423–438.

Hanane Kteich, Na Li, Usashi Chatterjee, Zied Bouraoui, and Steven Schockaert. 2024. Modelling commonsense commonalities with multi-facet concept embeddings. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 1467–1480. Association for Computational Linguistics.

Nitesh Kumar and Steven Schockaert. 2023. Solving hard analogy questions with relation embedding chains. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6224–6236. Association for Computational Linguistics.

Na Li, Hanane Kteich, Zied Bouraoui, and Steven Schockaert. 2023. Distilling semantic concept embeddings from contrastively fine-tuned language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 216–226. ACM.

Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1445–1455, Berlin, Germany. Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.

T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, B. Yang, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2018. Never-ending learning. Commun. ACM, 61(5):103–115.

Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou, and Juanzi Li. 2023. When does in-context learning fall short and why? A study on specification-heavy tasks. CoRR, abs/2311.08993.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In EMNLP, pages 2463–2473. Association for Computational Linguistics.

Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In EMNLP, pages 5418–5426. Association for Computational Linguistics.

Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: an atlas of machine commonsense if-then reasoning. In AAAI-JAAI-EAAI, pages 3027–3035. AAAI Press.

Zhengpeng Shi and Haoran Luo. 2024. Cre-lm: A domain-specific chinese relation extraction framework with fine-tuned large language model. arXiv preprint.

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, pages 4222–4235. Association for Computational Linguistics.

Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, pages 4444–4451. AAAI Press.

Vivek Srikumar and Dan Roth. 2013. Modeling semantic relations expressed by prepositions. Transactions of the Association for Computational Linguistics, 1:231–242.

Lu Sun, Yongliang Shen, and Weiming Lu. 2022a. Minimally-supervised relation induction from pre-trained language model. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1776–1786, Seattle, United States. Association for Computational Linguistics.

Lu Sun, Yongliang Shen, and Weiming Lu. 2022b. Minimally-supervised relation induction from pre-trained language model. In NAACL, pages 1776–1786. Association for Computational Linguistics.

Yueqing Sun, Qi Shi, Le Qi, and Yu Zhang. 2022c. JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answering. In NAACL, pages 5049–5060. Association for Computational Linguistics.

Peter D. Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. J. Artif. Intell. Res., 44:533–585.
Asahi Ushio, Jose Camacho-Collados, and Steven Schockaert. 2021. Distilling relation embeddings from pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9044–9062, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Asahi Ushio, Jose Camacho-Collados, and Steven Schockaert. 2023. Relbert: Embedding relations with language models. Preprint, arXiv:2310.00299.

Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. Dkn: Deep knowledge-aware network for news recommendation. In WWWC, WWW ’18, page 1835–1844. International World Wide Web Conferences Steering Committee.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, and Clement Delangue et al. 2020. Transformers: State-of-the-art natural language processing. In EMNLP, pages 38–45. Association for Computational Linguistics.

Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2018. One-shot relational learning for knowledge graphs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1980–1990, Brussels, Belgium. Association for Computational Linguistics.

Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In NAACL, pages 535–546. Association for Computational Linguistics.

Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50–61, Online. Association for Computational Linguistics.

Wenxuan Zhou and Muhao Chen. 2022. An improved baseline for sentence-level relation extraction. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2022 - Volume 2: Short Papers, Online only, November 20-23, 2022, pages 161–168. Association for Computational Linguistics.

Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pretrained language models. In AAAI-IAAI-EAAI, pages 9733–9740. AAAI Press.
A  Analysis of the quality of LLM generated data

A.1 Qualitative analysis

General observations  A qualitative analysis of the triplets generated using the prompts and examples in Appendix Tabs. 6 and 7 reveals that triplets that do not follow the expected (head, relation, tail) structure are present. Examples of such triplets include (fast, rapid, fleet) for Synonymy, as it is a sequence of 3 entities with no relation, and (shuffle, slide, is a way to) for Troponymy, as the relation and the tail have been swapped. Other triplets that do not follow the expected structure of a triplet have been discarded in the LLM output parsing step. Finally, we identified the presence of triplets generated with inconsistent separation between head, relation, and tail (e.g.: Hyponymy triplet (ferrari is a brand of, high, performance car) instead of (ferrari, is a brand of, high performance car)).

Aside from incorrect triplets, the LLM generates triplets that are meaningful and varied in terms of the entities involved and the relations used. As can be seen in Tab. 1, after lemmatizing using SpaCy7 and excluding stop words, the relation and triplet count does not drop by a significant amount. This is an illustration of the rich lexicon of relations generated for most relations categories, with 90% of Commonsense relations used in at most 6 triplets. For Attribute, Collection, Entailment, Meronymy, Member-collection, Similarity, and Troponymy, where the number of used relation lemmas is small compared to the other relation categories, the LLM adheres closely to the relations provided in the few-shot examples.

Complementary relation categories introduced
Compared to (Ushio et al., 2023) we consider 4 new categories of relations and separate 5 of the original categories into pairs of complementary categories. These new categories enable a wider variety of relation to be generated and account for the dependency of the LLM to prompt formulation and the few-shot examples chosen.

Temporal and Spatial relations contain both general temporal (e.g.: gives way to, before, after, etc.) and spatial (e.g.: in, on, beside, above, etc.) relations, and relations that are not exclusively temporal or spatial but contain a spatio-temporal aspect (e.g.: (plant, grows, flower), (earth, orbits, sun)). some of the relations can be seen as factual (e.g.: (earth, orbits, sun)) while other express likely relations (e.g.: (cat, in, basket)). The latter is particularly interesting as such triplets are not usually found in KGs, and correspond to commonsense knowledge extracted from the LLM used. Similarly, triplet using the Causal prompt cover explicit causal relations (e.g.: leads to, regulates, prevents, etc.) and relations with a causal aspect (e.g.: destroys, that means to cause the destruction), as well as strict (e.g.: destroys, produce, etc.) or non-strict (e.g.: support, promotes, etc.) causes. While not expected when designing the prompt, the prompt for Functional relations also resulted in relations with a causal aspect (e.g.: regulates, pollinates, filters, etc.) that correspond to a global and high-level view of what the function of an entity can be.

Some of the added categories provide more specific or higher quality triplets. For instance, the prompt for Entailment expands the natural causes described in Causal relations with logical (e.g.: (to be married, entails, having a wedding)) and commonsense (e.g.: (to sleep, entails, to be tired)) entailment, including definitions (e.g.: (to be helpful, entails, assist others)) that can be seen as bidirectional entailment. The prompt for Synonymy results in about 2 out of 5 triplets being incorrectly formed sequences of 3 entities without relations that are lists of synonyms (ex: (fast, rapid, fleet) or (cold, icy, glacial)). Comparatively, the more explicit and factual description used for Similarity results in no such triplets. Finally, while Meronymy triplets relate components and what they are a part of (e.g.: (petal, is a part of, flower)), roughly half of Member-collection relations specify the generic term use to designate a member (head) of a larger set (tail), which is a literal interpretation of the description in the prompt.

A.2 Quantitative analysis setup
To evaluate the extent of the above-mentioned phenomena, we performed a human evaluation by submitting a sample of the generated triplets to an evaluator among the authors. Inspired by (Jiang et al., 2024), we adapted their evaluation metrics into 8 multiple-choice questions to fit human evaluation and the range of relations generated. For each item, only one answer is possible. The exact description of the items and the possible answers, as well as the examples given in the instructions, are listed hereafter. For each triplet, all the information used during generation is provided to the evaluator

7https://spacy.io/api/lemmatizer, using the en_core_web_sm model
as in example Fig. 1.

Grammaticality: Is the triplet (h, r, t) well formed (use the examples as a reference)? If NO is selected, all other values are ignored, as the triplet is not valid.

• YES: Triplet is of the form (h, r, t) with r a relation from h to t, the latter two being of a similar nature.

• NO: In "(fast, rapid, fleet)", "rapid" is not a relation so the triplet is ungrammatical. "(honda cr, v, is a model)" is incorrectly separated into head, relation, and tail.

Topical Similarity: Does the relation fit the definition of the category used in the prompt? If NO is selected, "Uniqueness / diversity (relation)" is not accounted for.

• YES: For instance, relations "leads to, regulates, prevents" are exact match for the definition of causality.

• MARGINALLY: If we consider a relaxed interpretation of the definition, then yes. For instance relations "destroys" is a marginal match for causality, as it causes destruction.

• NO: For instance, "(honesty, is, essential for trust)" is not a valid triplet for causality, as "is" is not a causal relation.

Uniqueness / diversity (relation): Is the set of triplets different from the provided examples?

• REFORMULATION: Reuses a relation of an example relation (to a reformulation). For instance, "causes" is redundant with "is a cause of".

• SYNONYM: Relation with close meaning to an example relation, but a different root. For instance, "causes" is close to "influences".

• RELATED: Relation without close meaning to an example triplet. For instance, "causes" is related to "destroys".

Fatualness: Is the generated triplet true, or at least usually true?

• ALWAYS TRUE: There are no exception, for instance "(cat, is a, mammal)" USUALLY TRUE or COMMON-SENSE: There are some exceptions, but it can be commonly accepted, for instance "(cat, sleeps in, basket)".

• RARELY TRUE or SUBJECTIVE: There are some instances where it is true, but usually it is not, for instance "(pig, eats, duck foie gras)".

• FALSE or HALLUCINATION: Never true, for instance "(cat, is a, fish)", or nonsensical, for instance "(cat, sleeps in, dog)".

Granularity: Does the triplet express a broad relation or a very specific relation that belongs to the relation category?

• FINE-GRAINED: Can be relaxed into triplets that would better fit the relation category, for instance for causality "(speeding, is the main cause of, road traffic injuries)" could be relaxed into "(speeding, causes, road traffic injuries)" and still be adequate.

• MEDIUM-GRAINED: Can not be split into triplets that would better fit the relation category, for instance for causality "(storm, causes, floods)".

• COARSE-GRAINED: The triplet expresses a broad relation, that could be refined into one or multiple triplets that better fit the relation category, for instance for causality "(storm, influences, crops)" could be refined into "(storm, destroys, crops)".

Originality: Is the generated triplet informative and non-trivial?

• TRIVIAL: Triplets that contain no common-sense knowledge, for instance "(cat, eats, cat food)" where de facto "cat food" is food for "cats", or "(good day, opposite of, bad day)" that opposes good with bad with no necessary knowledge on "day".

• INFORMATIVE: Triplets that contain common-sense knowledge, for instance "(to be employed, entails, to have a salary)".

• ORIGINAL or INVOLVED: Triplets that contain specialist or involved knowledge, for instance "(blue light, more energetic than, red light)" or "(to be employed, entails, to pay taxes)".

Uniqueness / diversity (head): Is the set of triplets different enough from the provided examples?
• REFORMULATION: Reuses the head or tail of an example triplet (to a reformulation). For instance, "storm" is redundant with "stormy weather".

• SYNONYM: Has a close meaning to an example head or tail, but a different root. For instance, "storm" is synonymous to "tempest".

• RELATED: Is in the lexical field of an example head or tail. For instance, "storm" is related to "flood".

• UNRELATED: Is not in the lexical field of an example head or tail. For instance, "storm" is unrelated to "cat".

Uniqueness/diversity (tail): Is the set of triplets different enough from the provided examples?

• REFORMULATION: Reuses the head or tail of an example triplet (to a reformulation). For instance, "storm" is redundant with "stormy weather".

• SYNONYM: Has a close meaning to an example head or tail, but a different root. For instance, "storm" is synonymous to "tempest".

• RELATED: Is in the lexical field of an example head or tail. For instance, "storm" is related to "flood".

• UNRELATED: Is not in the lexical field of an example head or tail. For instance, "storm" is unrelated to "cat".

A.3 Quantitative analysis results

Two of the authors and a non-expert graduate student were recruited for the evaluation.

In a first pilot evaluation, performed by one of the authors, the first item (Grammaticality) was evaluated on 20 randomly selected triplets of each category, and the other items were evaluated on the first 10 selected triplets. In total, 320 triplets were evaluated for Grammaticality only and 160 triplet were evaluated for all items. The results of the pilot study are presented in Tab. 5.

In the second wave of evaluation, evaluators were presented with 10 randomly selected triplets of each category, the first 5 being shared across all annotators for inter annotator agreement measure. In total for all 3 annotators including the pilot study, 80 items were shared and 80 were specific to each annotator. The results are summarized in Tab. 4. Cohen’s \( \kappa \) score of inter annotator agreement is as follows for each item:

• Grammaticality: 0.86
• Topical Similarity: 0.41
• Fatualness: 0.39
• Granularity: 0.26
• Originality: 0.16
• Uniqueness / diversity (relation): 0.29
• Uniqueness / diversity (head): 0.30
• Uniqueness / diversity (tail): 0.34

Reliability and overall quality of the triplets
Of the 320 triplets considered in the pilot study, 11 were ungrammatical (1 from Common-sense and the other 10 from Synonymy), meaning they do not follow the expected HEAD, RELATION, TAIL structure. Over the 160 triplets analyzed in details, 6 were ungrammatical, so the questionnaire items excluding Grammaticality were studied over 154 triplets. Out of those 154 triplets, 5 were hallucinations as can be seen in the Fatualness item, which means they do not describe true information. In the full scale study, similar proportions were observed, with less than 4% of ungrammatical triplets or hallucinations.

<table>
  <tr>
    <th colspan="2">Triplet: ('to buy a computer', 'entails', 'using software')</th>
  </tr>
  <tr>
    <th>Head:</th>
    <td>to buy a computer</td>
  </tr>
  <tr>
    <th>Relation:</th>
    <td>entails</td>
  </tr>
  <tr>
    <th>Tail:</th>
    <td>using software</td>
  </tr>
  <tr>
    <th>Category:</th>
    <td>entailment</td>
  </tr>
  <tr>
    <th>Description:</th>
    <td>entailment triplets that describe logical relationships where one concept or action logically infers another</td>
  </tr>
  <tr>
    <th>Examples:</th>
    <td>
      to sleep, entails, to be tired<br>
      to eat, entails, to be hungry<br>
      to rain, entails, to carry an umbrella<br>
      to study, entails, to learn<br>
      to drive, entails, having a license
    </td>
  </tr>
</table>

Figure 1: Context provided for a triplet ('to buy a computer', 'entails', 'using software') generated for the category Entailment.
<table>
  <tr>
    <th>Item</th>
    <th>Choice</th>
    <th>Proportion of item (in %)</th>
  </tr>
  <tr>
    <td>Grammaticality</td>
    <td>YES</td>
    <td>96.74</td>
  </tr>
  <tr>
    <td></td>
    <td>NO</td>
    <td>3.26</td>
  </tr>
  <tr>
    <td>Topical Similarity</td>
    <td>YES</td>
    <td>92.73</td>
  </tr>
  <tr>
    <td></td>
    <td>MARGINALLY</td>
    <td>4.49</td>
  </tr>
  <tr>
    <td></td>
    <td>NO</td>
    <td>2.78</td>
  </tr>
  <tr>
    <td>Fatualness</td>
    <td>ALWAYS TRUE</td>
    <td>54.70</td>
  </tr>
  <tr>
    <td></td>
    <td>USUALLY T. or C.-S.</td>
    <td>32.05</td>
  </tr>
  <tr>
    <td></td>
    <td>RARELY T. or SUBJ.</td>
    <td>9.40</td>
  </tr>
  <tr>
    <td></td>
    <td>FALSE or HALLU.</td>
    <td>3.85</td>
  </tr>
  <tr>
    <td>Granularity</td>
    <td>FINE-GRAINED</td>
    <td>4.49</td>
  </tr>
  <tr>
    <td></td>
    <td>MEDIUM-GRAINED</td>
    <td>94.87</td>
  </tr>
  <tr>
    <td></td>
    <td>COARSE-GRAINED</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Originality</td>
    <td>ORIGINAL or INV.</td>
    <td>12.69</td>
  </tr>
  <tr>
    <td></td>
    <td>INFORMATIVE</td>
    <td>78.56</td>
  </tr>
  <tr>
    <td></td>
    <td>TRIVIAL</td>
    <td>8.75</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (relation)</td>
    <td>RELATED</td>
    <td>16.08</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>34.36</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>49.56</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (head)</td>
    <td>UNRELATED</td>
    <td>82.69</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>8.55</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>3.63</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>5.13</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (tail)</td>
    <td>UNRELATED</td>
    <td>83.54</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>12.18</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>2.14</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>2.14</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (highest relatedness between head and tail per triplet)</td>
    <td>UNRELATED</td>
    <td>79.28</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>10.04</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>4.27</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>6.41</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (lowest relatedness between head and tail per triplet)</td>
    <td>UNRELATED</td>
    <td>86.97</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>10.68</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>1.50</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>0.85</td>
  </tr>
</table>

Table 4: Detailed result for each relation category, cumulated over all answers of the 3 annotators. The top items correspond to items in the questionnaire while the bottom ones are computed from the answers to “Uniqueness / diversity (head)” and “Uniqueness / diversity (tail)”. “USUALLY T. or C.-S.” stands for “USUALLY TRUE or COMMON SENSE”; “RARELY T. or SUBJ.” stands for “RARELY T. or SUBJ.”; “FALSE or HALLU.” stands for “FALSE or HALLU.”; “ORIGINAL or INV.” stands for “ORIGINAL or INV.”.

If we consider the added value of the generated triplet measured with the Originality item, we can observe only 8.75% of trivial items such as (big opportunity, opposite of, small opportunity) or (dell inspiron, is a model of, dell). Interestingly, close to 12.7% of triplets can be considered as displaying non-trivial knowledge, such as (strut support, is a component of, aircraft), (to have a dream, entails, setting goals) or (finger print, similar to, signature). Most of these involved triplets belong to the Functional category, such as (historian, preserves, traditions), or the Attribute and Meronymy categories.

Quality of the relation In the pilot study, only 2 triplets had relations that did not match the relation category they were generated for (Topical Similarity: NO). Additionally, as can be seen in the Granularity item, 3 Causal triplets were overly precise (FINE-GRAINED), and only 2 triplets were not precise enough (COARSE-GRAINED). Overall, the vast majority of the generated relation were suitable in granularity and meaning for the category they belong to.

The originality of the generated triplets in terms of the relation can be observed with the Uniqueness / diversity (relation) item. Some relation categories where a variety of relations were used in the examples (such as Functional, Common-Sense, and Causal), display a variety of relations in the generated triplets, while for categories with a narrow set of valid formulations (like Entailment, Attribute, Collocation, etc.) the generate triplets closely match the provided examples.

Originality of the head and tail While the Originality item reveals how trivial or involved the knowledge expressed by a triplet is, it is not sufficient to confirm that the generated triplets are not just repetitions of the elements in the provided examples. To address this, the Uniqueness / diversity (head) and Uniqueness / diversity (tail) items allow us to consider the originality of the HEAD and TAIL of each triplet. Over all HEADs and TAILS, we observe above 80% of terms that are completely unrelated to the provided examples for the category, with less than 10% of REFORMULATIONS and SYNONYMs to terms present in the examples. This indicates that while the majority of triplets contain original elements, a significant amount of HEADs and TAILS are still close to the provided examples. On the plus side, when we consider both the HEAD and TAIL of the triplet, only 2.35% of triplets have both HEAD and TAIL
as a SYNONYM or REFORMULATION (lowest relatedness, last group of rows of Tab. 4). Comparatively, 89.32% of triplets have both HEAD and TAIL at most in the same lexical field of the ones of the examples (highest relatedness, before-last group of rows of Tab. 4).

Overall, despite the lack of instruction like "do not reuse terms that appear in the examples for the HEAD or TAIL", the LLM tends to generate diverse HEADs and TAILS compared to the provided examples.

B  LLMs Prompts

<table>
  <tr>
    <th>You are tasked with identifying the most analogous pair of words based on the given example.<br>The example pair is: lie and prevaricate.<br><br>From the following list, choose the most fitting pair:<br><br>1. betray and trust<br>2. philander and donate<br>3. waver and falter<br>4. deride and praise<br>5. corroborate and doubt<br><br>Only provide the number of the correct answer.</th>
  </tr>
</table>

Figure 2: Example of the prompt used for GPT-4o and Llama-3.3-70B-Instruct in 0-shot mode for the analogy question task.

<table>
  <tr>
    <th>You are tasked with identifying the most fitting relationship of the given example.<br>The example pair is: turtle and live.<br><br>From the following list, choose the most fitting relationship:<br><br>1. hyper<br>2. coord<br>3. mero<br>4. random<br>5. attri<br>6. event<br><br>Only provide the number of the correct answer.</th>
  </tr>
</table>

Figure 3: Example of the prompt used for GPT-4o and Llama-3.3-70B-Instruct in 0-shot mode for the lexical relation classification task.

C  LLMs performance

The performance of GPT-4o in both 0-shot and 5-shot settings is reported in Appendix Tab. 8 for analogy question and Appendix Tab. 9 for lexical relation classification. Note that the evaluation is limited to the first 100 items, which should be considered when interpreting the results.

For analogy questions, Llama3.3 and GPT-4o in 0-shot and 5-shot settings show strong performance in specific datasets like BATS and Google but has inconsistent results overall with average scores of 56.49% (0-shot) and 59.59% (5-shot) for Llama3.3. This contrasts with our proposed models and ReBERT, with for instance ReBERT-large demonstrating strong performance across datasets, achieving an average accuracy of 63.37%, with particularly high scores on Google and NELL.

Llama3.3 performs poorly in the 0-shot setting for lexical relation classification, with an average score of 43.0%, and struggles on datasets like K&H+N (13%) and CogALexV. In the 5-shot setting, its performance improves significantly, achieving an average of 70.4% and performing well on ROOT09 and CogALexV. However, it still falls short of task-specific models like MultiPRE-large, underscoring the advantages of specialized relational encoders. While Llama3.3 shows promise in few-shot, it remains less effective than relational encoders, further emphasizing the importance of
<table>
  <tr>
    <th rowspan="2">Item</th>
    <th rowspan="2">Choice</th>
    <th colspan="19">TOTAL</th>
  </tr>
  <tr>
    <th>enrollment</th>
    <th>hypernymy</th>
    <th>attribute</th>
    <th>hyponymy</th>
    <th>functional</th>
    <th>metonymy</th>
    <th>temporal</th>
    <th>common noun</th>
    <th>member-of relation</th>
    <th>similarity</th>
    <th>synonymy</th>
    <th>collocation</th>
    <th>toponymy</th>
    <th>antonymy</th>
    <th>spatial</th>
    <th>cultural</th>
    <th>TOTAL</th>
  </tr>
  <tr>
    <td>Grammaticality</td>
    <td>YES</td>
    <td>20</td><td>20</td><td>20</td><td>20</td><td>20</td><td>20</td><td>19</td><td>20</td><td>19</td><td>20</td><td>10</td><td>20</td><td>20</td><td>20</td><td>20</td><td>309</td>
  </tr>
  <tr>
    <td></td>
    <td>NO</td>
    <td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>11</td>
  </tr>
  <tr>
    <td>Topical Similarity</td>
    <td>YES</td>
    <td>10</td><td>10</td><td>10</td><td>9</td><td>10</td><td>8</td><td>8</td><td>9</td><td>10</td><td>4</td><td>10</td><td>10</td><td>10</td><td>10</td><td>10</td><td>147</td>
  </tr>
  <tr>
    <td></td>
    <td>MARGINALLY</td>
    <td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td>
  </tr>
  <tr>
    <td></td>
    <td>NO</td>
    <td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>2</td>
  </tr>
  <tr>
    <td>Fatualness</td>
    <td>ALWAYS TRUE</td>
    <td>3</td><td>7</td><td>5</td><td>10</td><td>2</td><td>9</td><td>5</td><td>3</td><td>3</td><td>6</td><td>4</td><td>8</td><td>10</td><td>9</td><td>0</td><td>2</td><td>86</td>
  </tr>
  <tr>
    <td></td>
    <td>USUALLY T. or C.-S.</td>
    <td>7</td><td>0</td><td>2</td><td>0</td><td>7</td><td>1</td><td>3</td><td>4</td><td>7</td><td>3</td><td>1</td><td>2</td><td>0</td><td>0</td><td>3</td><td>8</td><td>48</td>
  </tr>
  <tr>
    <td></td>
    <td>RARELY T. or SUBJ.</td>
    <td>0</td><td>1</td><td>3</td><td>0</td><td>1</td><td>0</td><td>1</td><td>2</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>6</td><td>0</td><td>15</td>
  </tr>
  <tr>
    <td></td>
    <td>FALSE or HALLU.</td>
    <td>0</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>5</td>
  </tr>
  <tr>
    <td>Granularity</td>
    <td>FINE-GRAINED</td>
    <td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>3</td>
  </tr>
  <tr>
    <td></td>
    <td>MEDIUM-GRAINED</td>
    <td>10</td><td>9</td><td>10</td><td>10</td><td>9</td><td>10</td><td>10</td><td>9</td><td>10</td><td>5</td><td>10</td><td>10</td><td>10</td><td>10</td><td>7</td><td>149</td>
  </tr>
  <tr>
    <td></td>
    <td>COARSE-GRAINED</td>
    <td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>2</td>
  </tr>
  <tr>
    <td>Originality</td>
    <td>ORIGINAL or INV.</td>
    <td>5</td><td>1</td><td>5</td><td>0</td><td>8</td><td>4</td><td>1</td><td>2</td><td>2</td><td>3</td><td>0</td><td>1</td><td>1</td><td>0</td><td>2</td><td>4</td><td>39</td>
  </tr>
  <tr>
    <td></td>
    <td>INFORMATIVE</td>
    <td>5</td><td>9</td><td>5</td><td>9</td><td>2</td><td>5</td><td>9</td><td>7</td><td>8</td><td>7</td><td>5</td><td>8</td><td>9</td><td>8</td><td>6</td><td>110</td>
  </tr>
  <tr>
    <td></td>
    <td>TRIVIAL</td>
    <td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>5</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (relation)</td>
    <td>RELATED</td>
    <td>0</td><td>1</td><td>0</td><td>10</td><td>0</td><td>1</td><td>8</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>3</td><td>24</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>0</td><td>2</td><td>0</td><td>4</td><td>0</td><td>2</td><td>0</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>3</td><td>12</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>10</td><td>7</td><td>10</td><td>5</td><td>10</td><td>6</td><td>1</td><td>10</td><td>5</td><td>10</td><td>10</td><td>10</td><td>8</td><td>4</td><td>8</td><td>116</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (head)</td>
    <td>UNRELATED</td>
    <td>8</td><td>9</td><td>10</td><td>5</td><td>7</td><td>7</td><td>5</td><td>8</td><td>6</td><td>5</td><td>2</td><td>8</td><td>8</td><td>10</td><td>4</td><td>7</td><td>109</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>1</td><td>0</td><td>5</td><td>1</td><td>3</td><td>3</td><td>0</td><td>3</td><td>4</td><td>2</td><td>2</td><td>0</td><td>2</td><td>1</td><td>2</td><td>1</td><td>27</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>1</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>7</td><td>7</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>0</td><td>1</td><td>0</td><td>2</td><td>0</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>4</td><td>1</td><td>11</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (tail)</td>
    <td>UNRELATED</td>
    <td>9</td><td>9</td><td>9</td><td>5</td><td>9</td><td>6</td><td>7</td><td>8</td><td>5</td><td>6</td><td>3</td><td>8</td><td>8</td><td>10</td><td>4</td><td>6</td><td>112</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>0</td><td>1</td><td>5</td><td>1</td><td>3</td><td>2</td><td>1</td><td>4</td><td>3</td><td>1</td><td>2</td><td>0</td><td>5</td><td>3</td><td>3</td><td>33</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>5</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>4</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (head + tail)</td>
    <td>UNRELATED</td>
    <td>17</td><td>18</td><td>19</td><td>10</td><td>16</td><td>13</td><td>12</td><td>16</td><td>11</td><td>11</td><td>5</td><td>16</td><td>20</td><td>8</td><td>13</td><td>221</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>1</td><td>1</td><td>1</td><td>10</td><td>2</td><td>6</td><td>5</td><td>1</td><td>7</td><td>7</td><td>3</td><td>1</td><td>0</td><td>7</td><td>4</td><td>60</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>2</td><td>0</td><td>1</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>2</td><td>2</td><td>12</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>1</td><td>1</td><td>0</td><td>0</td><td>2</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>5</td><td>1</td><td>15</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (highest relatedness between head and tail per triplet)</td>
    <td>UNRELATED</td>
    <td>8</td><td>9</td><td>9</td><td>5</td><td>7</td><td>6</td><td>5</td><td>8</td><td>5</td><td>5</td><td>2</td><td>8</td><td>8</td><td>10</td><td>3</td><td>5</td><td>103</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>1</td><td>0</td><td>1</td><td>5</td><td>1</td><td>3</td><td>3</td><td>0</td><td>4</td><td>3</td><td>2</td><td>0</td><td>3</td><td>2</td><td>3</td><td>2</td><td>30</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>2</td><td>2</td><td>7</td><td>7</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>1</td><td>1</td><td>0</td><td>0</td><td>2</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>4</td><td>1</td><td>14</td>
  </tr>
  <tr>
    <td>Uniqueness / diversity (lowest relatedness between head and tail per triplet)</td>
    <td>UNRELATED</td>
    <td>9</td><td>9</td><td>10</td><td>5</td><td>9</td><td>7</td><td>7</td><td>8</td><td>6</td><td>6</td><td>3</td><td>8</td><td>8</td><td>10</td><td>5</td><td>8</td><td>118</td>
  </tr>
  <tr>
    <td></td>
    <td>RELATED</td>
    <td>0</td><td>1</td><td>5</td><td>1</td><td>3</td><td>2</td><td>1</td><td>3</td><td>4</td><td>1</td><td>1</td><td>2</td><td>4</td><td>2</td><td>3</td><td>30</td>
  </tr>
  <tr>
    <td></td>
    <td>SYNONYM</td>
    <td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>5</td><td>0</td><td>5</td>
  </tr>
  <tr>
    <td></td>
    <td>REFORMULATION</td>
    <td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td>
  </tr>
</table>

Table 5: Detailed result of the pilot study, for each relation category. The top items correspond to items in the questionnaire while the bottom ones are computed from the answers to “Uniqueness / diversity (head)” and “Uniqueness / diversity (tail)”. “USUALLY T. or C.-S.” stands for “USUALLY TRUE or COMMON SENSE”; “RARELY T. or SUBJ.” stands for “RARELY TRUE or SUBJECTIVE”; “FALSE or HALLU.” stands for “FALSE or HALLUCINATION”; “ORIGINAL or INV.” stands for “ORIGINAL or INVOLVED”.
<table>
  <tr>
    <th>Relation</th>
    <th>Corresponding RelSim relation</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Commonsense</td>
    <td>–</td>
    <td>knowing knowledge triplets that describe commonsense relation</td>
  </tr>
  <tr>
    <td>Functional</td>
    <td>–</td>
    <td>functional triplets that describe the function or role of an object or entity</td>
  </tr>
  <tr>
    <td>Collocation</td>
    <td>–</td>
    <td>collocation triplets that describe words that frequently occur together</td>
  </tr>
  <tr>
    <td>Troponymy</td>
    <td>–</td>
    <td>troponymy triplets that describe specific ways in which an action is performed</td>
  </tr>
  <tr>
    <td>Antonymy Attribute</td>
    <td>Antonym Attribute</td>
    <td>antonymy triplets that describe opposite concepts or entities attribute triplets that describe inherent properties or qualities of objects or entities</td>
  </tr>
  <tr>
    <td>Causal Entailment</td>
    <td>Cause-Purpose Cause-Purpose</td>
    <td>causal triplets that describe cause and effect relationships entailment triplets that describe logical relationships where one concept or action logically infers another</td>
  </tr>
  <tr>
    <td>Spatial</td>
    <td>Space-Time</td>
    <td>spatial triplets that describe spatial relationship between objects or entities</td>
  </tr>
  <tr>
    <td>Temporal</td>
    <td>Space-Time</td>
    <td>temporal triplets that describe the temporal relationship between events or states</td>
  </tr>
  <tr>
    <td>Hyponymy</td>
    <td>Hypernym</td>
    <td>hyponymy triplets that describe the relationship where one term is a subtype or a specific instance of another</td>
  </tr>
  <tr>
    <td>Hypernymy</td>
    <td>Hypernym</td>
    <td>hypernymy triplets that describe the relationship where one term encompasses a broader category to which the other belongs</td>
  </tr>
  <tr>
    <td>Meronymy</td>
    <td>Meronym</td>
    <td>meronymy triplets that describe part-whole relationships</td>
  </tr>
  <tr>
    <td>Member-collection</td>
    <td>Meronym</td>
    <td>member-collection triplets that describe the relationship between an individual item and the group to which it belongs</td>
  </tr>
  <tr>
    <td>Synonymy Similarity</td>
    <td>Synonym Synonym</td>
    <td>synonymy triplets that describe similar or identical concepts similarity triplets that describe the likeness or analogies between different concepts or entities</td>
  </tr>
</table>

Table 6: Relations considered and corresponding description for the prompts of the LLM generated data.
<table>
  <tr>
    <th>Relation</th>
    <th>Few-shot examples</th>
  </tr>
  <tr>
    <td>Commonsense</td>
    <td>(pen, writes with, ink), (chair, sits on, person), (fire, burns, wood), (house, lives in, family), (pencil, used for, drawing)</td>
  </tr>
  <tr>
    <td>Functional</td>
    <td>(engine, powers, machine), (nurse, cares, sick person), (beekeeper, monitors, honey production), (ocean, regulates, climate), (carpenter, builds, house), (researcher, discovers, new knowledge)</td>
  </tr>
  <tr>
    <td>Collocation</td>
    <td>(fire, and, flames), (wind, and, breeze), (coffee, and, cream), (tea, and, leaves), (sand, and, beach), (mountain, and, peak)</td>
  </tr>
  <tr>
    <td>Troponymy</td>
    <td>(hurry, is a way to, move), (gallop, is a way to, run), (trudge, is a way to, walk), (amble, is a way to, walk), (skip, is a way to, walk), (dash, is a way to, run)</td>
  </tr>
  <tr>
    <td>Antonymy</td>
    <td>(joy, opposite of, sorrow), (empty, opposite of, full), (silence, opposite of, noise), (morning, opposite of, night), (old, opposite of, young), (pure, opposite of, impure)</td>
  </tr>
  <tr>
    <td>Attribute</td>
    <td>(snow, has, whiteness), (silk, has, softness), (copper, has, conductivity), (butter, has, richness), (charcoal, has, carbon content), (perfume, has, fragrance)</td>
  </tr>
  <tr>
    <td>Causal</td>
    <td>(rain, brings, flowers), (learning, improves, memory), (eating, provides, energy), (sleep, helps, concentration), (reading, expands, vocabulary), (meditation, reduces, stress)</td>
  </tr>
  <tr>
    <td>Entailment</td>
    <td>(to read a book, entails, having eyes), (to write, entails, having a pen), (to bake a cake, entails, having flour), (to plant a garden, entails, having soil), (to learn a language, entails, studying vocabulary), (to be married, entails, having a wedding)</td>
  </tr>
  <tr>
    <td>Spatial</td>
    <td>(chair, beside, sofa), (kite, in, sky), (dog, on, beach), (phone, in, pocket), (book, on, desk), (man, on, bike)</td>
  </tr>
  <tr>
    <td>Temporal</td>
    <td>(childhood, before, adolescence), (learning, before, graduation), (voting, before, election), (winter, before, spring), (childhood, before, adulthood), (learning, before, promotion)</td>
  </tr>
  <tr>
    <td>Hyponymy</td>
    <td>(ferrari, is a brand of, car), (pineapple, is a type of, fruit), (guitar, is a type of, instrument), (laptop, is a type of, computer), (dolphin, is a type of, mammal), (trumpet, is a type of, instrument)</td>
  </tr>
  <tr>
    <td>Hypernymy</td>
    <td>(sport, categorizes, basketball), (animal, encompasses, cat), (music, includes, jazz), (food, categorizes, pizza), (building, covers, house), (poem, encompasses, sonnet)</td>
  </tr>
  <tr>
    <td>Meronymy</td>
    <td>(gear, is a component of, machine), (atom, is a component of, molecule), (fin, is a part of, fish), (bolt, is a component of, lock), (thread, is a part of, fabric), (blade, is a part of, windmill)</td>
  </tr>
  <tr>
    <td>Member-collection</td>
    <td>(bee, member of, swarm), (fish, part of, school), (tree, part of, forest), (book, part of, library), (student, member of, class)</td>
  </tr>
  <tr>
    <td>Synonymy</td>
    <td>(expensive, similar to, costly), (beautiful, similar to, lovely), (angry, similar to, irate), (busy, similar to, hectic), (soft, similar to, gentle), (sharp, similar to, pointed)</td>
  </tr>
  <tr>
    <td>Similarity</td>
    <td>(wings, similar to, arms of a butterfly), (bee, similar to, worker in a factory), (pen, similar to, brush), (voice, similar to, song), (wheels, similar to, pedals of a bicycle), (clouds, similar to, cotton balls)</td>
  </tr>
</table>

Table 7: Relations considered and corresponding few-shot examples for the prompts of the LLM generated data.
Figure 4: Example of the proposed models for the triplet (hurry, is a way to, move) from the Troponymy category. UniPRE uses prompt1 and MultiPRE uses prompt5 from Tab. 10.

task-specific training for lexical relation classification.

D  Details for our models

D.1 Structure

In Fig. 4 we illustrate the workings of the three models proposed in Sec. 4. \( \oplus \) corresponds to concatenation, and \( \odot \) corresponds to the Hadamar product.

D.2 Training hyperparameters

The training process for all encoders uses early stopping with a patience of 30, a batch size of 256, a learning rate of \( 1 \times 10^{-5} \), and a temperature parameter set to 0.03. These hyperparameters are chosen to ensure stability and convergence across diverse model variants and tasks.

D.3 Prompts

Tab. 10 show all different prompts used. Prompts 1 to 4 are designed for the Uni-Aspect Relation Encoder (UniPRE), while Prompts 5 and 6 are tailored for the Multi-Aspects Encoder (MultiPRE).

D.4 Analogy

Refer to Tab. 11 for a comparison between our DeBERTa-based encoder and RelBERT on the analogy question task.

D.4.1 Semantic Properties Encoder

Tabs. 12 to 17 detail the performance of the SemPRE model on Analogy Questions for DeBERTa, RoBERTa, and BERT base and large checkpoints, for each combination of fine-tuning datasets.

D.4.2 Single and Multi Aspect LM

Tabs. 18 to 23 detail the performance of the UniPRE and MultiPRE on Analogy Questions for DeBERTa, RoBERTa, and BERT base and large checkpoints, for each combination of fine-tuning datasets.

D.5 Lexical relation classification

Refer to Tab. 24 for a comparison between our DeBERTa-based encoder and RelBERT on the lexical relation classification task.

D.5.1 Semantic Properties Encoder

Tabs. 25 to 30 detail the performance of SemPRE on Lexical Relation Classification for DeBERTa, RoBERTa, and BERT base and large checkpoints, for each combination of fine-tuning datasets.

D.5.2 Single and Multi Aspect LM

Tabs. 31 to 36 detail the performance of UniPRE and MultiPRE on Lexical Relation Classification for DeBERTa, RoBERTa, and BERT base and large checkpoints, for each combination of fine-tuning datasets.
<table>
  <tr>
    <th>Model</th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>Google</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>GPT-4o 0-shot</td>
    <td>84</td>
    <td>81</td>
    <td>99</td>
    <td><b>100</b></td>
    <td>25</td>
    <td>48</td>
    <td>79</td>
    <td>13</td>
    <td>66.13</td>
  </tr>
  <tr>
    <td>GPT-4o 5-shot</td>
    <td><b>87</b></td>
    <td><b>86</b></td>
    <td><b>100</b></td>
    <td>99</td>
    <td>32</td>
    <td><b>89</b></td>
    <td><b>79</b></td>
    <td>26</td>
    <td><b>74.75</b></td>
  </tr>
  <tr>
    <td>Llama3.3 0-shot</td>
    <td>76</td>
    <td>68</td>
    <td>95</td>
    <td>97</td>
    <td>22</td>
    <td>38</td>
    <td>45</td>
    <td><b>29</b></td>
    <td>58.75</td>
  </tr>
  <tr>
    <td>Llama3.3 5-shot</td>
    <td>81</td>
    <td>69</td>
    <td>97</td>
    <td>98</td>
    <td><b>36</b></td>
    <td>55</td>
    <td>54</td>
    <td>25</td>
    <td>64.38</td>
  </tr>
</table>

Table 8: Accuracy (in %) on analogy questions, was test on the 100 first items per datasets.

<table>
  <tr>
    <th>Model</th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>GPT-4o 0-shot</td>
    <td>57</td>
    <td>75</td>
    <td>61</td>
    <td>56</td>
    <td>83</td>
    <td>66.4</td>
  </tr>
  <tr>
    <td>GPT-4o 5-shot</td>
    <td><b>79</b></td>
    <td><b>91</b></td>
    <td><b>70</b></td>
    <td><b>84</b></td>
    <td><b>85</b></td>
    <td><b>81.8</b></td>
  </tr>
  <tr>
    <td>Llama3.3 0-shot</td>
    <td>43</td>
    <td>52</td>
    <td>58</td>
    <td>13</td>
    <td>44</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>Llama3.3 5-shot</td>
    <td>69</td>
    <td>73</td>
    <td>64</td>
    <td>71</td>
    <td>75</td>
    <td>70.4</td>
  </tr>
</table>

Table 9: Micro F1-score (in %) on lexical relation classification, was test on the 100 first items per datasets.

<table>
  <tr>
    <th>Name</th>
    <th>Model</th>
    <th>Template</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>UniPRE</td>
    <td>The relationship between [HEAD] and [TAIL] is <mask>.</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>UniPRE</td>
    <td>The word that best describes the relationship between [HEAD] and [TAIL] is <mask>.</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>UniPRE</td>
    <td>People often use the word <mask> to describe the relationship between [HEAD] and [TAIL].</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>UniPRE</td>
    <td>One property of [HEAD] is to be the <mask> of [TAIL].</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>MultiPRE</td>
    <td>- One property of [HEAD] is to be the <mask> of [TAIL].<br>- Usually, we are [TAIL] <mask> [HEAD].</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>MultiPRE</td>
    <td>- One property of [HEAD] is to be the <mask> of [TAIL].<br>- Usually, we are [TAIL] <mask> [HEAD].<br>- In terms of science, [HEAD] is the <mask> of [TAIL].</td>
  </tr>
</table>

Table 10: Link between prompt name and prompt template

<table>
  <tr>
    <th>Model</th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>Google</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>RelBERT-base</td>
    <td>59.65</td>
    <td>57.41</td>
    <td>70.32</td>
    <td>89.20</td>
    <td><b>25.93</b></td>
    <td>62.0</td>
    <td>66.67</td>
    <td>39.77</td>
    <td>58.86</td>
  </tr>
  <tr>
    <td colspan="10">RelKB</td>
  </tr>
  <tr>
    <td>UniPRE-base</td>
    <td>42.11</td>
    <td>49.31</td>
    <td>69.93</td>
    <td>88.8</td>
    <td>19.31</td>
    <td>76.5</td>
    <td>59.02</td>
    <td><b>41.19</b></td>
    <td>55.77</td>
  </tr>
  <tr>
    <td>MultiPRE-base</td>
    <td>53.51</td>
    <td>56.48</td>
    <td>72.21</td>
    <td>92.4</td>
    <td>19.43</td>
    <td><b>78.83</b></td>
    <td>62.84</td>
    <td>35.07</td>
    <td>58.85</td>
  </tr>
  <tr>
    <td>SemPRE-base</td>
    <td>34.65</td>
    <td>32.64</td>
    <td>45.53</td>
    <td>47.4</td>
    <td>10.4</td>
    <td>71.5</td>
    <td>32.24</td>
    <td>20.47</td>
    <td>36.85</td>
  </tr>
  <tr>
    <td colspan="10">SemRelLM</td>
  </tr>
  <tr>
    <td>UniPRE-base</td>
    <td>63.16</td>
    <td>67.59</td>
    <td>72.15</td>
    <td>89.0</td>
    <td>20.61</td>
    <td>59.67</td>
    <td>63.93</td>
    <td>34.06</td>
    <td>58.77</td>
  </tr>
  <tr>
    <td>MultiPRE-base</td>
    <td><b>68.42</b></td>
    <td>69.91</td>
    <td>77.6</td>
    <td><b>94.4</b></td>
    <td>23.08</td>
    <td>66.83</td>
    <td>66.67</td>
    <td>35.32</td>
    <td>62.78</td>
  </tr>
  <tr>
    <td>SemPRE-base</td>
    <td>34.21</td>
    <td>40.05</td>
    <td>50.64</td>
    <td>53.8</td>
    <td>15.28</td>
    <td>55.83</td>
    <td>26.78</td>
    <td>20.55</td>
    <td>37.14</td>
  </tr>
  <tr>
    <td colspan="10">RelKB+SemRelLM</td>
  </tr>
  <tr>
    <td>UniPRE-base</td>
    <td>63.6</td>
    <td>67.13</td>
    <td>75.32</td>
    <td>91.2</td>
    <td>21.23</td>
    <td>74.17</td>
    <td>64.48</td>
    <td>35.32</td>
    <td>61.56</td>
  </tr>
  <tr>
    <td>MultiPRE-base</td>
    <td><b>67.98</b></td>
    <td><b>71.06</b></td>
    <td><b>78.04</b></td>
    <td>93.6</td>
    <td>24.75</td>
    <td>72.33</td>
    <td><b>68.85</b></td>
    <td>37.25</td>
    <td><b>64.23</b></td>
  </tr>
  <tr>
    <td>SemPRE-base</td>
    <td>34.65</td>
    <td>39.58</td>
    <td>50.47</td>
    <td>63.8</td>
    <td>15.53</td>
    <td>76.5</td>
    <td>37.16</td>
    <td>23.32</td>
    <td>42.63</td>
  </tr>
</table>

Table 11: Accuracy (in %) on analogy questions. For our models, we use DeBERTa. RelBERT models are reproduced following (Ushio et al., 2023).
<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>34.65</td>
    <td>39.58</td>
    <td>50.47</td>
    <td><b>63.8</b></td>
    <td>15.53</td>
    <td><b>76.5</b></td>
    <td><b>37.16</b></td>
    <td><b>23.32</b></td>
    <td><b>42.63</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>34.65</td>
    <td>32.64</td>
    <td>45.53</td>
    <td>47.4</td>
    <td>10.4</td>
    <td>71.5</td>
    <td>32.24</td>
    <td>20.47</td>
    <td>36.85</td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>34.21</td>
    <td><b>40.05</b></td>
    <td><b>50.64</b></td>
    <td>53.8</td>
    <td>15.28</td>
    <td>55.83</td>
    <td>26.78</td>
    <td>20.55</td>
    <td>37.14</td>
  </tr>
</table>

Table 12: SemPRE performance on analogy with deberta-base.

<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>28.95</td>
    <td>31.48</td>
    <td>25.74</td>
    <td>32.8</td>
    <td>5.94</td>
    <td>26.50</td>
    <td>14.75</td>
    <td>7.63</td>
    <td>21.72</td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>26.75</td>
    <td><b>31.71</b></td>
    <td><b>34.07</b></td>
    <td><b>37.4</b></td>
    <td><b>8.23</b></td>
    <td><b>39.67</b></td>
    <td><b>26.23</b></td>
    <td><b>11.74</b></td>
    <td><b>26.97</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>28.95</td>
    <td>30.56</td>
    <td>26.18</td>
    <td>28.8</td>
    <td>6.19</td>
    <td>24.00</td>
    <td>12.57</td>
    <td>8.47</td>
    <td>20.71</td>
  </tr>
</table>

Table 13: SemPRE performance on analogy with roberta-large.

<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>38.60</td>
    <td>40.05</td>
    <td><b>57.53</b></td>
    <td><b>62.2</b></td>
    <td><b>16.96</b></td>
    <td>69.83</td>
    <td>38.25</td>
    <td>24.83</td>
    <td><b>43.53</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>31.58</td>
    <td>32.64</td>
    <td>49.58</td>
    <td>53.6</td>
    <td>11.51</td>
    <td><b>74.67</b></td>
    <td><b>56.28</b></td>
    <td><b>26.85</b></td>
    <td>42.09</td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>37.72</td>
    <td><b>40.51</b></td>
    <td>49.58</td>
    <td>54.8</td>
    <td>15.53</td>
    <td>54.00</td>
    <td>16.39</td>
    <td>23.32</td>
    <td>36.48</td>
  </tr>
</table>

Table 14: SemPRE performance on analogy with deberta-large.

<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>24.12</td>
    <td><b>29.63</b></td>
    <td><b>33.24</b></td>
    <td><b>45.0</b></td>
    <td><b>7.49</b></td>
    <td>45.83</td>
    <td>18.03</td>
    <td><b>12.5</b></td>
    <td>26.98</td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>30.7</td>
    <td>28.94</td>
    <td>26.29</td>
    <td>37.8</td>
    <td>4.02</td>
    <td><b>59.0</b></td>
    <td><b>26.78</b></td>
    <td>8.39</td>
    <td><b>27.74</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>6.32</td>
    <td>28.01</td>
    <td>27.52</td>
    <td>31.8</td>
    <td>6.75</td>
    <td>23.83</td>
    <td>6.01</td>
    <td>9.06</td>
    <td>19.91</td>
  </tr>
</table>

Table 15: SemPRE performance on analogy with roberta-base

<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>39.47</td>
    <td>45.6</td>
    <td><b>48.25</b></td>
    <td>51.4</td>
    <td><b>26.73</b></td>
    <td>65.33</td>
    <td>49.73</td>
    <td>21.56</td>
    <td>43.51</td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>36.84</td>
    <td>36.11</td>
    <td>52.75</td>
    <td><b>54.6</b></td>
    <td>19.43</td>
    <td><b>69.83</b></td>
    <td><b>61.75</b></td>
    <td><b>25.08</b></td>
    <td><b>44.55</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>40.79</td>
    <td><b>45.83</b></td>
    <td>43.41</td>
    <td>41.8</td>
    <td>25.06</td>
    <td>51.83</td>
    <td>31.15</td>
    <td>19.04</td>
    <td>37.36</td>
  </tr>
</table>

Table 16: SemPRE performance on analogy with bert-base
<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>37.28</td>
    <td>45.14</td>
    <td><b>52.03</b></td>
    <td><b>57.8</b></td>
    <td><b>28.03</b></td>
    <td>66.33</td>
    <td>60.66</td>
    <td><b>23.07</b></td>
    <td><b>46.29</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>34.21</td>
    <td>33.56</td>
    <td>49.36</td>
    <td>45.2</td>
    <td>21.97</td>
    <td><b>71.0</b></td>
    <td><b>61.75</b></td>
    <td>22.15</td>
    <td>42.4</td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td><b>41.67</b></td>
    <td><b>47.92</b></td>
    <td>48.75</td>
    <td>54.4</td>
    <td>27.6</td>
    <td>57.0</td>
    <td>47.54</td>
    <td>22.82</td>
    <td>43.46</td>
  </tr>
</table>

Table 17: SemPRE performance on analogy with bert-large

<table>
  <tr>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>60.96</td>
    <td>59.72</td>
    <td>69.65</td>
    <td>88.6</td>
    <td>17.88</td>
    <td>53.33</td>
    <td>44.26</td>
    <td>24.16</td>
    <td>52.32</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>63.16</td>
    <td>62.27</td>
    <td>71.76</td>
    <td>90.0</td>
    <td>19.55</td>
    <td>52.5</td>
    <td>50.27</td>
    <td>21.73</td>
    <td>53.91</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>63.16</td>
    <td>62.27</td>
    <td>71.93</td>
    <td>89.0</td>
    <td>20.42</td>
    <td>49.67</td>
    <td>47.54</td>
    <td>25.0</td>
    <td>53.62</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>63.6</td>
    <td>67.13</td>
    <td>75.32</td>
    <td>91.2</td>
    <td>21.23</td>
    <td>74.17</td>
    <td>64.48</td>
    <td>35.32</td>
    <td>61.56</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td><b>69.74</b></td>
    <td><b>71.06</b></td>
    <td>77.49</td>
    <td>91.2</td>
    <td>22.28</td>
    <td>72.17</td>
    <td>64.48</td>
    <td>35.32</td>
    <td>62.97</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>67.98</td>
    <td><b>71.06</b></td>
    <td><b>78.04</b></td>
    <td>93.6</td>
    <td><b>24.75</b></td>
    <td>72.33</td>
    <td><b>68.85</b></td>
    <td>37.25</td>
    <td><b>64.23</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>47.81</td>
    <td>46.53</td>
    <td>61.87</td>
    <td>83.8</td>
    <td>13.68</td>
    <td>67.33</td>
    <td>45.36</td>
    <td>20.72</td>
    <td>48.39</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>53.51</td>
    <td>52.08</td>
    <td>69.98</td>
    <td>89.8</td>
    <td>14.67</td>
    <td>67.5</td>
    <td>50.82</td>
    <td>17.95</td>
    <td>52.04</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>42.54</td>
    <td>49.07</td>
    <td>68.32</td>
    <td>80.2</td>
    <td>16.21</td>
    <td>70.0</td>
    <td>62.84</td>
    <td>32.21</td>
    <td>52.67</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>42.11</td>
    <td>49.31</td>
    <td>69.93</td>
    <td>88.8</td>
    <td>19.31</td>
    <td>76.5</td>
    <td>59.02</td>
    <td>41.19</td>
    <td>55.77</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>44.74</td>
    <td>53.24</td>
    <td>73.54</td>
    <td>90.8</td>
    <td>18.32</td>
    <td>76.33</td>
    <td>60.11</td>
    <td><b>38.51</b></td>
    <td>56.95</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>53.51</td>
    <td>56.48</td>
    <td>72.21</td>
    <td>92.4</td>
    <td>19.43</td>
    <td><b>78.83</b></td>
    <td>62.84</td>
    <td>35.07</td>
    <td>58.85</td>
  </tr>
  <tr>
    <td colspan="9"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>61.4</td>
    <td>61.34</td>
    <td>70.71</td>
    <td>88.4</td>
    <td>17.64</td>
    <td>46.17</td>
    <td>37.7</td>
    <td>23.66</td>
    <td>50.88</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>62.72</td>
    <td>63.89</td>
    <td>69.09</td>
    <td>87.4</td>
    <td>17.33</td>
    <td>47.17</td>
    <td>40.44</td>
    <td>26.01</td>
    <td>51.76</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>61.84</td>
    <td>63.43</td>
    <td>67.59</td>
    <td>85.6</td>
    <td>17.39</td>
    <td>46.67</td>
    <td>40.44</td>
    <td>21.81</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>63.16</td>
    <td>67.59</td>
    <td>72.15</td>
    <td>89.0</td>
    <td>20.61</td>
    <td>59.67</td>
    <td>63.93</td>
    <td>34.06</td>
    <td>58.77</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>65.35</td>
    <td>64.58</td>
    <td>70.54</td>
    <td>88.2</td>
    <td>20.61</td>
    <td>62.0</td>
    <td>56.83</td>
    <td>32.89</td>
    <td>57.62</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>68.42</td>
    <td>69.91</td>
    <td>77.6</td>
    <td><b>94.4</b></td>
    <td>23.08</td>
    <td>66.83</td>
    <td>66.67</td>
    <td>35.32</td>
    <td>62.78</td>
  </tr>
</table>

Table 18: Accuracy (in %) for Uni-Aspect (prompts 1 to 4) and Multi-Aspect (prompts 5 and 6) Encoders on Analogy Tasks with deberta-base
<table>
  <tr>
    <th></th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="10"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>57.89</td>
    <td>65.05</td>
    <td>65.76</td>
    <td>78.4</td>
    <td>19.37</td>
    <td>71.5</td>
    <td>70.49</td>
    <td>32.47</td>
    <td>57.62</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>59.21</td>
    <td>62.73</td>
    <td>67.59</td>
    <td>82.0</td>
    <td>21.66</td>
    <td>73.0</td>
    <td>72.68</td>
    <td>31.12</td>
    <td>58.75</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>59.21</td>
    <td>63.66</td>
    <td>70.87</td>
    <td>86.0</td>
    <td>22.52</td>
    <td>73.5</td>
    <td>65.03</td>
    <td>32.05</td>
    <td>59.1</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>63.6</td>
    <td>67.13</td>
    <td>70.48</td>
    <td>84.6</td>
    <td>26.98</td>
    <td><b>80.0</b></td>
    <td><b>76.5</b></td>
    <td>42.45</td>
    <td><b>63.97</b></td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>64.04</td>
    <td>65.51</td>
    <td>66.93</td>
    <td>81.4</td>
    <td>24.57</td>
    <td>78.83</td>
    <td>67.76</td>
    <td>42.11</td>
    <td>61.39</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>63.6</td>
    <td>68.29</td>
    <td>70.65</td>
    <td>84.2</td>
    <td><b>28.03</b></td>
    <td>77.67</td>
    <td>73.77</td>
    <td>39.01</td>
    <td>63.15</td>
  </tr>
  <tr>
    <td colspan="10"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>45.61</td>
    <td>52.31</td>
    <td>70.26</td>
    <td>84.2</td>
    <td>21.04</td>
    <td>74.0</td>
    <td>71.04</td>
    <td>42.53</td>
    <td>57.62</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>45.61</td>
    <td>54.4</td>
    <td>75.6</td>
    <td>87.6</td>
    <td>21.04</td>
    <td>74.67</td>
    <td>74.86</td>
    <td>53.78</td>
    <td>60.95</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>48.25</td>
    <td>52.55</td>
    <td>67.87</td>
    <td>83.4</td>
    <td>21.41</td>
    <td>73.5</td>
    <td>63.93</td>
    <td>33.47</td>
    <td>55.55</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>54.39</td>
    <td>55.09</td>
    <td>78.93</td>
    <td>92.0</td>
    <td>23.95</td>
    <td>77.5</td>
    <td>71.58</td>
    <td>54.03</td>
    <td>63.43</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>50.44</td>
    <td>55.56</td>
    <td>76.99</td>
    <td>90.4</td>
    <td>22.83</td>
    <td>75.33</td>
    <td>68.85</td>
    <td><b>54.45</b></td>
    <td>61.86</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>47.81</td>
    <td>53.94</td>
    <td><b>79.6</b></td>
    <td><b>93.0</b></td>
    <td>27.66</td>
    <td>78.33</td>
    <td>69.95</td>
    <td>48.74</td>
    <td>62.38</td>
  </tr>
  <tr>
    <td colspan="10"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>60.09</td>
    <td>63.43</td>
    <td>61.37</td>
    <td>79.0</td>
    <td>20.67</td>
    <td>63.5</td>
    <td>71.58</td>
    <td>31.29</td>
    <td>56.37</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>63.16</td>
    <td>64.12</td>
    <td>60.92</td>
    <td>82.2</td>
    <td>19.74</td>
    <td>56.67</td>
    <td>66.67</td>
    <td>25.42</td>
    <td>54.86</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>59.21</td>
    <td>62.5</td>
    <td>62.09</td>
    <td>80.0</td>
    <td>18.19</td>
    <td>53.67</td>
    <td>65.57</td>
    <td>27.43</td>
    <td>53.58</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>61.84</td>
    <td>65.51</td>
    <td>64.81</td>
    <td>82.6</td>
    <td>22.77</td>
    <td>70.0</td>
    <td>71.04</td>
    <td>34.98</td>
    <td>59.19</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td><b>66.67</b></td>
    <td><b>68.75</b></td>
    <td>62.03</td>
    <td>80.6</td>
    <td>22.65</td>
    <td>66.0</td>
    <td>74.32</td>
    <td>37.42</td>
    <td>59.8</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>64.04</td>
    <td><b>68.75</b></td>
    <td>68.93</td>
    <td>83.8</td>
    <td>26.92</td>
    <td>70.0</td>
    <td>73.77</td>
    <td>36.33</td>
    <td>61.57</td>
  </tr>
</table>

Table 19: Accuracy (in %) for Uni-Aspect (prompts 1 to 4) and Multi-Aspect (prompts 5 and 6) Encoders on Analogy Tasks with roberta-large

<table>
  <tr>
    <th></th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="10"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>70.18</td>
    <td>73.15</td>
    <td>78.32</td>
    <td>88.4</td>
    <td>21.41</td>
    <td>66.83</td>
    <td>67.21</td>
    <td>32.72</td>
    <td>62.28</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>66.67</td>
    <td>68.52</td>
    <td>82.49</td>
    <td>93.2</td>
    <td>20.48</td>
    <td>62.33</td>
    <td>58.47</td>
    <td>29.78</td>
    <td>60.24</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>64.04</td>
    <td>69.91</td>
    <td>79.99</td>
    <td>90.8</td>
    <td>21.41</td>
    <td>64.5</td>
    <td>63.39</td>
    <td>36.66</td>
    <td>61.34</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>72.81</td>
    <td>74.54</td>
    <td>82.66</td>
    <td>93.6</td>
    <td>20.3</td>
    <td>73.83</td>
    <td>71.04</td>
    <td>40.69</td>
    <td>66.18</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>72.37</td>
    <td>73.15</td>
    <td>82.27</td>
    <td>94.8</td>
    <td>21.47</td>
    <td>68.17</td>
    <td>67.76</td>
    <td>38.42</td>
    <td>64.8</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>71.05</td>
    <td>73.15</td>
    <td>83.77</td>
    <td>94.6</td>
    <td>25.56</td>
    <td>69.83</td>
    <td>65.57</td>
    <td>39.51</td>
    <td>65.38</td>
  </tr>
  <tr>
    <td colspan="10"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>55.26</td>
    <td>59.26</td>
    <td>80.21</td>
    <td>90.2</td>
    <td>18.07</td>
    <td>69.67</td>
    <td>70.49</td>
    <td>43.54</td>
    <td>60.84</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>60.09</td>
    <td>61.11</td>
    <td>80.82</td>
    <td>94.2</td>
    <td>18.19</td>
    <td>74.5</td>
    <td>69.4</td>
    <td>38.76</td>
    <td>62.13</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>56.58</td>
    <td>59.26</td>
    <td>78.77</td>
    <td>91.2</td>
    <td>18.25</td>
    <td>73.5</td>
    <td>65.57</td>
    <td>41.02</td>
    <td>60.52</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>61.84</td>
    <td>60.65</td>
    <td>79.66</td>
    <td>92.8</td>
    <td>19.74</td>
    <td>77.17</td>
    <td>68.85</td>
    <td>47.9</td>
    <td>63.58</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>60.53</td>
    <td>65.05</td>
    <td>79.1</td>
    <td>91.8</td>
    <td>18.63</td>
    <td>75.83</td>
    <td>68.85</td>
    <td>54.7</td>
    <td>64.31</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>61.84</td>
    <td>67.36</td>
    <td>82.32</td>
    <td>93.6</td>
    <td>20.3</td>
    <td>76.33</td>
    <td>70.49</td>
    <td>53.52</td>
    <td>65.72</td>
  </tr>
  <tr>
    <td colspan="10"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>67.98</td>
    <td>69.68</td>
    <td>76.6</td>
    <td>92.0</td>
    <td>19.55</td>
    <td>46.17</td>
    <td>56.83</td>
    <td>29.53</td>
    <td>57.29</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>71.49</td>
    <td>72.22</td>
    <td>80.1</td>
    <td>93.4</td>
    <td>21.29</td>
    <td>56.17</td>
    <td>52.46</td>
    <td>30.54</td>
    <td>59.71</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>67.98</td>
    <td>71.76</td>
    <td>76.43</td>
    <td>90.0</td>
    <td>18.44</td>
    <td>53.33</td>
    <td>53.01</td>
    <td>30.62</td>
    <td>57.7</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>73.68</td>
    <td>75.0</td>
    <td>79.1</td>
    <td>92.0</td>
    <td>20.36</td>
    <td>56.17</td>
    <td>69.95</td>
    <td>37.92</td>
    <td>63.02</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>71.93</td>
    <td>70.6</td>
    <td>78.93</td>
    <td>92.8</td>
    <td>21.35</td>
    <td>62.5</td>
    <td>66.67</td>
    <td>38.34</td>
    <td>62.89</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>70.61</td>
    <td>74.31</td>
    <td>80.77</td>
    <td>92.2</td>
    <td>23.21</td>
    <td>62.67</td>
    <td>64.48</td>
    <td>37.16</td>
    <td>63.18</td>
  </tr>
</table>

Table 20: Accuracy (in %) for Uni-Aspect (prompts 1 to 4) and Multi-Aspect (prompts 5 and 6) Encoders on Analogy Tasks with deberta-large
<table>
  <tr>
    <th></th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="10">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>59.65</td>
    <td>62.27</td>
    <td>63.15</td>
    <td>82.8</td>
    <td>24.94</td>
    <td>74.83</td>
    <td>65.57</td>
    <td>33.98</td>
    <td>58.4</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>58.33</td>
    <td>61.57</td>
    <td>63.59</td>
    <td>85.4</td>
    <td>25.43</td>
    <td>73.83</td>
    <td>59.56</td>
    <td>34.14</td>
    <td>57.73</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>55.7</td>
    <td>60.65</td>
    <td>67.09</td>
    <td>86.0</td>
    <td>24.81</td>
    <td>72.17</td>
    <td>59.56</td>
    <td>31.96</td>
    <td>57.24</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>55.7</td>
    <td>59.72</td>
    <td>64.81</td>
    <td>86.0</td>
    <td>28.03</td>
    <td>77.33</td>
    <td>67.76</td>
    <td>36.91</td>
    <td>59.53</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>59.65</td>
    <td>63.89</td>
    <td>71.98</td>
    <td>92.8</td>
    <td>30.69</td>
    <td>79.67</td>
    <td>66.12</td>
    <td>37.67</td>
    <td>62.81</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>60.53</td>
    <td>62.96</td>
    <td>73.43</td>
    <td>93.8</td>
    <td>32.12</td>
    <td>77.67</td>
    <td>65.57</td>
    <td>39.09</td>
    <td>63.15</td>
  </tr>
  <tr>
    <th colspan="10">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>45.18</td>
    <td>46.3</td>
    <td>70.59</td>
    <td>86.0</td>
    <td>20.73</td>
    <td>74.83</td>
    <td>70.49</td>
    <td>47.23</td>
    <td>57.67</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>45.61</td>
    <td>49.77</td>
    <td>70.48</td>
    <td>86.2</td>
    <td>22.71</td>
    <td>74.67</td>
    <td>62.3</td>
    <td>41.11</td>
    <td>56.61</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>39.04</td>
    <td>46.3</td>
    <td>68.2</td>
    <td>84.0</td>
    <td>21.6</td>
    <td>74.17</td>
    <td>60.11</td>
    <td>39.09</td>
    <td>54.06</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>49.12</td>
    <td>49.77</td>
    <td>70.43</td>
    <td>88.8</td>
    <td>21.84</td>
    <td>77.0</td>
    <td>69.95</td>
    <td>48.32</td>
    <td>59.4</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>47.37</td>
    <td>54.4</td>
    <td>72.04</td>
    <td>88.0</td>
    <td>26.42</td>
    <td>76.17</td>
    <td>66.67</td>
    <td>38.84</td>
    <td>58.74</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>48.68</td>
    <td>53.47</td>
    <td>75.21</td>
    <td>94.2</td>
    <td>24.44</td>
    <td>77.17</td>
    <td>71.04</td>
    <td>49.75</td>
    <td>61.75</td>
  </tr>
  <tr>
    <th colspan="10">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>59.65</td>
    <td>62.04</td>
    <td>59.92</td>
    <td>82.2</td>
    <td>23.82</td>
    <td>67.83</td>
    <td>64.48</td>
    <td>32.05</td>
    <td>56.5</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>53.51</td>
    <td>59.26</td>
    <td>64.54</td>
    <td>85.2</td>
    <td>25.0</td>
    <td>55.83</td>
    <td>57.92</td>
    <td>26.01</td>
    <td>53.41</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>53.95</td>
    <td>56.94</td>
    <td>62.92</td>
    <td>85.2</td>
    <td>24.32</td>
    <td>59.83</td>
    <td>64.48</td>
    <td>29.78</td>
    <td>54.68</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>57.89</td>
    <td>62.73</td>
    <td>63.42</td>
    <td>85.6</td>
    <td>26.05</td>
    <td>72.33</td>
    <td>72.68</td>
    <td>35.15</td>
    <td>59.48</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>59.65</td>
    <td>64.35</td>
    <td>68.59</td>
    <td>91.0</td>
    <td>29.46</td>
    <td>76.83</td>
    <td>66.12</td>
    <td>36.83</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>59.21</td>
    <td>62.27</td>
    <td>66.81</td>
    <td>87.4</td>
    <td>28.28</td>
    <td>71.83</td>
    <td>66.12</td>
    <td>37.08</td>
    <td>59.88</td>
  </tr>
</table>

Table 21: Accuracy (in %) for Uni-Aspect (prompts 1 to 4) and Multi-Aspect (prompts 5 and 6) Encoders on Analogy Tasks with roberta-base

<table>
  <tr>
    <th></th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="10">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>43.42</td>
    <td>53.94</td>
    <td>63.2</td>
    <td>75.4</td>
    <td>26.55</td>
    <td>67.33</td>
    <td>63.93</td>
    <td>28.44</td>
    <td>52.78</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>44.74</td>
    <td>49.31</td>
    <td>58.92</td>
    <td>72.6</td>
    <td>27.48</td>
    <td>62.67</td>
    <td>61.2</td>
    <td>23.66</td>
    <td>50.07</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>47.37</td>
    <td>50.23</td>
    <td>61.37</td>
    <td>74.0</td>
    <td>25.37</td>
    <td>65.33</td>
    <td>57.92</td>
    <td>25.42</td>
    <td>50.88</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>50.0</td>
    <td>53.94</td>
    <td>70.09</td>
    <td>81.8</td>
    <td>28.4</td>
    <td>73.67</td>
    <td>72.68</td>
    <td>34.4</td>
    <td>58.12</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>47.81</td>
    <td>57.87</td>
    <td>74.21</td>
    <td>88.4</td>
    <td>30.2</td>
    <td>74.0</td>
    <td>63.39</td>
    <td>33.47</td>
    <td>58.67</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>45.61</td>
    <td>48.15</td>
    <td>74.1</td>
    <td>87.2</td>
    <td>32.36</td>
    <td>72.67</td>
    <td>68.31</td>
    <td>30.45</td>
    <td>57.36</td>
  </tr>
  <tr>
    <th colspan="10">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>39.91</td>
    <td>39.81</td>
    <td>60.87</td>
    <td>67.6</td>
    <td>17.95</td>
    <td>68.83</td>
    <td>59.56</td>
    <td>20.97</td>
    <td>46.94</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>37.28</td>
    <td>39.58</td>
    <td>62.76</td>
    <td>64.6</td>
    <td>20.85</td>
    <td>69.67</td>
    <td>68.31</td>
    <td>22.73</td>
    <td>48.22</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>39.04</td>
    <td>39.12</td>
    <td>62.76</td>
    <td>71.8</td>
    <td>18.19</td>
    <td>72.0</td>
    <td>64.48</td>
    <td>26.59</td>
    <td>49.25</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>42.54</td>
    <td>42.13</td>
    <td>73.6</td>
    <td>85.8</td>
    <td>23.45</td>
    <td>74.33</td>
    <td>74.32</td>
    <td>40.52</td>
    <td>57.09</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>44.3</td>
    <td>42.59</td>
    <td>74.82</td>
    <td>87.0</td>
    <td>26.49</td>
    <td>75.67</td>
    <td>67.76</td>
    <td>35.23</td>
    <td>56.73</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>45.61</td>
    <td>44.68</td>
    <td>73.87</td>
    <td>88.0</td>
    <td>28.34</td>
    <td>75.83</td>
    <td>72.68</td>
    <td>34.56</td>
    <td>57.95</td>
  </tr>
  <tr>
    <th colspan="10">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>46.05</td>
    <td>53.24</td>
    <td>59.37</td>
    <td>69.0</td>
    <td>24.75</td>
    <td>63.67</td>
    <td>56.83</td>
    <td>25.5</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>42.98</td>
    <td>49.07</td>
    <td>54.25</td>
    <td>63.2</td>
    <td>27.6</td>
    <td>57.33</td>
    <td>50.82</td>
    <td>19.8</td>
    <td>45.63</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>42.54</td>
    <td>50.23</td>
    <td>50.69</td>
    <td>58.6</td>
    <td>23.27</td>
    <td>45.0</td>
    <td>44.81</td>
    <td>15.27</td>
    <td>41.3</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>47.81</td>
    <td>53.7</td>
    <td>66.43</td>
    <td>79.2</td>
    <td>28.84</td>
    <td>65.17</td>
    <td>66.12</td>
    <td>33.39</td>
    <td>55.08</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>49.12</td>
    <td>57.64</td>
    <td>71.76</td>
    <td>81.8</td>
    <td>29.7</td>
    <td>65.83</td>
    <td>63.93</td>
    <td>33.31</td>
    <td>56.64</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>47.81</td>
    <td>56.71</td>
    <td>73.1</td>
    <td>87.4</td>
    <td>32.12</td>
    <td>66.83</td>
    <td>64.48</td>
    <td>31.8</td>
    <td>57.53</td>
  </tr>
</table>

Table 22: Accuracy (in %) for Uni-Aspect (prompts 1 to 4) and Multi-Aspect (prompts 5 and 6) Encoders on Analogy Tasks with bert-base
<table>
  <tr>
    <th></th>
    <th>U2</th>
    <th>U4</th>
    <th>BATS</th>
    <th>GOOGLE</th>
    <th>SCAN</th>
    <th>NELL</th>
    <th>T-REX</th>
    <th>CN</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="10">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td><td>46.93</td><td>51.62</td><td>62.59</td><td>75.8</td><td>21.84</td><td>63.67</td><td>52.46</td><td>20.39</td><td>49.41</td>
  </tr>
  <tr>
    <td>prompt2</td><td>54.82</td><td>56.94</td><td>68.43</td><td>82.0</td><td>27.29</td><td>65.5</td><td>71.58</td><td>30.29</td><td>57.11</td>
  </tr>
  <tr>
    <td>prompt3</td><td>55.7</td><td>59.95</td><td>69.48</td><td>81.2</td><td>27.66</td><td>69.0</td><td>71.58</td><td>30.7</td><td>58.16</td>
  </tr>
  <tr>
    <td>prompt4</td><td>52.19</td><td>58.8</td><td>72.15</td><td>87.0</td><td>31.56</td><td>72.17</td><td>73.77</td><td>35.4</td><td>60.38</td>
  </tr>
  <tr>
    <td>prompt5</td><td>57.46</td><td>60.42</td><td>75.82</td><td>91.4</td><td>31.68</td><td>73.17</td><td>65.03</td><td>35.49</td><td>61.31</td>
  </tr>
  <tr>
    <td>prompt6</td><td>59.65</td><td>62.5</td><td>75.43</td><td>90.8</td><td>29.76</td><td>73.83</td><td>68.85</td><td>35.74</td><td>62.07</td>
  </tr>
  <tr>
    <th colspan="10">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td><td>42.11</td><td>45.14</td><td>67.76</td><td>77.6</td><td>21.97</td><td>70.17</td><td>70.49</td><td>38.51</td><td>54.22</td>
  </tr>
  <tr>
    <td>prompt2</td><td>38.16</td><td>46.06</td><td>67.93</td><td>81.2</td><td>22.22</td><td>69.17</td><td>74.86</td><td>38.93</td><td>54.82</td>
  </tr>
  <tr>
    <td>prompt3</td><td>43.42</td><td>47.22</td><td>64.59</td><td>78.4</td><td>22.34</td><td>69.0</td><td>72.13</td><td>30.37</td><td>53.43</td>
  </tr>
  <tr>
    <td>prompt4</td><td>42.98</td><td>47.22</td><td>73.99</td><td>88.6</td><td>26.24</td><td>75.33</td><td>73.22</td><td>41.95</td><td>58.69</td>
  </tr>
  <tr>
    <td>prompt5</td><td>45.61</td><td>51.39</td><td>78.04</td><td>88.4</td><td>26.73</td><td>75.83</td><td>72.13</td><td>48.07</td><td>60.78</td>
  </tr>
  <tr>
    <td>prompt6</td><td>43.86</td><td>45.37</td><td>72.98</td><td>86.6</td><td>24.63</td><td>75.5</td><td>72.68</td><td>39.6</td><td>57.65</td>
  </tr>
  <tr>
    <th colspan="10">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td><td>55.7</td><td>60.42</td><td>61.7</td><td>74.2</td><td>21.66</td><td>64.67</td><td>61.75</td><td>27.18</td><td>53.41</td>
  </tr>
  <tr>
    <td>prompt2</td><td>56.14</td><td>59.26</td><td>66.43</td><td>79.2</td><td>27.35</td><td>64.67</td><td>55.74</td><td>30.29</td><td>54.89</td>
  </tr>
  <tr>
    <td>prompt3</td><td>47.81</td><td>55.79</td><td>64.2</td><td>76.4</td><td>26.79</td><td>59.67</td><td>56.28</td><td>28.1</td><td>51.88</td>
  </tr>
  <tr>
    <td>prompt4</td><td>53.95</td><td>61.11</td><td>71.76</td><td>87.4</td><td>30.57</td><td>69.17</td><td>77.6</td><td>36.16</td><td>60.97</td>
  </tr>
  <tr>
    <td>prompt5</td><td>59.21</td><td>63.66</td><td>73.15</td><td>89.2</td><td>30.45</td><td>65.5</td><td>69.95</td><td>33.98</td><td>60.64</td>
  </tr>
  <tr>
    <td>prompt6</td><td>58.33</td><td>61.11</td><td>74.71</td><td>90.4</td><td>31.81</td><td>65.67</td><td>69.4</td><td>36.74</td><td>61.02</td>
  </tr>
</table>

Table 23: Accuracy (in %) for Uni-Aspect (prompts 1 to 4) and Multi-Aspect (prompts 5 and 6) Encoders on Analogy Tasks with bert-large

<table>
  <tr>
    <th>Model</th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>RelBERT-base</td>
    <td>76.92</td>
    <td>71.64</td>
    <td>48.85</td>
    <td>85.78</td>
    <td>78.33</td>
    <td>72.30</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>UniPRE-base</td>
    <td>86.54</td>
    <td>73.38</td>
    <td>55.75</td>
    <td>91.04</td>
    <td>84.33</td>
    <td>78.21</td>
  </tr>
  <tr>
    <td>MultiPRE-base</td>
    <td>88.46</td>
    <td>83.58</td>
    <td>64.37</td>
    <td>93.11</td>
    <td>88.33</td>
    <td>83.57</td>
  </tr>
  <tr>
    <td>SemPRE-base</td>
    <td>77.08</td>
    <td>66.17</td>
    <td>48.28</td>
    <td>91.48</td>
    <td>70.67</td>
    <td>70.74</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>UniPRE-base</td>
    <td>84.29</td>
    <td>78.36</td>
    <td>60.92</td>
    <td>91.11</td>
    <td>88.33</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td>MultiPRE-base</td>
    <td>89.26</td>
    <td>82.84</td>
    <td>66.67</td>
    <td>92.81</td>
    <td>90.33</td>
    <td>84.38</td>
  </tr>
  <tr>
    <td>SemPRE-base</td>
    <td>74.84</td>
    <td>67.66</td>
    <td>44.25</td>
    <td>91.78</td>
    <td>70.33</td>
    <td>69.77</td>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>UniPRE-base</td>
    <td>84.62</td>
    <td>77.61</td>
    <td>63.22</td>
    <td>91.04</td>
    <td>88.33</td>
    <td>80.96</td>
  </tr>
  <tr>
    <td>MultiPRE-base</td>
    <td>89.1</td>
    <td>83.58</td>
    <td>66.67</td>
    <td>93.56</td>
    <td>90.33</td>
    <td>84.65</td>
  </tr>
  <tr>
    <td>SemPRE-base</td>
    <td>75.64</td>
    <td>67.66</td>
    <td>43.68</td>
    <td>91.93</td>
    <td>74.67</td>
    <td>70.72</td>
  </tr>
</table>

Table 24: Micro F1-score (in %) on lexical relation classification. For our models, we use DeBERTa and train with RelBERT dataset.

<table>
  <tr>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="6">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>75.64</td>
    <td>67.66</td>
    <td>43.68</td>
    <td>91.93</td>
    <td>74.67</td>
    <td>70.72</td>
  </tr>
  <tr>
    <th colspan="6">RelKB</th>
  </tr>
  <tr>
    <td>77.08</td>
    <td>66.17</td>
    <td>48.28</td>
    <td>91.48</td>
    <td>70.67</td>
    <td>70.74</td>
  </tr>
  <tr>
    <th colspan="6">SemRelLM</th>
  </tr>
  <tr>
    <td>74.84</td>
    <td>67.66</td>
    <td>44.25</td>
    <td>91.78</td>
    <td>70.33</td>
    <td>69.77</td>
  </tr>
</table>

Table 25: Semantic Properties performance on lexical relation classification with deberta-base
<table>
  <tr>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>71.15</td>
    <td>74.38</td>
    <td>43.68</td>
    <td>87.78</td>
    <td>67.67</td>
    <td>68.93</td>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>77.4</td>
    <td>73.88</td>
    <td>44.83</td>
    <td>88.52</td>
    <td>72.0</td>
    <td>71.33</td>
  </tr>
  <tr>
    <td colspan="6"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>71.31</td>
    <td>74.13</td>
    <td>45.4</td>
    <td>87.33</td>
    <td>69.0</td>
    <td>69.43</td>
  </tr>
</table>

Table 26: Semantic Properties performance on lexical relation classification with roberta-large

<table>
  <tr>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>80.13</td>
    <td>72.14</td>
    <td>46.55</td>
    <td>93.26</td>
    <td>74.0</td>
    <td>73.22</td>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>80.93</td>
    <td>69.4</td>
    <td>41.38</td>
    <td>92.74</td>
    <td>69.67</td>
    <td>70.82</td>
  </tr>
  <tr>
    <td colspan="6"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>79.33</td>
    <td>69.9</td>
    <td>47.13</td>
    <td>93.11</td>
    <td>78.0</td>
    <td>73.49</td>
  </tr>
</table>

Table 27: Semantic Properties performance on lexical relation classification with deberta-large

<table>
  <tr>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>68.75</td>
    <td>73.13</td>
    <td>44.25</td>
    <td>82.0</td>
    <td>65.0</td>
    <td>66.63</td>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>58.65</td>
    <td>72.14</td>
    <td>33.91</td>
    <td>81.41</td>
    <td>57.33</td>
    <td>60.69</td>
  </tr>
  <tr>
    <td colspan="6"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>63.46</td>
    <td>72.14</td>
    <td>39.66</td>
    <td>83.78</td>
    <td>63.33</td>
    <td>64.47</td>
  </tr>
</table>

Table 28: Semantic Properties performance on lexical relation classification with roberta-base

<table>
  <tr>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>82.37</td>
    <td>74.88</td>
    <td>48.28</td>
    <td>90.3</td>
    <td>76.33</td>
    <td>74.43</td>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>80.29</td>
    <td>73.63</td>
    <td>53.45</td>
    <td>90.52</td>
    <td>75.33</td>
    <td>74.64</td>
  </tr>
  <tr>
    <td colspan="6"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>79.65</td>
    <td>76.87</td>
    <td>50.57</td>
    <td>90.81</td>
    <td>74.0</td>
    <td>74.38</td>
  </tr>
</table>

Table 29: Semantic Properties performance on lexical relation classification with bert-base

<table>
  <tr>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB+SemRelLM</i></td>
  </tr>
  <tr>
    <td>83.33</td>
    <td>75.37</td>
    <td>58.05</td>
    <td>92.44</td>
    <td>77.0</td>
    <td>77.24</td>
  </tr>
  <tr>
    <td colspan="6"><i>RelKB</i></td>
  </tr>
  <tr>
    <td>82.37</td>
    <td>72.64</td>
    <td>48.28</td>
    <td>93.48</td>
    <td>76.33</td>
    <td>74.62</td>
  </tr>
  <tr>
    <td colspan="6"><i>SemRelLM</i></td>
  </tr>
  <tr>
    <td>83.81</td>
    <td>73.88</td>
    <td>52.3</td>
    <td>92.52</td>
    <td>79.0</td>
    <td>76.3</td>
  </tr>
</table>

Table 30: Semantic Properties performance on lexical relation classification with bert-large
<table>
  <tr>
    <th></th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>84.29</td>
    <td>78.61</td>
    <td>58.62</td>
    <td>90.74</td>
    <td>89.33</td>
    <td>80.32</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.33</td>
    <td>78.11</td>
    <td>55.75</td>
    <td>90.96</td>
    <td>88.0</td>
    <td>79.23</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>84.29</td>
    <td>76.62</td>
    <td>54.6</td>
    <td>91.56</td>
    <td>87.67</td>
    <td>78.95</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>84.62</td>
    <td>77.61</td>
    <td>63.22</td>
    <td>91.04</td>
    <td>88.33</td>
    <td>80.96</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>87.82</td>
    <td>81.84</td>
    <td>66.09</td>
    <td>92.96</td>
    <td>88.67</td>
    <td>83.48</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>89.1</td>
    <td>83.58</td>
    <td>66.67</td>
    <td>93.56</td>
    <td>90.33</td>
    <td>84.65</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>82.53</td>
    <td>72.89</td>
    <td>50.57</td>
    <td>90.96</td>
    <td>86.33</td>
    <td>76.66</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>86.06</td>
    <td>73.13</td>
    <td>52.87</td>
    <td>88.81</td>
    <td>87.33</td>
    <td>77.64</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>85.9</td>
    <td>72.89</td>
    <td>52.3</td>
    <td>90.0</td>
    <td>83.33</td>
    <td>76.88</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>86.54</td>
    <td>73.38</td>
    <td>55.75</td>
    <td>91.04</td>
    <td>84.33</td>
    <td>78.21</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>87.82</td>
    <td>77.61</td>
    <td>64.37</td>
    <td>92.0</td>
    <td>86.0</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>88.46</td>
    <td>83.58</td>
    <td>64.37</td>
    <td>93.11</td>
    <td>88.33</td>
    <td>83.57</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>82.05</td>
    <td>77.61</td>
    <td>54.6</td>
    <td>91.04</td>
    <td>89.33</td>
    <td>78.93</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>82.21</td>
    <td>78.61</td>
    <td>57.47</td>
    <td>90.81</td>
    <td>90.0</td>
    <td>79.82</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>81.57</td>
    <td>76.62</td>
    <td>54.02</td>
    <td>90.96</td>
    <td>88.0</td>
    <td>78.23</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>84.29</td>
    <td>78.36</td>
    <td>60.92</td>
    <td>91.11</td>
    <td>88.33</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>89.42</td>
    <td>78.86</td>
    <td>68.39</td>
    <td>93.11</td>
    <td>88.33</td>
    <td>83.62</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>89.26</td>
    <td>82.84</td>
    <td>66.67</td>
    <td>92.81</td>
    <td>90.33</td>
    <td>84.38</td>
  </tr>
</table>

Table 31: MultiPRE performance on lexical relation classification with deberta-base

<table>
  <tr>
    <th></th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>85.9</td>
    <td>76.37</td>
    <td>63.22</td>
    <td>91.26</td>
    <td>85.67</td>
    <td>80.48</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>86.7</td>
    <td>76.87</td>
    <td>64.94</td>
    <td>91.85</td>
    <td>87.33</td>
    <td>81.54</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>88.3</td>
    <td>75.12</td>
    <td>62.07</td>
    <td>91.93</td>
    <td>87.67</td>
    <td>81.02</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>87.98</td>
    <td>77.11</td>
    <td>64.37</td>
    <td>91.7</td>
    <td>88.33</td>
    <td>81.9</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>85.74</td>
    <td>75.87</td>
    <td>64.94</td>
    <td>92.22</td>
    <td>83.67</td>
    <td>80.49</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>88.46</td>
    <td>79.35</td>
    <td>68.39</td>
    <td>93.63</td>
    <td>89.33</td>
    <td>83.83</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>80.93</td>
    <td>72.89</td>
    <td>56.32</td>
    <td>89.85</td>
    <td>84.67</td>
    <td>76.93</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.33</td>
    <td>73.13</td>
    <td>54.6</td>
    <td>89.7</td>
    <td>84.33</td>
    <td>77.02</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>84.78</td>
    <td>72.89</td>
    <td>50.57</td>
    <td>89.04</td>
    <td>83.0</td>
    <td>76.06</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>85.26</td>
    <td>76.12</td>
    <td>63.22</td>
    <td>89.41</td>
    <td>83.67</td>
    <td>79.54</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>86.22</td>
    <td>75.37</td>
    <td>65.52</td>
    <td>91.04</td>
    <td>85.0</td>
    <td>80.63</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>88.14</td>
    <td>79.1</td>
    <td>63.22</td>
    <td>92.67</td>
    <td>86.67</td>
    <td>81.96</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>84.13</td>
    <td>76.87</td>
    <td>62.64</td>
    <td>91.33</td>
    <td>84.0</td>
    <td>79.79</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>82.69</td>
    <td>76.62</td>
    <td>59.2</td>
    <td>90.15</td>
    <td>81.67</td>
    <td>78.07</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>82.53</td>
    <td>76.37</td>
    <td>60.92</td>
    <td>90.44</td>
    <td>83.0</td>
    <td>78.65</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>84.94</td>
    <td>75.87</td>
    <td>59.2</td>
    <td>91.33</td>
    <td>83.67</td>
    <td>79.0</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>84.62</td>
    <td>75.37</td>
    <td>64.37</td>
    <td>90.59</td>
    <td>83.67</td>
    <td>79.72</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>87.34</td>
    <td>79.35</td>
    <td>64.37</td>
    <td>92.74</td>
    <td>84.67</td>
    <td>81.69</td>
  </tr>
</table>

Table 32: MultiPRE performance on lexical relation classification with roberta-large
<table>
  <tr>
    <th></th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>89.58</td>
    <td>79.85</td>
    <td>68.97</td>
    <td>93.78</td>
    <td>87.67</td>
    <td>83.97</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>91.35</td>
    <td>80.85</td>
    <td>70.69</td>
    <td>93.85</td>
    <td>90.33</td>
    <td>85.41</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>88.62</td>
    <td>80.85</td>
    <td>66.67</td>
    <td>93.78</td>
    <td>90.0</td>
    <td>83.98</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>88.94</td>
    <td>82.09</td>
    <td>63.79</td>
    <td>93.26</td>
    <td>89.0</td>
    <td>83.42</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>91.67</td>
    <td>86.32</td>
    <td>67.24</td>
    <td>93.56</td>
    <td>87.33</td>
    <td>85.22</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>91.67</td>
    <td>90.8</td>
    <td>67.82</td>
    <td>94.74</td>
    <td>91.0</td>
    <td>87.21</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>90.54</td>
    <td>75.87</td>
    <td>63.79</td>
    <td>93.26</td>
    <td>86.33</td>
    <td>81.96</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>91.35</td>
    <td>75.37</td>
    <td>58.05</td>
    <td>93.26</td>
    <td>89.33</td>
    <td>81.47</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>90.38</td>
    <td>76.12</td>
    <td>64.94</td>
    <td>93.48</td>
    <td>88.0</td>
    <td>82.58</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>88.14</td>
    <td>78.86</td>
    <td>67.24</td>
    <td>93.48</td>
    <td>83.67</td>
    <td>82.28</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>89.42</td>
    <td>82.09</td>
    <td>65.52</td>
    <td>93.85</td>
    <td>86.33</td>
    <td>83.44</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>90.54</td>
    <td>86.82</td>
    <td>64.94</td>
    <td>94.37</td>
    <td>83.0</td>
    <td>83.93</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>88.94</td>
    <td>80.35</td>
    <td>63.22</td>
    <td>93.7</td>
    <td>89.67</td>
    <td>83.18</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>89.26</td>
    <td>81.09</td>
    <td>63.79</td>
    <td>93.7</td>
    <td>91.67</td>
    <td>83.9</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>88.3</td>
    <td>80.6</td>
    <td>65.52</td>
    <td>92.81</td>
    <td>90.33</td>
    <td>83.51</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>88.46</td>
    <td>78.86</td>
    <td>64.94</td>
    <td>93.41</td>
    <td>89.33</td>
    <td>83.0</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>91.51</td>
    <td>83.33</td>
    <td>66.09</td>
    <td>94.52</td>
    <td>87.33</td>
    <td>84.56</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>92.63</td>
    <td>85.07</td>
    <td>67.24</td>
    <td>94.37</td>
    <td>89.67</td>
    <td>85.8</td>
  </tr>
</table>

Table 33: MultiPRE performance on lexical relation classification with deberta-large

<table>
  <tr>
    <th></th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>85.42</td>
    <td>77.11</td>
    <td>62.07</td>
    <td>90.15</td>
    <td>83.33</td>
    <td>79.62</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>86.06</td>
    <td>77.11</td>
    <td>59.2</td>
    <td>89.11</td>
    <td>85.67</td>
    <td>79.43</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>85.42</td>
    <td>78.86</td>
    <td>61.49</td>
    <td>89.78</td>
    <td>86.33</td>
    <td>80.38</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>84.29</td>
    <td>77.61</td>
    <td>61.49</td>
    <td>89.85</td>
    <td>82.0</td>
    <td>79.05</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>86.86</td>
    <td>80.6</td>
    <td>64.37</td>
    <td>90.74</td>
    <td>87.0</td>
    <td>81.91</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>88.14</td>
    <td>82.34</td>
    <td>61.49</td>
    <td>92.07</td>
    <td>84.0</td>
    <td>81.61</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>81.73</td>
    <td>74.38</td>
    <td>58.05</td>
    <td>86.96</td>
    <td>81.67</td>
    <td>76.56</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.81</td>
    <td>74.13</td>
    <td>56.9</td>
    <td>88.07</td>
    <td>84.33</td>
    <td>77.45</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>82.85</td>
    <td>72.89</td>
    <td>54.6</td>
    <td>87.63</td>
    <td>83.67</td>
    <td>76.33</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>82.85</td>
    <td>74.88</td>
    <td>54.6</td>
    <td>87.26</td>
    <td>81.0</td>
    <td>76.12</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>86.06</td>
    <td>77.11</td>
    <td>61.49</td>
    <td>89.93</td>
    <td>84.67</td>
    <td>79.85</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>87.66</td>
    <td>81.84</td>
    <td>62.64</td>
    <td>91.11</td>
    <td>86.0</td>
    <td>81.85</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>85.58</td>
    <td>76.37</td>
    <td>62.64</td>
    <td>89.41</td>
    <td>82.67</td>
    <td>79.33</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>85.1</td>
    <td>79.1</td>
    <td>56.9</td>
    <td>89.85</td>
    <td>83.0</td>
    <td>78.79</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>85.42</td>
    <td>79.6</td>
    <td>58.62</td>
    <td>89.85</td>
    <td>84.0</td>
    <td>79.5</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>85.1</td>
    <td>77.11</td>
    <td>62.64</td>
    <td>88.96</td>
    <td>85.33</td>
    <td>79.83</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>88.14</td>
    <td>80.35</td>
    <td>60.92</td>
    <td>90.67</td>
    <td>85.0</td>
    <td>81.02</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>87.18</td>
    <td>80.35</td>
    <td>63.22</td>
    <td>90.81</td>
    <td>86.0</td>
    <td>81.51</td>
  </tr>
</table>

Table 34: MultiPRE performance on lexical relation classification with roberta-base
<table>
  <tr>
    <th></th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>85.58</td>
    <td>75.37</td>
    <td>58.05</td>
    <td>89.85</td>
    <td>82.0</td>
    <td>78.17</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.81</td>
    <td>75.12</td>
    <td>51.72</td>
    <td>90.59</td>
    <td>82.33</td>
    <td>76.71</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>84.13</td>
    <td>75.87</td>
    <td>56.32</td>
    <td>90.67</td>
    <td>81.67</td>
    <td>77.73</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>83.65</td>
    <td>75.37</td>
    <td>56.32</td>
    <td>90.67</td>
    <td>82.67</td>
    <td>77.74</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>86.7</td>
    <td>78.36</td>
    <td>58.05</td>
    <td>90.59</td>
    <td>85.67</td>
    <td>79.87</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>87.66</td>
    <td>78.11</td>
    <td>59.77</td>
    <td>91.85</td>
    <td>82.67</td>
    <td>80.01</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>81.73</td>
    <td>72.89</td>
    <td>43.1</td>
    <td>87.04</td>
    <td>78.67</td>
    <td>72.69</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.17</td>
    <td>72.89</td>
    <td>47.13</td>
    <td>89.48</td>
    <td>83.33</td>
    <td>75.2</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>83.33</td>
    <td>72.89</td>
    <td>48.28</td>
    <td>88.0</td>
    <td>81.67</td>
    <td>74.83</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>82.21</td>
    <td>73.38</td>
    <td>52.3</td>
    <td>88.37</td>
    <td>79.0</td>
    <td>75.05</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>83.81</td>
    <td>76.87</td>
    <td>57.47</td>
    <td>90.37</td>
    <td>82.67</td>
    <td>78.24</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>85.9</td>
    <td>77.61</td>
    <td>59.77</td>
    <td>91.41</td>
    <td>83.33</td>
    <td>79.6</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>84.46</td>
    <td>74.38</td>
    <td>54.6</td>
    <td>89.26</td>
    <td>80.67</td>
    <td>76.67</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.17</td>
    <td>75.37</td>
    <td>51.15</td>
    <td>90.59</td>
    <td>80.0</td>
    <td>76.06</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>78.53</td>
    <td>73.38</td>
    <td>51.15</td>
    <td>89.11</td>
    <td>79.0</td>
    <td>74.23</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>85.1</td>
    <td>75.37</td>
    <td>58.62</td>
    <td>90.3</td>
    <td>81.67</td>
    <td>78.21</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>85.58</td>
    <td>76.87</td>
    <td>58.62</td>
    <td>90.81</td>
    <td>85.33</td>
    <td>79.44</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>87.66</td>
    <td>79.35</td>
    <td>60.92</td>
    <td>91.48</td>
    <td>83.33</td>
    <td>80.55</td>
  </tr>
</table>

Table 35: MultiPRE performance on lexical relation classification with bert-base

<table>
  <tr>
    <th></th>
    <th>BLESS</th>
    <th>CogALexV</th>
    <th>EVALution</th>
    <th>K&amp;H+N</th>
    <th>ROOT09</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">RelKB+SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>82.85</td>
    <td>76.37</td>
    <td>54.02</td>
    <td>91.48</td>
    <td>83.67</td>
    <td>77.68</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>85.26</td>
    <td>76.62</td>
    <td>54.6</td>
    <td>91.85</td>
    <td>84.0</td>
    <td>78.47</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>87.18</td>
    <td>76.12</td>
    <td>56.9</td>
    <td>90.89</td>
    <td>84.33</td>
    <td>79.08</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>86.86</td>
    <td>77.36</td>
    <td>63.79</td>
    <td>92.67</td>
    <td>82.33</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>87.5</td>
    <td>79.6</td>
    <td>62.64</td>
    <td>92.52</td>
    <td>84.67</td>
    <td>81.39</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>87.66</td>
    <td>78.36</td>
    <td>66.09</td>
    <td>92.59</td>
    <td>83.67</td>
    <td>81.67</td>
  </tr>
  <tr>
    <th colspan="7">RelKB</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>83.81</td>
    <td>73.63</td>
    <td>55.17</td>
    <td>89.93</td>
    <td>81.67</td>
    <td>76.84</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>84.62</td>
    <td>73.13</td>
    <td>53.45</td>
    <td>90.0</td>
    <td>82.0</td>
    <td>76.64</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>84.29</td>
    <td>72.89</td>
    <td>55.75</td>
    <td>88.81</td>
    <td>81.67</td>
    <td>76.68</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>83.97</td>
    <td>73.88</td>
    <td>58.05</td>
    <td>90.89</td>
    <td>80.33</td>
    <td>77.42</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>86.54</td>
    <td>77.11</td>
    <td>62.64</td>
    <td>91.11</td>
    <td>83.67</td>
    <td>80.21</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>85.9</td>
    <td>77.36</td>
    <td>59.77</td>
    <td>92.59</td>
    <td>81.67</td>
    <td>79.46</td>
  </tr>
  <tr>
    <th colspan="7">SemRelLM</th>
  </tr>
  <tr>
    <td>prompt1</td>
    <td>84.13</td>
    <td>76.37</td>
    <td>56.9</td>
    <td>90.59</td>
    <td>82.0</td>
    <td>78.0</td>
  </tr>
  <tr>
    <td>prompt2</td>
    <td>83.33</td>
    <td>77.36</td>
    <td>54.02</td>
    <td>91.7</td>
    <td>84.0</td>
    <td>78.08</td>
  </tr>
  <tr>
    <td>prompt3</td>
    <td>85.58</td>
    <td>76.12</td>
    <td>56.32</td>
    <td>91.93</td>
    <td>82.67</td>
    <td>78.52</td>
  </tr>
  <tr>
    <td>prompt4</td>
    <td>85.58</td>
    <td>76.37</td>
    <td>58.05</td>
    <td>92.74</td>
    <td>81.33</td>
    <td>78.81</td>
  </tr>
  <tr>
    <td>prompt5</td>
    <td>87.18</td>
    <td>78.11</td>
    <td>60.34</td>
    <td>92.89</td>
    <td>86.0</td>
    <td>80.9</td>
  </tr>
  <tr>
    <td>prompt6</td>
    <td>88.62</td>
    <td>79.35</td>
    <td>60.92</td>
    <td>92.67</td>
    <td>84.0</td>
    <td>81.11</td>
  </tr>
</table>

Table 36: MultiPRE performance on lexical relation classification with bert-large

Metareview score: 4
Metareview: Paper proposes new relational encoders to capture diverse and nuanced semantic properties of entity pairs. It introduces a method for automatically generating diverse relationship datasets using LLMs. Experiments show that both the proposed encoders and the generated dataset improve performance for tasks requiring relational information.
Summary of reasons to publish: - Paper introduces three relational encoders to capture different semantic aspects of entity-entity relations. The proposed encoders seem to outperform the baselines in a few relevant tasks. 
- Paper presents an automatically generated dataset that covers a larger set of relations
- Evaluation is sound and comprehensive, particularly when zooming into more nuanced relations.