Venue: NAACL 2025

# Conformalized Answer Set Prediction for Knowledge Graph Embedding

Yuqicheng Zhu\({}^{\dagger\ddagger}\), Nico Potyka\({}^{\lx@sectionsign}\), Jiarong Pan\({}^{\dagger*}\), Bo Xiong\({}^{\diamond}\)

**Yunjie He\({}^{\dagger\ddagger}\), Evgeny Kharlamov\({}^{\flat}\), Steffen Staab\({}^{\dagger\natural}\)**

\({}^{\dagger}\)University of Stuttgart, \({}^{\ddagger}\)Bosch Center for AI, \({}^{\lx@sectionsign}\)Cardiff University, \({}^{\flat}\) University of Oslo,

\({}^{\natural}\) University of Southampton, \({}^{*}\) Eindhoven University, \(\diamond\) Stanford University

yuqicheng.zhu@de.bosch.com

###### Abstract

Knowledge graph embeddings (KGE) apply machine learning methods on knowledge graphs (KGs) to provide non-classical reasoning capabilities based on similarities and analogies. The learned KG embeddings are typically used to answer queries by ranking all potential answers, but rankings often lack a meaningful probabilistic interpretation - lower-ranked answers do not necessarily have a lower probability of being true. This limitation makes it difficult to quantify uncertainty of model's predictions, posing challenges for the application of KGE methods in high-stakes domains like medicine. We address this issue by applying the theory of conformal prediction that allows generating answer sets, which contain the correct answer with probabilistic guarantees. We explain how conformal prediction can be used to generate such answer sets for link prediction tasks. Our empirical evaluation on four benchmark datasets using six representative KGE methods validates that the generated answer sets satisfy the probabilistic guarantees given by the theory of conformal prediction. We also demonstrate that the generated answer sets often have a sensible size and that the size adapts well with respect to the difficulty of the query.

## 1 Introduction

Knowledge Graph Embeddings (KGE) map entities and predicates into numerical vectors, providing non-classical reasoning capabilities by exploiting similarities and analogies between entities and relations Wang et al. (2017); Biswas et al. (2023). KGE models are typically evaluated through link prediction Bordes et al. (2013); Sun et al. (2019); Nickel et al. (2011). To answer queries in the form of \(\langle head~{}entity,predicate,?\rangle\) or \(\langle?,predicate,tail~{}entity\rangle\), all possible entities are ranked according to their plausibility scores returned by the KGE models. The higher the correct answer is ranked, the more precise the model's prediction is for that particular query.

However, high-ranked entities does not necessarily correlate with high likelihood of correctness Tabacof and Costabello (2020); Safavi et al. (2020). For instance, in a medical diagnosis setting, ranking "common cold" above "cancer X" may imply various scenarios: (1) the common cold is highly likely while cancer is improbable, (2) both conditions are likely but the common cold is more probable, or (3) neither is likely, but the common cold is relatively more probable. These interpretations can lead to vastly different treatments, making rankings alone insufficient for medical decisions. It is especially important for the patient to know when cancer can be confidently excluded from consideration.

Although uncertainty quantification is crucial for high-stakes applications like in medicine domain, where reliable predictions and risk assessment are essential, methods for quantifying uncertainty in KGE model predictions remain underexplored. Existing approaches often reply on off-the-shelf calibration techniques, such as Platt scaling Platt et al. (1999) and Isotonic regression Kruskal (1964), to map uncalibrated plausibility scores to the expected correctness of predictions Tabacof and Costabello (2020); Safavi et al. (2020). However, perfect calibration is impossible in practice Gupta et al. (2020), both Platt scaling and Isotonic regression are empirical calibration techniques that lack formal probabilistic guarantees and are highly sensitive to the calibration set.

In contrast, our work diverges from these approaches by applying the theory of conformal prediction Vovk et al. (2005) to quantify uncertainty with formal statistical guarantees. Conformal prediction assigns a score to each candidate answer entity and defines a threshold to construct answer sets that include the truth answer with a desired confidence level. The size of the answer set reflects

the uncertainty of the model's output for a given query. To the best of our knowledge, this is the first method that does not merely convert plausibility scores into probabilities but instead ensures statistical validity in the uncertainty quantification of the predictions within the context of KGE.

In our paper, we carefully design conformal predictors tailored to the link prediction task such that the answer sets are (1) probabilistically guaranteed to include the true answer entity at a specified confidence level, (2) tight, and (3) adaptive, providing smaller sets for easier queries than harder ones. We perform extensive experiments on commonly used benchmark datasets and a variety of KGE methods. Our empirical results show that: (1) conformal predictors satisfy the statistical guarantees in Proposition 1 and produce tighter answer sets compared to other baselines (Experiments 1); (2) conformal predictors generate answer sets that adapt to query difficulty, yielding smaller sets for easier queries than for harder ones (Experiment 2); (3) high-quality answer sets can be obtained with a relatively small calibration set (Experiment 3); and (4) conformal predictors are effective across different user-specified error rates (Experiment 4).

## 2 Related Work

As highlighted in recent work Zhu et al. (2024), predictions from KGE models exhibit substantial uncertainty, even among models with similar overall performance. This phenomenon, termed predictive multiplicity, arises from the highly non-convex loss functions employed by KGE methods. It leads to multiple models that capture distinct patterns from the training KG and generalize differently. Consequently, quantifying the uncertainty of KGE methods is critical; however, this remains a relatively underexplored area of research.

Existing approaches incorporate uncertainty into KGE by modeling entities and relations using probability distributions He et al. (2015); Xiao et al. (2016). However, these methods primarily focus on enhancing the performance of KGE models through more expressive representations, without systematically analyzing or rigorously evaluating the quality of uncertainty in embeddings or predictions.

Furthermore, research by Tabacof and Costabello (2020) and Safavi et al. (2020) applies off-the-shelf calibration techniques, such as Platt scaling and Isotonic regression, to KGE methods. These techniques aim to convert uncalibrated plausibility scores into probabilities by minimizing the negative log-likelihood on a validation set. However, these approaches are quite sensitive to the validation set and do not provide formal guarantees about the generated probabilities.

This paper applies conformal prediction, which has its roots in online learning literature, is a method that produces predictive sets ensuring coverage guarantees Vovk et al. (2005). This approach has been successfully applied across various domains, including image classification Angelopoulos et al. (2021), natural language processing Maltoudoglou et al. (2020) and node classification/regression on graphs Huang et al. (2024); Zargarbashi et al. (2023); Zagarbashi and Bojchevski (2023). However, to the best of our knowledge, it has not yet been applied to KGE.

## 3 Preliminaries

### Knowledge Graph Embedding

We consider a knowledge graph (KG) \({\cal G}\subseteq E\times R\times E\) defined over a set \(E\) of entities and a set \(R\) of relation names. The elements in \({\cal G}\) are called triples and denoted as \(<h,r,t>\). A KGE model \(M_{\theta}:E\times R\times E\rightarrow\mathbb{R}\) associates each triple with a score that measures the plausibility that the triple holds. The parameters \(\theta\) are learned to let \(M_{\theta}\) assign higher plausibility scores to positive triples (real facts) while assigning lower plausibility scores to negative triples (false facts).

Note that the interpretation of plausibility scores varies across different types of KGE methods. In _distance-based models_ like TransE Bordes et al. (2013) and RotatE Sun et al. (2019), the plausibility score is determined by the negative distance in the embedding space. In _semantic matching models_ such as RESCAL Nickel et al. (2011) and DistMult Yang et al. (2015), plausibility scores are derived from similarity measures.

### Conformal Prediction

Conformal prediction (a.k.a conformal inference) is a general framework for producing answer sets that cover the ground truth with probabilistic guarantees Vovk et al. (2005). In this section, we recall some basics from Vovk et al. (2005); Shafer and Vovk (2008).

Let \((x_{i},y_{i})\) denote a data point with an object \(x_{i}\) and its label \(y_{i}\). The objects are elements of an _object space_\({\cal X}\), and the labels are elements of 

a _label space_\({\cal Y}\). For a more compact notation, we write \(z_{i}\) for \((x_{i},y_{i})\), and call \({\cal Z}:={\cal X}\times{\cal Y}\) the _example space_. Furthermore, we let \(Z_{1:n}=\{z_{1},\ldots z_{n}\}\subseteq{\cal Z}\) be the set of \(n\) examples and denote \({\cal Z}^{*}\) as the set of all possible example sets.

A conformal predictor \(\Gamma:{\cal Z}^{*}\times{\cal X}\to 2^{\cal Y}\) aims to predict a subset of \({\cal Y}\) large enough to cover the ground truth with high probability. Given a training set \(Z_{1:n}\) and any new object \(x_{n+1}\in{\cal X}\), the conformal predictor \(\Gamma\) should, for every probability of error \(\epsilon\in(0,1)\), produce a answer set \(\Gamma^{\epsilon}(Z_{1:n},x_{n+1})\) for the input object \(x_{n+1}\) that contains the ground truth label \(y_{n+1}\) with probability at least \(1-\epsilon\). Moreover, the answer sets are required to shrink as \(\epsilon\) increases: \(\Gamma^{\epsilon_{1}}\subseteq\Gamma^{\epsilon_{2}}\) whenever \(\epsilon_{1}\geq\epsilon_{2}\).

To specify such a conformal predictor, we first need to define a _nonconformity measure_\(S:{\cal Z}^{*}\times{\cal Z}\rightarrow\overline{\mathbb{R}}\). \(S(Z_{1:n},z_{n+1})\) measures how unusual the example \(z_{n+1}\) is as an element of \(Z_{1:n}\). Given any such a nonconformity measure \(S\), if we construct the answer set \(\Gamma^{\epsilon}(Z_{1:n},x_{n+1})\) by including all \(y\in{\cal Y}\) such that

\[\frac{|\{i=1,\ldots,n+1:\alpha_{i}\geq\alpha_{n+1}\}|}{n+1}>\epsilon, \tag{1}\]

where

\[\begin{split}&\alpha_{i}:=S(Z_{1:n}\cup\{(x_{n+1},y)\},(x_{i}, y_{i})),i=1,\ldots,n\\ &\alpha_{n+1}:=S(Z_{1:n}\cup\{(x_{n+1},y)\},(x_{n+1},y)),\end{split}\]

then we have following probabilistic guarantees:

**Theorem 1** (Vovk et al. (2005); Lei et al. (2018)).: _Suppose \(n\) is large, and a set of examples \(Z_{1:n+1}\) are independent and identically distributed (i.i.d.). Given \(\epsilon\in(0,1)\), the answer set of the object \(x_{n+1}\) constructed by a conformal predictor \(\Gamma^{\epsilon}(Z_{1:n},x_{n+1})\) cover the ground truth \(y_{n+1}\) with a probability of at least \(1-\epsilon\)_

\[\mathbb{P}(y_{n+1}\in\Gamma^{\epsilon}(Z_{1:n},x_{n+1}))\geq 1-\epsilon \tag{2}\]

_furthermore, if there are no ties between \(\alpha_{i}\), then it is also holds that_

\[\mathbb{P}(y_{n+1}\in\Gamma^{\epsilon}(Z_{1:n},x_{n+1}))\leq 1-\epsilon+\frac{1 }{n+1} \tag{3}\]

The proof of Equation 2 is provided in Vovk et al. (2005), section 2.1.3). Intuitively, the construction of \(\Gamma^{\epsilon}(Z_{1:n},x_{n+1})\) can be understood as an application of the widely accepted Neyman-Pearson theory Lehmann et al. (1986) for hypothesis testing and confidence intervals Shafer and Vovk (2008). Here, we test for all \(y\in{\cal Y}\) that the hypothesis \(H\) (the example \((x_{n+1},y)\) conforms to \(Z_{1:n}\)) by evaluating the nonconformity score of \((x_{n+1},y)\). We construct the answer set by including all \(y\), for which \((x_{n+1},y)\) is not rejected by the test.

Additionally, the proof of Equation 3 is detailed in Lei et al. (2018, Appendix A.1). Notably, the theorem remains valid under the weaker assumption of _exchangeability_Vovk et al. (2005), section 2.1.1).

## 4 KGE-based Answer Set Prediction

In this section, we formally define the KGE-based answer set prediction task and outline three key desiderata guiding the development of effective set predictors. We then introduce and discuss several basic set predictors.

### Problem Definition and Desiderata

We reformulate the link prediction task as an answer set prediction task. Instead of object-label pairs \((x_{i},y_{i})\) in section 3, each data point is a triple \(tr(q_{i},e_{i})\). Here, \(q_{i}\) denotes a query in form of either \(\langle h,r,?\rangle\) or \(\langle?,r,t\rangle\), and \(tr(q,e)\) corresponds to the respective triple \(\langle h,r,e\rangle\) or \(\langle e,r,t\rangle\).

Given a set of (training) triples \({\cal T}_{1:n}=\{tr(q_{1},e_{1}),\ldots,tr(q_{n},e_{n})\}\), a test query \(q_{n+1}\) and a user-specific error rate \(\epsilon\), we aim to predict a set of entities \(\hat{C}(q_{n+1})\subseteq E\) that covers the true answer \(e_{n+1}\) with probability at least \(1-\epsilon\).

\[\mathbb{P}(e_{n+1}\in\hat{C}(q_{n+1}))\geq 1-\epsilon \tag{4}\]

We refer to Equation 4 as the _coverage desideratum_. However, this criterion alone is insufficient, as it can be trivially met by a predictor that always outputs sets containing all possible answers. To develop sensible set predictors, we also consider the _size desideratum_ and the _adaptiveness desideratum_. The size desideratum emphasizes the need for smaller sets, as smaller sets are generally more informative. The adaptiveness desideratum requires that the set sizes reflect query difficulty: smaller sets should correspond to easier queries, while larger sets should be used for harder queries.

### Basic Set Predictors

**Naive Predictor**. Given a query, assume the KGE model provides the probability of each possible answer entity being true. A straightforward approach towards our goal is to construct the set by 

including entities from highest to lowest probability until their sum exceeds the threshold \(1-\epsilon\). We refer to this approach as the _naive_ predictor and provide its pseudocode in Algorithm 2 (Appendix A). However, the plausibility scores provided by KGE models are not calibrated. We convert these plausibility scores into a "probability distribution" using a softmax function.

**Platt Predictor**. Following the recommendations of Tabacof and Costabello (2020); Safavi et al. (2020), we improve the naive predictor by using a multiclass Platt scaling Guo et al. (2017) to calibrate the plausibility scores and then construct sets based on these calibrated probabilities. We refer to this method as the _Platt_ predictor and provide more details of this predictor in Appendix A.2.

**TopK Predictor**. Another straightforward approach is to construct the set with the Top-K entities from the ranking, referred to as the _topk_ predictor. We select \(K\) to ensure the Top-K entities cover the correct answers for \(1-\epsilon\) of the validation queries.

## 5 Conformal Prediction for KGE-based Answer Set Prediction

To improve the basic set predictors, we apply conformal prediction, a general framework that requires adaptation to be effective in the context of KGE. The two essential components in this design are the nonconformity measure and the method for constructing answer sets. In this section, we propose several KGE-specific nonconformity measures and outline an efficient approach to constructing answer sets.

### Nonconformity Measures

The probabilistic guarantees in Theorem 1 hold under i.i.d assumption, regardless of the data distribution or the definition of the nonconformity measure. However, the size of the resulting answer sets depends on how effectively the nonconformity measure captures the underlying structure of the data. Next, we introduce several nonconformity measures for KGE models and explain the rationale behind each one.

Formally, given a set of training triples \({\cal T}_{1:n}\) and a test triple \(t_{n+1}:=tr(q_{n+1},e_{n+1})\), the nonconformity measure \(S({\cal T}_{1:n},t_{n+1})\) estimates how unusual the triple \(t_{n+1}\) is as a part of \({\cal T}_{1:n}\).

**NegScore**. The underlying idea of KGE methods is to assign higher plausibility scores to positive triples and lower scores to negative triples. Therefore, a natural choice for the nonconformity score is the negative value of the plausibility score. The intuition here is that a lower plausibility score indicates a higher nonconformity, suggesting that the triple is less consistent with the existing triples represented in the training set. Formally, let \(M_{{\cal T}_{1:n}}\) denote a KGE model trained on \({\cal T}_{1:n}\), then the corresponding nonconformity measure is defined as

\[S({\cal T}_{1:n},t_{n+1})=-M_{{\cal T}_{1:n}}(t_{n+1}) \tag{5}\]

**Minmax**. While the NegScore predictor directly uses the raw plausibility scores, the scale of these scores can vary significantly across different queries, potentially affecting the consistency and reliability of the nonconformity measure. To address this, we normalize the scores for each query using min-max scaling. This ensures that the nonconformity score reflects not only the raw plausibility but also the relative position of the triple within the score distribution for all possible triples in a given query. We then define the nonconformity measure as

\[S({\cal T}_{1:n},t_{n+1})=-\overline{M}_{{\cal T}_{1:n}}(t_{n+1}), \tag{6}\]

where

\[\begin{split}&\overline{M}(tr(q,e))=\\ &\frac{M(tr(q,e))-\min_{e^{\prime}\in E}M(tr(q,e^{\prime}))}{ \max_{e^{\prime}\in E}M(tr(q,e^{\prime}))-\min_{e^{\prime}\in E}M(tr(q,e^{ \prime}))}.\end{split} \tag{7}\]

**Softmax**. Another approach to normalizing plausibility scores is by using the softmax function, which converts the plausibility scores into a (uncalibrated) "probability distribution" over all possible answers for a given query. Unlike min-max scaling, Softmax scaling is more sensitive to the relative differences between scores, naturally highlighting the most likely triples while acknowledging others. This can result in more nuanced nonconformity measures, especially when the score distribution has varying degrees of separation between true and false triples. The nonconformity score is then defined as the of softmax outputs and the "ground truth" probability, which is assumed to be \(1\) for the true answer.

\[S({\cal T}_{1:n},t_{n+1})=1-\hat{M}_{{\cal T}_{1:n}}(t_{n+1}), \tag{8}\]

where

\[\hat{M}(tr(q,e))=\frac{\exp(M(tr(q,e)))}{\sum_{e^{\prime}\in E}\exp{(M(q,e^{ \prime}))}}.\] (9) 



### Answer Set Construction

If we construct answer sets as describe in Section 3, we need to retrain the KGE model with \({\cal T}_{1:n}\cup\left\{tr(q_{n+1},e)\right\}\) and recalculate the nonconformity scores of training triples for testing each triple \(tr(q_{n+1},e)\) (for all \(e\in E\)). It is computationally not feasible for KGE methods, given the huge number of possible entities \(e\in E\) and the time-consuming training and hyper-parameter tuning process.

We adopt so called split/inductive conformal prediction [25, 4] to address this issue (see Algorithm 1 for details). The training set of size \(n\) is first divided into a proper training set \({\cal T}_{1:m}\) of size \(m<n\) and a calibration set \({\cal T}_{m+1:n}\) of size \(n-m\). Rather than using the entire training set to train the KGE model and evaluate nonconformity scores, we train the KGE model once on the proper training set \({\cal T}_{1:m}\) and use it to calculate the nonconformity scores on the calibration set \({\cal T}_{m+1:n}\). Intuitively, if the calibration set is chosen randomly and is sufficiently large, its empirical coverage should closely match the true coverage probability for a new query. This strategy significantly increases the efficiency of the conformal predictors while preserving the probabilistic guarantees in Theorem 1 [4].

Formally, in split conformal prediction, if we construct answer sets by including all entity \(e\in E\) such that

\[\frac{|\{i=m+1,\ldots,n+1:\alpha_{i}\geq\alpha_{n+1}\}|+1}{n-m+1}>\epsilon, \tag{11}\]

where

\[\begin{split}&\alpha_{i}:=S({\cal T}_{1:m},tr(q_{i},e_{i})),i= m+1,\ldots,n\\ &\alpha_{n+1}:=S({\cal T}_{1:m},tr(q_{n+1},e)).\end{split}\]

Based on Theorem 2.2 in Lei et al. (2018), we have the following corollary:

**Corollary 1**.: _Given a set of triples \({\cal T}_{1:n+1}\) that are i.i.d, an error rate \(\epsilon\in(0,1)\) and any nonconformity measure \(S\). If \(n\) is large, the answer set of a test query \(\hat{C}(q_{n+1})\) constructed following Equation 11 satisfies_

\[\mathbb{P}(e_{n+1}\in\hat{C}(q_{n+1}))\geq 1-\epsilon \tag{12}\]

_furthermore, if there are no ties between nonconformity scores in the calibration set \({\cal T}_{m+1:n}\), we have_

\[\mathbb{P}(e_{n+1}\in\hat{C}(q_{n+1}))\leq 1-\epsilon+\frac{1}{n-m+1} \tag{13}\]

The proof of Proposition 1 can be found in [4, Appendix A.1]1. Note that the additional assumption for Equation 13 is a quite weak assumption, by using a random tie-breaking rule, this assumption could be avoided entirely.

Footnote 1: Unlike [4], we split the training set unevenly, resulting in slight differences in Equation 13.

```
0:A training set \({\cal T}_{1:m}\), a calibration set \({\cal T}_{m+1:n}\), a testing query \(q_{n+1}\), an error rate \(\epsilon\) and a nonconformity measure \(S\).
1:
2:\(\triangleright\) Calibration Step
3:\(L^{\prime}\leftarrow\) an empty set
4:foreach triple \(tr(q^{\prime},e^{\prime})\)in \({\cal T}_{m+1:n}\)do
5:\(L^{\prime}\gets L^{\prime}\cup\left\{S({\cal T}_{1:m},tr(q^{\prime},e^{ \prime}))\right\}\)
6:endfor
7:\(t\leftarrow\lceil(|{\cal T}_{m+1:n}|+1)(1-\epsilon)/|{\cal T}_{m+1:n}|\rceil\)
8:\(\tau\gets t\) th quantile of elements in \(L^{\prime}\)
9:
10:\(\triangleright\) Prediction Step
11:\(L\leftarrow\) an empty set
12:foreach entity \(e\)in \(E\)do
13:\(L\gets L\cup\left\{(e,S({\cal T}_{1:m},tr(q_{n+1},e))\right\}\)
14:endfor
15:\(\hat{C}(q_{n+1})\leftarrow\) an empty set
16:for\((e,s)\)in \(L\)do
17:if\(s<\tau\)then
18:\(\hat{C}(q_{n+1})\leftarrow\hat{C}(q_{n+1})\cup\left\{e\right\}\)
19:endif
20:endfor
21:return\(\hat{C}(q_{n+1})\)
```

**Algorithm 1** Pseudocode for Split Conformal Prediction.

### Time Complexity Analysis

The use of split conformal prediction in Section 5.2 significantly reduces the computational effort by leveraging a pre-trained KGE model, eliminating the need to train a KGE model for each \(tr(q_{test},e)\) to recompute nonconformity scores. This approach ensures that the scalability of our method is independent of the training graph size. In this section, we analyze the time complexity of the proposed method.

Let \(T\) represent the time required to compute nonconformity scores. For a calibration set \({\cal T}m+1:n\) of size \(n-m\) and a test set \({\cal T}1:j\) of size \(j\), the time complexity of split conformal

prediction is:

\[{\cal O}\bigg{(}(n-m+j)T+(n-m)\log(n-m)\\ +j\log(n-m)\bigg{)}. \tag{14}\]

Here, the term \((n-m)\log(n-m)\) accounts for sorting the nonconformity scores in the calibration set, while \(\log(n-m)\) for finding the rank of a test nonconformity score. Since \(T\) scales linearly with the number of entities \(|E|\), the complexity can be expressed as:

\[{\cal O}\bigg{(}(n-m+j)|E|+(n-m)\log(n-m)\\ +j\log(n-m)\bigg{)}. \tag{15}\]

As \(n-m\) and \(j\) are independent of \(|E|\) and typically small, the asymptotic complexity for large \(|E|\) becomes \({\cal O}(|E|)\). This result highlights the efficiency and scalability of the method for larger KGs.

## 6 Experiments

In this section, we present five experiments that evaluate the quality of the answer sets from (baseline) predictors in Section 4 (_naive_, _Platt_, _topk_) and conformal predictors (_NegScore_, _Softmax_, _Minmax_) in Section 5 based on coverage, size and adaptiveness desiderata.

**Datasets and Backbone KGE Methods.** In our experiments, we use four commonly used benchmark link prediction datasets: WN18 Bordes et al. (2013), WN18RR Dettmers et al. (2018), FB15k Bordes et al. (2013) and FB15k237 Toutanova and Chen (2015) and six representative KGE methods: TransE Bordes et al. (2013), RotatE Sun et al. (2019), RESCAL Nickel et al. (2011), DistMult Yang et al. (2015), ComplEx Trouillon et al. (2016) and ConvE Dettmers et al. (2018). We provide more information about the experimental settings in Appendix B.

**Evaluation Metrics.** A good set predictor should cover the true answer with a probability of at least \(1-\epsilon\) (coverage desideratum) and provide smaller sets (size desideratum). Let \(\mathcal{T}_{1:j}\) be a set of triples disjoint with \(\mathcal{T}_{1:n}\) as test set. We measure the following _empirical coverage probability_ to verify the coverage desideratum, i.e., \(\mathbb{P}(e_{n+1}\in\hat{C}(q_{n+1}))\geq 1-\epsilon\):

\[\frac{1}{|\mathcal{T}_{1:j}|}\sum_{tr(q,e)\in\mathcal{T}_{1:j}} \mathbbm{1}[e\in\hat{C}(q)] \tag{16}\]

We measure the _average size_ of the answer sets as following:

\[\frac{1}{|\mathcal{T}_{1:j}|}\sum_{tr(q,e)\in\mathcal{T}_{1:j}} \lvert\hat{C}(q)\rvert \tag{17}\]

### Experiment 1: Coverage and Set Size on WN18 and FB15k

In this experiment, we evaluate desiderata by measuring the empirical coverage probability (\(coverage\)) and the average size (\(size\)) of answer sets for each method. Each procedure is repeated 15 times, and we report the mean and standard deviation (in brackets) across trials in Table 1. As usual in the evaluation of link prediction, for each query, we consider only answer candidates that did not already occur in the training and validation data.

As demonstrated in Proposition 1, conformal predictors consistently meet the coverage desideratum, with coverage tightly concentrated around \(1-\epsilon\). Compared to baseline predictors that also satisfy the coverage desideratum, conformal predictors outperform them in terms of producing smaller answer sets, thus better satisfying the size desideratum.

The naive predictor, on the other hand, frequently fails to meet the coverage desideratum, often providing lower coverage than necessary, indicating that the plausibility scores are generally overconfident. While applying a calibration technique to the naive predictor (Platt predictor) improves coverage, it still does not meet the coverage guarantee, and the resulting significant increase in set size makes it impractical for use.

The topk predictor meets the coverage desideratum but generally produces larger and fixed-sized answer sets compared to the conformal predictors. In Appendix A.3, we also discuss simpler fixed-sized predictors and compare them to the topk predictor. It is worth noting that the topk predictor can be viewed as a specific case of the conformal predictor, where the nonconformity score is defined by the rank position.

Additionally, we observed that there is no universally optimal nonconformity score for conformal predictors; the choice is model- and dataset-dependent. For instance, NegScore seems to better 

capture the nonconformity of triples in distance-based models (TransE, RotatE), while Softmax and Minmax scores are more suitable for semantic matching models (RESCAL, DistMult, and ComplEx).

The calibration technique in Tabacof and Costabello (2020); Safavi et al. (2020) should theoretically enhance the design of the nonconformity measure and thereby improve the conformal predictor. However, in our setting, it fails to do so. The results are presented in Table 7, with a discussion of potential reasons provided in the Appendix C.

Due to space constraints, additional results, including those without filtering existing answers and results from more datasets, are provided in Appendix D. The conclusions are consistent across all scenarios.

### Experiment 2: Adaptiveness of Answer Sets

This experiment aims to determine whether the size of answer sets adapts well to the difficulty of the query. Unfortunately, there is no well justified way to evaluate query difficulty at the moment. We therefore follow the experimental protocol used in Angelopoulos et al. (2021) for computer vision tasks. The authors evaluate difficulty by looking at the rank of the true label in the ranking obtained from the classifier by ordering labels according to their softmax-probabilities. The higher the rank, the more difficult the query. Analogously, we use the rank of the true answer given by the KGE model to evaluate query difficulty.

We categorize queries by difficulty levels based on the rank of the true answer (e.g., 1-100, 101-200, etc.). For each difficulty level, we calculate

\begin{table}
\begin{tabular}{c c r l|l l l l l l} \hline \hline  & \multicolumn{3}{c}{WN18} & \multicolumn{3}{c}{FB15k} \\ model & MR & methods & coverage & size & model & MR & methods & coverage & size \\ \hline \multirow{6}{*}{TransE} & \multirow{4}{*}{245.82 (6.368)} & naive & 0.44 (0.004) & 12.28 (1.262) & & & naive & 0.73 (0.001) & 258.14 (0.834) \\  & & Platt & **0.85** (0.022) & 4034.41 (89.765) & & & & Platt & 0.84 (0.002) & 1197.64 (45.923) \\  & & topic & 0.90 (0.000) & 48.01 (0.739) & & & topk & 0.90 (0.000) & 336.68 (1.332) \\  & & NegScore & 0.90 (0.001) & **20.99 (0.587)** & & & NegScore & 0.90 (0.001) & **45.18 (0.280)** \\  & & Softmax & 0.90 (0.001) & 112.80 (4.650) & & & Softmax & 0.90 (0.000) & 418.48 (2.390) \\  & & Minimax & 0.90 (0.001) & 113.57 (5.098) & & & Minimax & 0.90 (0.000) & 275.27 (3.217) \\ \hline \multirow{6}{*}{RotatE} & \multirow{4}{*}{478.13 (44.173)} & naive & 0.91 (0.003) & 17690.17 (117.856) & & & naive & 0.71 (0.003) & 748.09 (5.216) \\  & & Platt & 0.90 (0.002) & 16950.17 (116.477) & & & Platt & 0.88 (0.002) & 1156.33 (7.552) \\  & & topic & 0.90 (0.001) & 50.85 (2.440) & & & topk & 0.90 (0.000) & 406.43 (3.752) \\  & & NegScore & 0.90 (0.002) & **1.27 (0.040)** & & & SizeGroot & 0.90 (0.001) & **52.31 (H.605)** \\  & & Softmax & 0.90 (0.001) & 1.91 (0.249) & & & Softmax & 0.90 (0.000) & 140.36 (3.196) \\  & & Minimax & 0.90 (0.003) & 3.88 (0.698) & & & Minimax & 0.90 (0.001) & 42.35 (1.064) \\ \hline \multirow{6}{*}{RESCAL} & \multirow{4}{*}{321.73 (21.501)} & naive & 0.58 (0.011) & 300.09 (24.737) & & & naive & 0.58 (0.016) & 121.14 (13.125) \\  & & Platt & 0.80 (0.005) & 2012.25 (23.393) & & & Platt & 0.87 (0.003) & 615.92 (24.759) \\  & & topk & 0.90 (0.001) & 54.46 (1.640) & & & topk & 0.90 (0.000) & 394.18 (2.357) \\  & &NegScore & 0.91 (0.001) & 45.50 (3.630) & & & NegScore & 0.90 (0.001) & 168.64 (4.506) \\  & & Softmax & 0.90 (0.001) & 2.14 (0.062) & & & Softmax & 0.90 (0.000) & **72.61 (0.491)** \\  & & Minimax & 0.90 (0.002) & **2.02 (0.075)** & & & Minimax & 0.90 (0.001) & 79.62 (5.573) \\ \hline \multirow{6}{*}{DistMult} & \multirow{4}{*}{370.21 (20.313)} & naive & 0.47 (0.002) & 36.54 (7.024) & & & naive & 0.35 (0.013) & 19.47 (0.897) \\  & & Phat & 0.84 (0.001) & 1265.82 (20.375) & & & Plant & 0.90 (0.001) & 4851.01 (2.341) \\  & & topk & 0.90 (0.001) & 57.48 (2.439) & & & logk & 0.90 (0.000) & 362.91 (1.414) \\  & & NegScore & 0.90 (0.001) & 2244.87 (405.033) & & & NegScore & 0.90 (0.000) & 156.02 (2.872) \\  & & Softmax & 0.90 (0.001) & 2.02 (0.047) & & & Softmax & 0.90 (0.000) & **28.40 (0.421)** \\  & & Minimax & 0.90 (0.002) & **1.51 (0.049)** & & & Minimax & 0.90 (0.000) & 40.35 (0.894) \\ \hline \multirow{6}{*}{ComplEx} & \multirow{4}{*}{454.21 (27.914)} & naive & 0.94 (0.007) & 19968.55 (142.153) & & & naive & 0.28 (0.011) & 502.04 (1.309) \\  & & Platt & 0.86 (0.002) & 1678.28 (125.356) & & & Platt & 0.90 (0.001) & 592.43 (3.375) \\  & & topk & 0.91 (0.002) & 40.58 (1.811) & & & topk & 0.90 (0.000) & 414.06 (4.429) \\  & & NegScore & 0.90 (0.003) & 1.47 (0.008) & & & SegCorre & 0.90 (0.001) & 991.92 (2.37) \\  & & Softmax & 0.90 (0.001) & **1.10 (0.007)** & & & Softmax & 0.90 (0.001) & **37.20 (0.927)** \\  & & Minimax & 0.90 (0.002) & 2.83 (0.171) & & & Minimax & 0.90 (0.001) & 118.43 (20.418) \\ \hline \multirow{6}{*}{ConvE} & \multirow{4}{*}{311.27 (12.598)} & naive & 0.50 (0.015) & 3.42 (1.423) & & & naive & 0.45 (0.026) & 819.05 (111.870) \\  & & Platt & 0.84 (0.007) & 79.09 (1.785) & & & & Platt & 0.88 (0.014) & 4824.86 (69.23) \\ \cline{1-1}  & & topk & 0.90 (0.001) & 53.21 (1.475) & & & topk & 0.90 (0.000) & 386.02 (2.317) \\ \cline{1-1}  & & NegScore & 0.90 (0.002) & 1.50 (0.067) & & & & NegScore & 0.90 (0.000) & **44.39 (2.225)** \\ \cline{1-1}  & & Softmax & 0.9 

the average size of answer sets. Figure 1 illustrates the size of answer sets stratified by query difficulty. The x-axis represents rank intervals from \(1\) to \(3000\), segmented into \(100\)-rank bins (reflecting different difficulty levels), while the y-axis shows the average size of answer sets within each interval.

We observe that the size of answer sets generated by conformal predictors closely aligns with the difficulty levels of the queries, thereby fulfilling the adaptiveness desideratum. This is a valuable property because, in practice, the true answer to a query is unknown. By examining the size of the answer set, we can estimate the predictive uncertainty for the query.

### Experiment 3: Impact of Calibration Set Size on Answer Set Quality

In this experiment, we investigate the impact of size of the calibration set, \({\cal T}_{m+1:n}\), on the quality of answer sets in terms of coverage and size desiderata. We randomly sampled calibration sets of \(10\), \(100\), \(200\), and \(500\) triples from the validation set for use in conformal prediction. We then evaluated the coverage and average size of the resulting answer sets. This process was repeated \(20\) times to compute the mean and standard deviation of the results. For comparison, we also evaluated the answer sets generated using the entire validation set as the calibration set.

As shown in Proposition 1, the coverage of conformal predictors with an i.i.d calibration set should fall between \(1-\epsilon\) and \(1-\epsilon+\frac{1}{n-m}\), where \(n-m\) is the size of the calibration set. This is confirmed by the results in Table 2. The size of answer sets generated by split conformal predictors depends on the alignment between the distribution of nonconformity scores in the calibration set and those in the original training set (which includes both the proper training set and the calibration set). A larger calibration set typically better represents the original training set, leading to tighter answer sets, as confirmed by the results in Table 2. Notably, even with a relatively small calibration set, the quality of the answer sets closely approximates that obtained using the entire validation set.

### Experiment 4: Impact of Error Rate on Answer Set Quality

In this experiment, we examine the effect of the user-specified error rate (\(\epsilon\)) on the quality of answer sets. Figure 2 illustrates how \(\epsilon\) influences the size of answer sets (upper diagram) and the coverage of answer sets (lower diagram) across various predictors. The red line in lower diagram correspond to the desired coverage \(1-\epsilon\).

As expected, the size of answer sets decreases as \(\epsilon\) increases, aligning with the requirements discussed in Section 3. The topk and conformal predictors consistently generate smaller answer sets

Figure 1: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on RESCAL models) on the FB15k237 dataset, more results can be found in Figure 3 - 14 in Appendix.

 

compared to the naive and Platt predictors. Notably, conformal predictors produce the most compact answer sets when the error rate is set to a very low value. In terms of coverage, conformal predictors consistently meet the probabilistic guarantee in Proposition 1 across the range of values.

## Discussion

Our method predicts answer sets for queries with a guaranteed coverage of the true answer at a pre-specified probability, such as , while maintaining a small average size. Unlike ranking-based outputs, our approach is particularly well-suited for decision-making in high-stakes domains, including medical diagnosis, drug discovery, and fraud detection. For instance, a doctor could use our method to automatically eliminate a large number of irrelevant diseases, thereby referring the patient to the most appropriate specialists. Additionally, our method is easy to implement and is compatible not only with any KGE models but also with embedding methods capable of answering more complex queries Ren et al. (2020); He et al. (2024a,b) and approximate statistical reasoning in ontology Zhu et al. (2023, 2024b).

The adaptability of our answer sets to the uncertainty of queries also enables our method to quantify the predictive uncertainty of KGE models. This feature broadens the applicability of our approach by systematically identifying hard or uncertain queries during testing. Detecting such queries can help identify potential failure cases or outliers, alerting users when the model's predictions may be unreliable.

## Limitations

A limitation of our method is the requirement to divide the training set into two parts: one for training the model and another for calculating the nonconformity scores, due to the adoption of split conformal prediction. This division reduces the number of triples available for model training. However, this issue is mitigated by the fact that the validation set, typically reserved for hyperparameter tuning, can also serve as the calibration set. Moreover, as demonstrated in Experiment 4, even a small subset of the validation set is sufficient to produce nearly optimal answer sets.

Another limitation is that the probabilistic guarantee provided by Theorem 1 and Proposition 1 relies on the i.i.d. assumption, which may not hold under distribution shifts. We are currently extending our conformal predictors to covariant shift case, where only the input distribution changes while the conditional distribution remains the same. We begin with the simpler scenario where the likelihood ratio between the training and test distributions is known. Following Tibshirani et al. (2019), we weight each nonconformity score proportionally to the likelihood ratio to ensure the probabilistic guarantee in Proposition 1 holds beyond the i.i.d. assumption.

## Acknowledgements

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRSIS) for supporting Yuqicheng Zhu and Yunjie He. The work was partially supported by EU Projects Graph Massivizer (GA 101093202), Dome 4.0 (GA 953163), enRichMyData (GA 101070284) and SMARTY (GA 101140087).

Figure 2: This figure shows how the coverage and size of answer sets change with respect to across different predictors on the WN18 dataset.

 



## References

* N. Angelopoulos, S. Bates, M. I. Jordan, and J. Malik (2021)Uncertainty sets for image classifiers using conformal prediction.. In ICLR, Cited by: SS3.4.
* J. Bergstra and Y. Bengio (2012)Random search for hyper-parameter optimization. Journal of machine learning research13 (2). External Links: Link Cited by: SS5.
* R. Biswas, L. Kaffee, M. Cochez, S. Dumphrava, T. E. Jendal, M. Lissandrini, V. Lopez, E. Loza Mencia, H. Paulheim, H. Sack, et al. (2023)Knowledge graph embeddings: open challenges and opportunities. Transactions on Graph Data and Knowledge1 (1), pp. 4-1. External Links: ISSN 0978-695X, Document, Link Cited by: SS1, SS2.
* A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko (2013)Translating embeddings for modeling multi-relational data. Advances in neural information processing systems26. Cited by: SS1, SS2.
* a knowledge graph embedding library for reproducible research. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 165-174. Cited by: SS1, SS2.
* T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel (2018)Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: SS1, SS2.
* G. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017)On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. Cited by: SS1, SS2.
* C. Gupta, A. Podkopaev, and A. Ramdas (2020)Distribution-free binary classification: prediction sets, confidence intervals and calibration. Advances in Neural Information Processing Systems33, pp. 3711-3723. Cited by: SS1, SS2.
* S. He, K. Liu, G. Ji, and J. Zhao (2015)Learning to represent knowledge graphs with gaussian embedding. In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 623-632. Cited by: SS1, SS2.
* 4286. Cited by: SS1, SS2.
* Y. He, B. Xiong, D. Hernandez, Y. Zhu, E. Kharlamov, and S. Staab (2024)Dage: dag query answering via relational combinator with logical constraints. In Proceedings of the ACM Web Conference 2025, Cited by: SS1, SS2.
* K. Huang, Y. Jin, E. Candes, and J. Leskovec (2024)Uncertainty quantification over graph with conormalized graph neural networks. Advances in Neural Information Processing Systems36. Cited by: SS1, SS2.
* J. B Kruskal (1964)Nonmetric multidimensional scaling: a numerical method. Psychometrika29 (2), pp. 115-129. Cited by: SS1, SS2.
* E. Leo Lehmann, J. P. Romano, and G. Casella (1986)Testing statistical hypotheses, volume 3. Springer. Cited by: SS1, SS2.
* J. Lei, M. G'Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman (2018)Distribution-free predictive inference for regression. Journal of the American Statistical Association113 (523), pp. 1094-1111. Cited by: SS1, SS2.
* J. Lei, A. Rinaldo, and L. Wasserman (2015)A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence74, pp. 29-43. Cited by: SS1, SS2.
* L. Maltoudoglou, A. Paisios, and H. Papadopoulos (2020)Bert-based conformal predictor for sentiment analysis. In Conformal and Probabilistic Prediction and Applications, pp. 269-284. Cited by: SS1, SS2.
* M. Nickel, V. Tresp, and H. Kriegel (2011)A three-way model for collective learning on multi-relational data. In ICML, pp. 809-816. Cited by: SS1, SS2.
* OpenAI (2024)Chatgpt(3.5)[large language model]. Note: [https://chat.openai.com](https://chat.openai.com) Cited by: SS1, SS2.
* John Platt et al. (1999)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers10 (3), pp. 61-74. Cited by: SS1, SS2.
* H. Ren, W. Hu, and J. Leskovec (2020)Query2box: reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations (ICLR), Cited by: SS1, SS2.
* D. Ruffinelli, S. Broscheit, and R. Gemulla (2019)You can teach an old dog new tricks! on training knowledge graph embeddings. In International Conference on Learning Representations, Cited by: SS1, SS2.
* T. Safavi, D. Koutra, and E. Meij (2020)Evaluating the calibration of knowledge graph embeddings for trustworthy link prediction. In EMNLP (1), pp. 8308-8321. Cited by: SS1, SS2.
* Glenn Shafer and Vladimir Vovk (2008)A tutorial on conformal prediction. Journal of Machine Learning Research9 (3). Cited by: SS1, SS2.
* Z. Sun, Z. Deng, J. Nie, and J. Tang (2019)Rotate: knowledge graph embedding by relational rotation in complex space. In ICLR (Poster), pp. . Cited by: SS1, SS2.



* Tabacof and Costabello (2020) Pedro Tabacof and Luca Costabello. 2020. Probability calibration for knowledge graph embedding models. In _ICLR_. OpenReview.net.
* Tibshirani et al. (2019) Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. 2019. Conformal prediction under covariate shift. _Advances in neural information processing systems_, 32.
* Toutanova and Chen (2015) Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In _Proceedings of the 3rd workshop on continuous vector space models and their compositionality_, pages 57-66.
* Trouillon et al. (2016) Theo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In _International conference on machine learning_, pages 2071-2080. PMLR.
* Vovk et al. (2005) Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. 2005. _Algorithmic learning in a random world_, volume 29. Springer.
* Wang et al. (2017) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. _IEEE transactions on knowledge and data engineering_, 29(12):2724-2743.
* Xiao et al. (2016) Han Xiao, Minlie Huang, and Xiaoyan Zhu. 2016. TransG : A generative model for knowledge graph embedding. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2316-2325, Berlin, Germany. Association for Computational Linguistics.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In _ICLR (Poster)_.
* Zagarbashi et al. (2023) Soroush H Zagarbashi, Simone Antonelli, and Aleksandar Bojchevski. 2023. Conformal prediction sets for graph neural networks. In _International Conference on Machine Learning_, pages 12292-12318. PMLR.
* Zargarbashi and Bojchevski (2023) Soroush H Zargarbashi and Aleksandar Bojchevski. 2023. Conformal inductive graph neural networks. In _The Twelfth International Conference on Learning Representations_.
* Zhu et al. (2024a) Yuqicheng Zhu, Nico Potyka, Mojtaba Nayyeri, Bo Xiong, Yunjie He, Evgeny Kharlamov, and Steffen Staab. 2024a. Predictive multiplicity of knowledge graph embeddings in link prediction. In _Findings of the Association for Computational Linguistics: EMNLP 2024_, pages 334-354.
* Zhu et al. (2024b) Yuqicheng Zhu, Nico Potyka, Bo Xiong, Trung-Kien Tran, Mojtaba Nayyeri, Evgeny Kharlamov, and Steffen Staab. 2024b. Approximating probabilistic inference in statistical EL with knowledge graph embeddings. _CoRR_, abs/2407.11821.
* Zhu et al. (2023) Yuqicheng Zhu, Nico Potyka, Bo Xiong, Trung-Kien Tran, Mojtaba Nayyeri, Steffen Staab, and Evgeny Kharlamov. 2023. Towards statistical reasoning with ontology embeddings. In _ISWC (Posters/Demos/Industry)_, volume 3632 of _CEUR Workshop Proceedings_. CEUR-WS.org 

Baseline Predictors

### Naive Predictor

We provide detailed pseudocode for naive predictor in this section. See Algorithm 2 for details.

```
0: A KGE model \(M_{\theta}\) trained on \(\mathcal{T}_{1:n}\), a testing query \(q_{n+1}\) and an error rate \(\epsilon\).
1:\(L\leftarrow\) an empty set
2:for each entity \(e\)in \(E\)do
3:\(L\gets L\cup\left\{(e,M_{\theta}(tr(q_{n+1},e)))\right\}\)
4:endfor
5: normalize the scores in \(L\) with softmax function.
6:\(\overline{L}\leftarrow\) sort elements in \(L\) based on normalized scores (from largest to smallest).
7:\(p\gets 0\)
8:\(\hat{C}(q_{n+1})\leftarrow\) an empty set
9:for\((e,s)\)in \(\overline{L}\)do
10:\(p\gets p+s\)
11:if\(p<1-\epsilon\)then
12:\(\hat{C}(q_{n+1})\leftarrow\hat{C}(q_{n+1})\cup\{e\}\)
13:endif
14:endfor
15:return\(\hat{C}(q_{n+1})\)
```

**Algorithm 2** Pseudocode for naive predictor.

### Platt Predictor

The Platt predictor enhances the naive predictor using a calibration technique. The only difference in its procedure, as outlined in line 5 of Algorithm 2, is the modification of softmax outputs through temperature scalingGuo et al. (2017) -- a multiclass extension of Platt scaling Platt et al. (1999).

Temperature scaling employs a single scalar parameter \(T>0\) across all possible answer entities for a given query. Let \(M_{\theta}(tr(q_{n+1}),e_{i})\) represent the plausibility score of entity \(e_{i}\in E\) for query \(q_{n+1}\). The calibrated score \(\hat{s}_{i}\) is then calculated as

\[\hat{s}_{i}=\sigma(M_{\theta}(tr(q_{n+1}),e_{i})/T), \tag{18}\]

where \(\sigma(\cdot)\) is the softmax function.

The parameter \(T\), known as the temperature, "softens" the softmax output by increasing its entropy when \(T>1\). As \(T\rightarrow\infty\), the probability \(\hat{s}_{i}\) approaches \(1/|E|\), indicating maximum uncertainty. When \(T=1\), the original softmax output is recovered. Conversely, as \(T\to 0\), the probability collapses to a point mass (\(\hat{s}_{i}=1\)). The optimal value of \(T\) is determined by minimizing the negative log-likelihood on the validation set.

\[NLL=-\frac{1}{N}\sum_{i=1}^{N}\log(\frac{\hat{s}_{i}}{\sum_{j=1}^{N}\hat{s}_{ j}}), \tag{19}\]

where \(N\) is the size of calibration (validation) set.

### Fixed-sized Predictor

The ranking-based metric \(Hits@K\) evaluate how often KGE models place the correct answers within the top-K entities, implicitly suggesting that the top-K entities should be chosen as answer sets. Based on \(Hits@K\), we evaluate the quality of _fixed-sized_ set predictor, which produce top-K entities (with a manually chosen \(K\)) as the answer set.

We select \(K\) values commonly used in Hits@K metrics (1, 3, 10, 100). The results in Table 3 and 4 demonstrate that coverage is highly sensitive to the choice of \(K\). Concretely, the fixed-sized set predictor either fails to meet the coverage desideratum or generate unnecessarily large answer sets.

Consequently, in the main body of the paper, we adopt the topk predictor, where we use \(K\) that cover the true answer in \(1-\epsilon\) of queries in the validation set. The topk predictor effectively balances the trade-off between coverage and average size. However, unlike conformal predictors, the topk predictor cannot adapt answer set sizes to the difficulty of individual queries, as it uses the same size for all queries.

## Appendix B Detailed Experimental Setting

### Information About KGE Models and Benchmark Datasets

We provide the statistics of the benchmark datasets in Table 5 and the scoring functions of KGE methods in Table 6.

\begin{table}
\begin{tabular}{l c c|c c c} \hline \hline  & WN18 & & \multicolumn{3}{c}{FB15k} \\ method & coverage & size & method & coverage & size \\ \hline top1 & 0.45 & 0.48 & top1 & 0.12 & 0.17 \\ top3 & 0.65 & 1.76 & top3 & 0.24 & 0.71 \\ top10 & 0.80 & 7.32 & top10 & 0.46 & 3.91 \\ top100 & 0.92 & 90.91 & top100 & 0.79 & 71.46 \\ topk & 0.90 & 52.96 & topk & 0.90 & 395.11 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of the fixed-size set predictors and the top-K predictor with \(K\) learned based on the validation set. The table reports the mean values of coverage and the average size of filtered answer sets over 15 trials. Results are based on the RESCAL model applied to the WN18 and FB15k datasets.

 



### Personal Identification Issue in FB15k and FB15k237

While FB15k and FB15k237 contain information about individuals, it typically focuses on well-known public figures such as celebrities, politicians, and historical figures. Since this information is already widely available online and in various public sources, its inclusion in Freebase doesn't significantly compromise individual privacy compared to datasets containing sensitive personal information.

### Details of Pre-training KGE Models

For training KGE models, we use the implementation of LibKGE (Broscheit et al., 2020) and basically follow the hyperparameter search strategy in (Ruffinelli et al., 2019). All experiments were conducted on a Linux machine with a 40GB NVIDIA A100 SXM4 GPU.

We first conduct quasi-random hyperparameter search via a Sobol sequence, which aims to distribute hyperparameter settings evenly to avoid "clumping" effects (Bergstra and Bengio, 2012). More specifically, for each dataset and model, we generated 30 different configurations per valid combination of training type and loss function. we added a short Bayesian optimization phase (best configuration so far + 30 new trials) to tune the hyperparameters further. All above steps are conducted using Ax framework ([https://ax.dev/](https://ax.dev/))

We use a large hyperparameter space including loss functions (pairwise margin ranking with hinge loss, binary cross entropy, cross entropy), regularization techniques (none/L1/L2/L3, dropout), optimizers (Adam, Adagrad), and initialization methods used in the KGE community as hyperparameters. We consider 128, 256, 512 as possible embedding sizes. More details see in (Ruffinelli et al., 2019, Table 5).

The hyperparameters of the baseline models are located within the software folder we submitted. Concretely, all configuration files (*.yaml) that we use for training baseline models/competing models/models for aggregation can be found in folder "configs".

## Appendix C Calibrated Conformal Predictor

Conformal prediction is a theoretical framework that quantifies predictive uncertainty by ensuring answer sets meet probabilistic guarantees, followed by identifying a nonconformity measure that minimizes the size of these sets. Optimal answer sets are achieved when nonconformity scores accurately reflect the confidence of the predictions.

While calibrating plausibility scores from KGE models should theoretically improve the naive predictor and the nonconformity measure for conformal predictor, our results suggest otherwise. As shown in Table 1, 8, 9 and 10, Platt predictors yield excessively large answer sets. Further experiments comparing softmax conformal prediction before and after temperature scaling (Table 7) reveal that temperature scaling generally increases answer set sizes. Although smaller sets are observed for TransE, they are still not competitive with the best predictors for TransE in Table 1. These observations contradict our expectations. We next explore the reasons for these outcomes in KGE models.

First, as detailed in Appendix A.2, temperature

\begin{table}
\begin{tabular}{l c|c} \hline \hline  & Scoring Function \(s(<h,r,t>)\) of KGE \\ \hline TransE (Bordes et al., 2013) & \(-||\mathbf{h}+\mathbf{r}-\boldsymbol{t}||_{1/2}\) \\ RotatE (Sun et al., 2019) & \(-||\mathbf{h}\circ\mathbf{r}-\boldsymbol{t}||_{g}\) \\ RESCAL (Nickel et al., 2011) & \(\mathbf{h}^{T}\mathbf{M}_{e}\mathbf{t}\) \\ DistMM (Yang et al., 2015) & \(\mathbf{h}^{T}diag(q)\mathbf{t}\) \\ ComplEx (Trouillon et al., 2016) & \(Re(\mathbf{h}^{T}diag(\tilde{v})\bar{\mathbf{t}})\) \\ ConvE (Dettmers et al., 2018) & \(f(vec(f[(\hat{\mathbf{h}};\mathbf{i})\star\omega))\mathbf{W}\mathbf{t}\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: The scoring function \(s(<h,r,t>)\) of KGE models used in this paper, where \(h,r,t\) denote the embeddings of \(h,r,t\), \(\diamond\) denotes Hadamard product. \(\widehat{\cdot}\) refers to conjugate for complex vectors in ComplEx, and 2D reshaping for real vectors in ConveE. \(*\) is operator for 2D convolution. \(\omega\) is the filters and \(W\) is the parameters for 2D convolutional layer.

\begin{table}
\begin{tabular}{l l l|l l l} \hline \hline  & \#Entity & \#Relation & \#Training & \#Validation & \#Test \\ \hline WN18 & 40,943 & 18 & 141,442 & 5,000 & 5,000 \\ WN18RR & 40,943 & 11 & 86,835 & 3,034 & 3,134 \\ FB15k & 14,951 & 1,345 & 483,142 & 50,000 & 59,071 \\ FB15k-237 & 14,541 & 237 & 272,115 & 17,535 & 20,466 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of the fixed-size set predictors and the top-K predictor with \(K\) learned based on the validation set. The table reports the mean values of coverage and the average size of answer sets over 15 trials. Results are based on the RESCAL model applied to the WN18 and FB15k datasets.



scaling adjusts plausibility scores by dividing by a temperature parameter \(T\), optimized by minimizing negative log-likelihood on validation set. This calibration assumes two key points: (1) **uniform miscalibration**, where plausibility scores are consistently miscalibrated across the model (e.g., the KGE model is uniformly overconfident or underconfident for all queries); and (2) **monotonic calibration**, where the relative ordering of plausibility scores aligns with calibrated probabilities. These assumptions are overly stringent for KGE models, which tend to be overconfident with queries that have many correct answers and underconfident with those having fewer correct answers. Additionally, the relative ordering of plausibility scores is highly sensitive to minor hyperparameter changes.

Moreover, applying temperature scaling or other calibration techniques requires formulating link prediction as a classification task. However, the validation set exhibits a long-tail distribution in the number of triples associated with certain entities, i.e. many entities have few or even no associated triples. It leads to insufficient data for effective calibration for entities associated with fewer triples.

## Appendix D Further Results for Coverage & Set Size Evaluation

### Coverage and Set Size on WN18RR and FB15k237

We repeated the experiment on WN18RR and FB15k237, datasets known to be more challenging than WN18 and FB15k due to the removal of inverse relations Toutanova and Chen (2015); Dettmers et al. (2018).

The results for the filtered answer sets are presented in Table 8, while the unfiltered results are available in Table 10 in Appendix D. The conclusions from Experiment 1 remain consistent; however, we observe a significant increase in set sizes for all set predictors, particularly for WN18RR. This increase is desirable, as it aligns with the adaptiveness desideratum, where the set predictor is expected to output smaller sets for simple queries and larger sets for harder ones.

## Appendix E Further Results for Adaptiveness Evaluation

In Figure 3 - 14, we show the size of answer sets stratified by the difficulty level of queries for different conformal predictors across six representative KGE models and four benchmark datasets.

 



\begin{table}
\begin{tabular}{c c l l|l r|l r r} \hline \hline  & & \multicolumn{2}{c}{WN18RR} & \multicolumn{3}{c}{FB15k237} \\ model & MR & methods & coverage & size & model & MR & methods & coverage & size \\ \hline \multirow{4}{*}{TransE} & & naive & 0.92 (0.002) & 12592.31 (39.396) & & & naive & 0.90 (0.006) & 805.70 (38.319) \\  & & Platt & 0.90 (0.002) & 10921.01 (32.441) & & & Platt & 0.90 (0.006) & 832.14 (36.711) \\  & & topk & 0.90 (0.002) & **3571.51 (144.178)** & & & topk & 0.90 (0.001) & 875.53 (7.835) \\  & & NegScore & 0.90 (0.002) & 9409.77 (252.614) & & & NegScore & 0.90 (0.001) & 1367.25 (39.240) \\  & & Softmax & 0.90 (0.001) & 4864.10 (160.461) & & Softmax & 0.90 (0.001) & **346.67 (3.099)** \\  & & Minmax & 0.90 (0.001) & 4371.36 (172.089) & & & Minmax & 0.90 (0.001) & 482.96 (8.236) \\ \hline \multirow{4}{*}{RotatE} & & naive & 0.98 (0.003) & 2ipot42.22 (78.389) & & & naive & 0.99 (0.000) & 4564.86 (16.479) \\  & & Platt & 0.92 (0.003) & 23041.24 (67.999) & & & Platt & 0.95 (0.001) & 1851.22 (12.442) \\  & & topk & 0.90 (0.003) & **7780.12 (1372.505)** & & & topk & 0.90 (0.001) & 786.96 (9.659) \\  & & NegScore & 0.90 (0.004) & 10135.01 (887.572) & & & NegScore & 0.90 (0.001) & 396.30 (6.841) \\  & & Softmax & 0.90 (0.003) & 8469.82 (312.540) & & & Softmax & 0.90 (0.001) & **309.38 (4.527)** \\  & & Minimax & 0.90 (0.003) & 8026.50 (691.561) & & & Minmax & 0.90 (0.001) & 3130.52 (5.611) \\ \hline \multirow{4}{*}{RESCAL} & & naive & 0.82 (0.004) & 19604.12 (54.324) & & & naive & 0.75 (0.026) & 311.80 (59.631) \\  & & Platt & 0.91 (0.006) & 25156.82 (67.922) & & & Platt & 0.85 (0.017) & 492.83 (68.246) \\  & & topk & 0.90 (0.006) & 20571.25 (619.329) & & & topk & 0.90 (0.001) & 810.16 (8.750) \\  & & NegScore & 0.90 (0.006) & 19813.44 (476.890) & & & NegScore & 0.90 (0.001) & 581.74 (16.583) \\  & & Softmax & 0.90 (0.006) & 25146.85 (397.278) & & & Softmax & 0.90 (0.001) & **261.58 (5.352)** \\  & & Minimax & 0.90 (0.005) & **18262.03 (484.686)** & & & Minmax & 0.90 (0.001) & 356.23 (20.744) \\ \hline \multirow{4}{*}{DistMult} & & naive & 0.87 (0.014) & 22687.34 (104.595) & & & naive & 0.82 (0.008) & 852.93 (110.675) \\  & & Plant & 0.90 (0.005) & 26100.71 (766.341) & & & Patt & 0.88 (0.007) & 1236.71 (122.667) \\  & & topick & 0.90 (0.005) & 18220.44 (660.313) & & & topk & 0.90 (0.001) & 785.39 (7.479) \\  & & NegScore & 0.90 (0.005) & 2735.97 (843.241) & & & NegScore & 0.90 (0.001) & 340.43 (8.278) \\  & & Softmax & 0.90 (0.004) & 24347.85 (1756.093) & & & Sofmax & 0.90 (0.001) & **276.85 (4.932)** \\  & & Minimax & 0.90 (0.005) & **17555.05 (822.261)** & & & Minmax & 0.90 (0.001) & 352.42 (31.161) \\ \hline \multirow{4}{*}{CompIEx} & & naive & 0.45 (0.008) & 5939.57 (192.280) & & & naive & 0.85 (0.010) & 1027.42 (134.273) \\  & & Platt & 0.83 (0.008) & 15307.46 (374.11) & & & Patt & 0.92 (0.010) & 1701.31 (145.989) \\  & & topk & 0.90 (0.006) & 19875.19 (410.275) & & & topk & 0.90 (0.001) & 757.90 (5.512) \\  & & NegScore & 0.90 (0.007) & 19858.11 (212.472) & & & NegScore & 0.90 (0.001) & 319.42 (6.025) \\  & & Softmax & 0.90 (0.004) & 18194.32 (595.990) & & & Softmax & 0.90 (0.001) & **271.05 (2.988)** \\  & & Minimax & 0.90 (0.003) & **14101.85 (447.917)** & & & Minnesota & 0.90 (0.001) & 365.28 (72.147) \\ \hline \multirow{4}{*}{ConvE} & & naive & 0.25 (0.006) & 1131.72 (88.750) & & & naive & 0.95 (0.006) & 1072.99 (100.842) \\  & & Platt & 0.82 (0.006) & 10955.50 (800.116) & & & Platt & 0.94 (0.005) & 814.70 (75.111) \\ \cline{1-1}  & & topk & 0.90 (0.005) & 18270.13 (1407.722) & & & topk & 0.90 (0.001) & 752.37 (5.652) \\ \cline{1-1}  & & NegScore & 0.90 (0.004) & 21094.54 (687.705) & & & NegScore & 0.90 (0.001) & 718.24 (44.095) \\ \cline{1-1} \cline{2-7}  & Softmax & 0.93 (0.006) & 19851.28 (756.774) & & & Sofmax & 0.90 (0.001) & 270.40 (3.336) \\ \cline{1-1} \cline{2-7}  & Minimax & 0.90 (0.003) & **17400.91 (644.163)** & & & Minmax & 0.90 (0.001) & **242.64 (1.843)** \\ \hline \end{tabular}
\end{table}
Table 8: Quality of the filtered answer sets on WN18RR and FB15k237 datasets. This table presents the mean rank (MR) of KGE models, along with the coverage and size of answer sets generated using various set predictors. Conformal predictors are underlined. Means and standard deviations over 15 trials are reported at the \(10\%\) level (\(\epsilon=0.1\)). Predictors that fail to meet the coverage threshold of \(1-\epsilon\) (0.9) are highlighted in red. The best predictors, which meet the coverage desideratum and minimize answer set size, are highlighted in bold.





\begin{table}
\begin{tabular}{c c r l|r r|l r r} \hline \hline  & \multicolumn{3}{c}{WN18} & \multicolumn{4}{c}{FB15k} \\ model & MR & methods & coverage & size & model & MR & methods & coverage & size \\ \hline \multirow{6}{*}{TransE} & \multirow{6}{*}{261.27 (6.365)} & naive & 0.44 (0.004) & 26.39 (1.278) & & & naive & 0.73 (0.001) & 406.43 (0.827) \\  & & Platt & 0.85(0.002) & 4060.21 (89.563) & & & Platt & 0.84 (0.002) & 1337.54 (44.123) \\  & & topk & 0.90 (0.000) & 54.67 (0.789) & & topk & 0.90 (0.000) & 401.00 (1.461) \\  & & NegScore & 0.90 (0.001) & **37.29 (0.721)** & & NegScore & 0.90 (0.001) & **155.40 (0.623)** \\  & & Softmax & 0.90 (0.001) & 129.67 (4.650) & & Softmax & 0.90 (0.000) & 569.80 (2.423) \\  & & Minmax & 0.90 (0.001) & 130.45 (5.100) & & Minimax & 0.90 (0.000) & 439.39 (9.242) \\ \hline \multirow{6}{*}{RotatE} & \multirow{6}{*}{493.17 (44.041)} & naive & 0.91 (0.003) & 17704.98 (117.899) & & naive & 0.71 (0.003) & 889.73 (5.109) \\  & & Platt & 0.90 (0.002) & 16966.23 (116.434) & & & Platt & 0.88 (0.002) & 1292.32 (7.532) \\  & & topk & 0.90 (0.001) & 57.73 (2.594) & & topk & 0.90 (0.000) & 479.50 (4.066) \\  & & NegScore & 0.90 (0.002) & 17.87 (0.022) & & NegScore & 0.90 (0.001) & 209.36 (0.665) \\  & & Softmax & 0.90 (0.001) & **13.61 (0.401)** & & Softmax & 0.90 (0.000) & 277.84 (5.434) \\  & & Minmax & 0.90 (0.003) & 20.53 (0.693) & & Minimax & 0.90 (0.001) & **198.67 (1.117)** \\ \hline \multirow{6}{*}{RESCAL} & \multirow{6}{*}{338.24 (21.476)} & naive & 0.58 (0.011) & 3166.68 (42.776) & & naive & 0.58 (0.016) & 274.69 (13.802) \\  & & Platt & 0.80 (0.005) & 2035.13 (222.193) & & & Platt & 0.87 (0.003) & 755.23 (22.263) \\  & & topk & 0.90 (0.001) & 61.60 (1.744) & & topk & 0.90 (0.000) & 464.00 (2.556) \\  & & NegScore & 0.91 (0.001) & 55.36 (3.529) & & NegScore & 0.90 (0.001) & 327.14 (4.569) \\  & & Softmax & 0.90 (0.001) & **15.16 (0.358)** & & Softmax & 0.90 (0.000) & **183.69 (3.650)** \\  & & Minmax & 0.90 (0.002) & 18.81 (0.080) & & Minimax & 0.90 (0.001) & 237.00 (5.822) \\ \hline \multirow{6}{*}{DistMult} & \multirow{6}{*}{386.83 (20.312)} & naive & 0.47 (0.002) & 51.35 (7.023) & & naive & 0.35 (0.013) & 162.65 (1.103) \\  & & Plant & 0.84 (0.001) & 1282.45 (204.44) & & & Plot & 0.90 (0.001) & 625.33 (2.300) \\  & & topk & 0.90 (0.001) & 64.80 (2.587) & & topk & 0.90 (0.000) & 430.00 (1.549) \\  & & NegScore & 0.90 (0.001) & 2261.78 (405.035) & & NegScore & 0.90 (0.000) & 210.35 (2.876) \\ \cline{1-1}  & & Softmax & 0.90 (0.001) & **9.16 (0.337)** & & Softmax & 0.90 (0.000) & **143.23 (4.601)** \\  & & Minmax & 0.90 (0.002) & 18.04 (0.525) & & Minimax & 0.90 (0.000) & 197.79 (0.892) \\ \hline \multirow{6}{*}{ComplEx} & \multirow{6}{*}{467.12 (27.864)} & naive & 0.94 (0.007) & 19984.22 (142.118) & & naive & 0.28 (0.011) & 189.44 (1.424) \\  & & Platt & 0.86 (0.002) & 16802.14 (125.322) & & Platt & 0.90 (0.001) & 1082.11 (9.211) \\  & & topk & 0.91 (0.002) & 4673.1 (9.948) & & topk & 0.90 (0.000) & 485.60 (4.800) \\  & & NegScore & 0.90 (0.003) & 18.36 (0.008) & & NegScore & 0.90 (0.001) & 257.63 (9.161) \\  & & Softmax & 0.90 (0.001) & **7.63 (0.133)** & & Softmax & 0.90 (0.001) & **178.62 (3.561)** \\  & & Minmax & 0.90 (0.002) & 19.72 (0.172) & & Minimax & 0.90 (0.001) & 278.10 (20.531) \\ \hline \multirow{6}{*}{ConvE} & \multirow{6}{*}{327.52 (12.602)} & naive & 0.50 (0.015) & 18.32 (1.536) & & naive & 0.45 (0.026) & 932.71 (112.137) \\  & & Platt & 0.84 (0.007) & 95.01 (1.772) & & PLatt & 0.88 (0.014) & 4984.11 (699.111) \\ \cline{1-1}  & & topk & 0.90 (0.001) & 60.27 (1.569) & & topk & 0.90 (0.000) & 455.13 (3.384) \\ \cline{1-1}  & & NegScore & 0.90 (0.002) & 17.47 (0.617) & & NegScore & 0.90 (0.000) & **203.47 (2.291)** \\ \cline{1-1} \cline{2-7}  & & Softmax & 0.90 (0.001) & **8.45 (0.666)** & & Softmax & 0.90 (0.000) & 217.82 (6.516) \\ \cline{1-1}  & & Minmax & 0.90 (0.001) & 17.80 (0.161) & & Minimax & 0.90 (0.000) & 206.62 (8.665) \\ \hline \end{tabular}
\end{table}
Table 9: Quality of the answer sets on WN18 and FB15k datasets. This table presents the mean rank (MR) of KGE models, along with the coverage and size of answer sets generated using various set predictors. Conformal predictors are underlined. Means and standard deviations over 15 trials are reported at the \(10\%\) level (\(\epsilon=0.1\)). Predictors that fail to meet the coverage threshold of \(1-\epsilon\) (0.9) are highlighted in red. The best predictors, which meet the coverage desideratum and minimize answer set size, are highlighted in bold.





\begin{table}
\begin{tabular}{c c l l|l l l l l l} \hline \hline  & & \multicolumn{3}{c}{WN18RR} & \multicolumn{4}{c}{FB15k237} \\ model & MR & methods & coverage & size & model & MR & methods & coverage & size \\ \hline \multirow{6}{*}{TransE} & & naive & 0.92 (0.002) & 12606.42 (39.399) & & & naive & 0.90 (0.006) & 997.60 (40.772) \\  & & Platt & 0.90 (0.002) & 10935.81 (31.241) & & & Platt & 0.90 (0.006) & 1022.32 (35.123) \\  & & topk & 0.90 (0.002) & **3585.87 (144.179)** & & & topk & 0.90 (0.001) & 993.33 (8.252) \\  & & NegScore & 0.90 (0.002) & 9424.01 (252.614) & & & NegScore & 0.90 (0.001) & 1529.21 (38.853) \\  & & Softmax & 0.90 (0.001) & 4878.40 (160.461) & & & Softmax & 0.90 (0.001) & **489.91 (4.252)** \\  & & Minmax & 0.90 (0.001) & 4385.71 (172.090) & & & Minmax & 0.90 (0.001) & 689.92 (8.100) \\ \hline \multirow{6}{*}{RotatE} & & naive & 0.98 (0.003) & 29068.66 (78.388) & & & naive & 0.99 (0.000) & 4776.32 (16.527) \\  & & Platt & 0.92 (0.003) & 23065.33 (65.139) & & & Platt & 0.95 (0.001) & 1992.54 (12.111) \\  & & topk & 0.90 (0.003) & **7794.53 (1372.506)** & & & topk & 0.90 (0.001) & 899.20 (10.186) \\  & & NegScore & 0.90 (0.004) & 10149.42 (887.574) & & & NegScore & 0.90 (0.001) & 601.35 (6.779) \\  & & Softmax & 0.90 (0.003) & 8484.13 (1332.541) & & & Softmax & 0.90 (0.001) & **420.48 (5.493)** \\  & & Minimax & 0.90 (0.003) & 8040.94 (691.558) & & & Minnesota & 0.90 (0.001) & 494.12 (7.390) \\ \hline \multirow{6}{*}{RESCAL} & & naive & 0.82 (0.004) & 19617.55 (54.247) & & & naive & 0.75 (0.026) & 518.49 (60.728) \\  & & Platt & 0.91 (0.006) & 25178.84 (67.922) & & & Platt & 0.85 (0.017) & 634.23 (67.233) \\  & & topk & 0.90 (0.006) & 20585.67 (619.329) & & & topk & 0.90 (0.001) & 923.93 (9.205) \\  & & NegScore & 0.90 (0.006) & 19827.86 (476.890) & & & NegScore & 0.90 (0.001) & 794.21 (16.356) \\  & & Softmax & 0.90 (0.006) & 25161.15 (397.282) & & & Softmax & 0.90 (0.001) & **441.73 (2.242)** \\  & & Minmax & 0.90 (0.005) & **18276.47 (484.688)** & & & Minmax & 0.90 (0.001) & 550.94 (20.262) \\ \hline \multirow{6}{*}{DistMult} & & naive & 0.87 (0.014) & 22700.68 (1040.718) & & & naive & 0.82 (0.008) & 1064.07 (111.022) \\  & & Plant & 0.90 (0.005) & 26117.52 (764.112) & & & Platt & 0.88 (0.007) & 1385.57 (122.190) \\  & & topick & 0.90 (0.005) & 18234.80 (660.315) & & & topk & 0.90 (0.001) & 895.13 (7.847) \\  & & NegScore & 0.90 (0.005) & 22750.38 (843.239) & & & NegScore & 0.90 (0.001) & 446.12 (7.461) \\  & & Softmax & 0.90 (0.004) & 24362.11 (1756.095) & & & Softmax & 0.90 (0.001) & **429.23 (4.660)** \\  & & Minimax & 0.90 (0.005) & **17569.49 (822.261)** & & & Minmax & 0.90 (0.001) & 553.92 (30.822) \\ \hline \multirow{6}{*}{CompIEx} & & naive & 0.45 (0.008) & **Sepo.18 (192.118)** & & & naive & 0.85 (0.010) & 1239.13 (1334.671) \\  & & Platt & 0.83 (0.008) & 15325.22 (372.89) & & & Platt & 0.92 (0.101) & 1852.47 (142.284) \\  & & topk & 0.90 (0.006) & 19799.60 (410.282) & & & topk & 0.90 (0.001) & 865.33 (5.839) \\  & & NegScore & 0.90 (0.006) & 19872.52 (221.472) & & & NegScore & 0.90 (0.001) & 427.18 (5.201) \\  & & Softmax & 0.90 (0.004) & 18208.61 (595.990) & & & Softmax & 0.90 (0.001) & **404.95 (1.924)** \\  & & Minimax & 0.90 (0.003) & 14116.23 (447.920) & & & Minimumx & 0.90 (0.001) & 564.83 (71.526) \\ \hline \multirow{6}{*}{ConvE} & & naive & 0.25 (0.006) & 1144.71 (88.743) & & naive & 0.95 (0.006) & 1286.00 (100.851) \\  & & Platt & 0.82 (0.006) & 10973.44 (801.106) & & & Platt & 0.94 (0.005) & 964.33 (73.154) \\ \cline{1-1}  & & topk & 0.90 (0.005) & 18284.53 (1047.723) & & & topk & 0.90 (0.001) & 860.40 (6.141) \\ \cline{1-1}  & & NegScore & 0.90 (0.004) & 21108.92 (687.713) & & & NegScore & 0.90 (0.001) & 931.44 (44.128) \\ \cline{1-1} \cline{2-10}  & & Softmax & 0.93 (0.006) & 19865.73 (756.774) & & & Sofmax & 0.90 (0.001) & **369.22 (2.583)** \\ \cline{1-1} \cline{2-10}  & & Minimax & 0.90 (0.003) & **17415.35 (644.162)** & & & Minimalx & 0.90 (0.001) & 441.82 (2.828) \\ \hline \end{tabular}
\end{table}
Table 10: Quality of the answer sets on WN18RR and FB15k237 datasets. This table presents the mean rank (MR) of KGE models, along with the coverage and size of answer sets generated using various set predictors. Conformal predictors are underlined. Means and standard deviations over 15 trials are reported at the \(10\%\) level (\(\epsilon=0.1\)). Predictors that fail to meet the coverage threshold of \(1-\epsilon\) (0.9) are highlighted in red. The best predictors, which meet the coverage desideratum and minimize answer set size, are highlighted in bold.



Figure 4: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on TransE models) on the FB15k237 dataset.

Figure 5: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on RotatE models) on the FB15k dataset.

Figure 6: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on RotatE models) on the FB15k237 dataset.

 

Figure 8: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on RESCAL models) on the FB15k237 dataset.

Figure 10: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on DistMult models) on the FB15k237 dataset.

Figure 9: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on DistMult models) on the FB15k dataset.

 

Figure 14: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on ConvE models) on the FB15k237 dataset.

Figure 12: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on ComplEx models) on the FB15k237 dataset.

Figure 13: This figure shows the size of answer sets stratified by the difficulty level of queries. It shows the adaptiveness of different conformal predictors (built on ConvE models) on the FB15k dataset.