ACTC: Active Threshold Calibration for Cold-Start Knowledge Graph Completion
Year: 2023

Abstract:
Self-supervised knowledge-graph completion (KGC) relies on estimating a scoring model over (entity, relation, entity)-tuples, for example, by embedding an initial knowledge graph. Prediction quality can be improved by calibrating the scoring model, typically by adjusting the prediction thresholds using manually annotated examples. In this paper, we attempt for the first time cold-start calibration for KGC, where no annotated examples exist initially for calibration, and only a limited number of tuples can be selected for annotation. Our new method ACTC finds good per-relation thresholds efficiently based on a limited set of annotated tuples. Additionally to a few annotated tuples, ACTC also leverages unlabeled tuples by estimating their correctness with Logistic Regression or Gaussian Process classifiers. We also experiment with different methods for selecting candidate tuples for annotation: density-based and random selection. Experiments with five scoring models and an oracle annotator show an improvement of 7% points when using ACTC in the challenging setting with an annotation budget of only 10 tuples, and an average improvement of 4% points over different budgets.

heading: 1 Introduction
text: Knowledge graphs (KG) organize knowledge about the world as a graph where entities (nodes) are connected by different relations (edges). The knowledge-graph completion (KGC) task aims at adding new information in the form of (entity, relation, entity) triples to the knowledge graph. The main objective is assigning to each triple a plausibility score, which defines how likely this triple belongs to the underlying knowledge base. These scores are usually predicted by the knowledge graph embedding (KGE) models. However, most KGC approaches do not make any binary decision and provide a ranking, not
classification, which does not allow one to use them as-is to populate the KGs (Speranskaya et al., 2020). To transform the scores into predictions (i.e., how probable is it that this triple should be included in the KG), decision thresholds need to be estimated. Then, all triples with a plausibility score above the threshold are classified as positive and included in the KG; the others are predicted to be negatives and not added to the KG. Since the initial KG includes only positive samples and thus cannot be used for threshold calibration, the calibration is usually performed on a manually annotated set of positive and negative tuples (decision set). However, manual annotation is costly and limited, and, as most knowledge bases include
1853
dozens (Ellis et al., 2018), hundreds (Toutanova and Chen, 2015) or even thousands (Auer et al., 2007) of different relation types, obtaining a sufficient amount of labeled samples for each relation may be challenging. This raises a question:
How to efficiently solve the cold-start thresholds calibration problem with minimal human input?
We propose a new method for Active Threshold Calibration ACTC1, which estimates the relation thresholds by leveraging unlabeled data additionally to human-annotated data. In contrast to already existing methods (Safavi and Koutra, 2020; Speranskaya et al., 2020) that use only the annotated samples, ACTC labels additional samples automatically with a trained predictor (Logistic Regression or Gaussian Process model) estimated on the KGE model scores and available annotations. A graphical illustration of ACTC is provided in Figure 1.
Our main contributions are:
• We are the first to study threshold tuning in a budget-constrained environment. This setting is more realistic and challenging in contrast to the previous works where large validation sets have been used for threshold estimation.
• We propose actively selecting examples for manual annotation, which is also a novel approach for the KGC setting.
• We leverage the unlabeled data to have more labels at a low cost without increasing the annotation budget, which is also a novel approach for the KGC setting.
Experiments on several datasets and with different KGE models demonstrate the efficiency of ACTC for different amounts of available annotated samples, even for as little as one.

heading: 2 Related Work
text: Knowledge graph embedding methods (Dettmers et al., 2017; Trouillon et al., 2016; Bordes et al., 2013; Nickel et al., 2011) have been originally evaluated on ranking metrics, not on the actual task of triple classification, which would be necessary for KGC. More recent works have acknowledged this problem by creating data sets for evaluating KGC (instead of ranking) and proposed simple
1The code for ACTC can be found here: https://github.com/anasedova/ACTC
algorithms for finding prediction thresholds from annotated triples (Speranskaya et al., 2020; Safavi and Koutra, 2020). In our work, we study the setting where only a limited amount of such annotations can be provided, experiment with different selection strategies of samples for annotation, and analyze how to use them best. Ostapuk et al. (2019) have studied active learning for selecting triples for training a scoring model for KG triples, but their method cannot perform the crucial step of calibration. They consequently only evaluate on ranking metrics, not measuring actual link prediction quality. In contrast, our approach focuses on selecting much fewer samples for optimal calibration of a scoring model (using positive, negative, and unlabeled samples).

heading: 3 ACTC: Active Threshold Calibration
text: ACTC consists of three parts: selection of samples for manual annotation, automatic labeling of additional samples, and estimating the per-relation thresholds based on all available labels (manual and automatic ones).
The first step is selecting unlabeled samples for human annotation. In ACTC this can be done in two ways. One option is a random sampling from the set of all candidate tuples (ACTCrndm; the pseudocodes can be found in Algorithm 1). However, not all annotations are equally helpful and informative for estimation. To select the representative and informative samples that the system can profit the most from, especially with a small annotation budget, we also introduce density-based selection ACTCdens inspired by the density-based selective sampling method in active learning (Agarwal et al., 2020; Zhu et al., 2008) (the pseudocode can be found in Algorithm 2 in Appendix A). The sample density is measured by summing the squared distances between this sample’s score (predicted by the KGE model) and the scores of other samples in the unlabeled dataset. The samples with the highest density are selected for human annotation.
In a constrained-budget setting with a limited amount of manual annotations available, there are sometimes only a few samples annotated for some relations and not even one for others. To mitigate this negative effect and to obtain good thresholds even with limited manual supervision, ACTC labels more samples (in addition to the manual annotations) with a classifier trained on the manually annotated samples to predict the labels based on
Algorithm 1 ACTCrndm algorithm Input: unlabeled dataset X , annotation budget size l, minimal decision set size n, KGE model M , classifier C : R→ [0, 1] Output: set of per-relation thresholds T
# Step 1: samples selection for human annotation 1: T ← a set of per-relational thresholds 2: Xgold ← randomly selected l samples from X 3: manually annotate Xgold with ygold labels 4: for relation r do 5: Xgoldr ← samples from Xgold with relation r 6: ygoldr ← manual labels for Xgoldr 7: scoresgoldr ← KGE model scores for Xgoldr 8: lr ← |Xgoldr |
# Step 2: automatically label additional samples 9: if n > lr then
10: Train a classifier Cr on scoresgoldr and ygoldr 11: Xautor ← rand. selected n− lr samples from X 12: scoresautor ← KGE model scores for Xautor 13: Predict yautor = Cr(scoresautor ) 14: Xdec = (Xgoldr , ygoldr ) ⋃ (Xautor , yautor ) 15: else 16: Xdec = (Xgoldr , ygoldr )
# Step 3: estimate per-relation threshold τr 17: τ ← 0, best_acc← 0 18: for score in scoresgoldr do 19: τi ← score 20: accuracyi ← acc(scoresgoldr , ygoldr |τi) 21: if accuracyi > best_acc then 22: τ ← τi 23: best_acc← accuracyi 24: T.append(τ)
the KGE model scores. We experiment with two classifiers: Logistic Regression (ACTC-LR) and Gaussian Processes (ACTC-GP). The amount of automatically labeled samples depends on hyperparameter n, which reflects the minimal amount of samples needed for estimating each threshold (see ablation study of different n values in Section 5). If the number of samples annotated for a relation r (lr) is larger or equal to n, only these lr annotated samples are used for threshold estimation. If the amount of manually annotated samples is insufficient (i.e., less than n), the additional n − lr samples are randomly selected from the dataset and labeled by a LR or GP classifier. The automatically labeled and manually annotated samples build a per-relation threshold decision set, which contains at least n samples for a relation r with (manual or predicted) labels. The threshold for relation r is later optimized on this decision set.
The final part of the algorithm is the estimation of the relation-specific thresholds. Each sample score from the decision set is tried out as a potential threshold; the relation-specific thresholds that maximize the local accuracy (calculated for this decision set) are selected.

heading: 4 Experiments
text: We evaluate our method on two KGC benchmark datasets extracted from Wikidata and augmented with manually verified negative samples: CoDEx-s and CoDEx-m2 (Safavi and Koutra, 2020). Some details on their organization are provided in Appendix B. The KGE models are trained on the training sets3. The ACTC algorithm is applied on the validation sets: the gold validation labels are taken as an oracle (manual annotations; in an interactive setting they would be presented to human annotators on-the-fly); the remaining samples are used unlabeled. The test set is not exploited during ACTC training and serves solely for testing purposes. The dataset statistics are provided in Table 1. We run our experiments with four KGE models: ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2017), TransE (Bordes et al., 2013), RESCAL (Nickel et al., 2011). More information is provided in Appendix C.

heading: 4.1 Baselines
text: ACTC is compared to three baselines. The first baseline LocalOpt (Acc) optimizes the per-relation thresholds towards the accuracy: for each relation, the threshold is selected from the embedding scores assigned to the samples with manual annotations that contain this relation, so that the local accuracy (i.e., accuracy, which is calculated only for these samples) is maximized (Safavi and Koutra, 2020). We also modified this approach into LocalOpt (F1) by changing the maximization metric to the local F1 score. The third baseline is GlobalOpt, where the thresholds are selected by iterative search over a manually defined grid (Speranskaya et al., 2020). The best thresholds are selected based on the global F1 score calculated for the whole dataset4. In all baselines, the samples for manual annotation are selected randomly.
2The third CoDEx dataset, CoDEx-L, is not used in our experiments as it does not provide negative samples.
3We use the trained models provided by dataset authors. 4Labels for samples that include relations for which thresholds have not yet been estimated are calculated using the default threshold of 0.5.

heading: 4.2 Results
text: We ran the experiments for the following number of manually annotated samples: 1, 2, 5, 10, 20, 50, 100, 200, 500, and 1000. Experimental setup details are provided in Appendix E. Table 2 provides the result averaging all experiments (here and further, n = 500 for a fair comparison; see Section 5 for analyze of n value), and our method ACTC outperforms the baselines in every tried setting as well as on average. Figure 2a also demonstrates the improvement of ACTCrndm over the baselines for every tried amount of manually annotated samples on the example of CoDEx-s dataset; the exact numbers of experiments with different budgets are provided in Appendix F. The density-based selection, on the other hand, achieves considerably better results when only few manually annotated samples are available (see Figure 2b). Indeed, choosing representative samples from the highly connected clusters can be especially useful in the case of lacking annotation. LRdense, which selects points from regions of high density, can be helpful for small annotation budgets since it selects samples that are similar to other samples. In contrast, when having
a sufficient annotation budget and after selecting a certain number of samples, dense regions are already sufficiently covered, and LRrndm provides a more unbiased sample from the entire distribution.

heading: 5 Ablation Study
text: A more detailed ablation study of different ACTC settings is provided in Appendix D.
Global Thresholds. All methods described above calibrate the per-relation thresholds. Another option is to define a uniform (uni) threshold, which works as a generic threshold for all
tuples regardless the relations involved. We implemented it as ACTC − LRuni method, where the additional samples are automatically labeled and used to build a decision dataset together with the manually annotated ones - in the same way as done for the relation-specific version, but only once for the whole dataset (thus, significantly reducing the computational costs). We also applied the LocalOpt(Acc) and LocalOpt(F1) baselines in the uniform setting. Figure 3 demonstrates the results obtained with the Conve KGE model and random selection mechanism on the CodEX-s dataset. Although the universal versions generally perform worse than the relation-specific, ACTCuni still outperforms the universal baselines and even relationspecific ones for a small annotation budget.
Different n values. An important parameter in ACLC is n, the minimal sufficient amount of (manually or automatically) labeled samples needed to calibrate the threshold. The ablation study of different n values is provided in Figure 4 on the example of ACTC−LRdens setting, averaged across all annotation budgets. ACTC performs as a quite stable method towards the n values. Even a configuration with a minimum value of n = 5 outperforms baselines with a small annotation budget or even with quite large one (e.g. for RESCAL).

heading: 6 Conclusion
text: In this work, we explored for the first time the problem of cold-start calibration of scoring models for knowledge graph completion. Our new method for active threshold calibration ACTC provides different strategies of selecting the samples for manual annotation and automatically labels additional tuples with Logistic Regression and Gaussian Processes classifiers trained on the manually annotated data. Experiments on datasets with oracle positive and negative triple annotations, and several KGE
models, demonstrate the efficiency of our method and the considerable increase in the classification performance even for tiny annotation budgets.

heading: 7 Limitations
text: A potential limitation of our experiments is the use of oracle validation labels instead of human manual annotation as in the real-world setting. However, all validation sets we used in our experiments were collected based on the manually defined seed set of entities and relations, carefully cleaned and augmented with manually labeled negative samples. Moreover, we chose this more easy-to-implement setting to make our results easily reproducible and comparable with future work.
Another limitation of experiments that use established data sets and focus on isolated aspects of knowledge-graph construction is their detachment from the real-world scenarios. Indeed, in reality knowledge graph completion is done in a much more complicated environment, that involves a variety of stakeholders and aspects, such as data verification, requirements consideration, user management and so on. Nevertheless, we do believe that our method, even if studied initially in isolation, can be useful as one component in real world knowledge graph construction.

heading: 8 Ethics Statement
text: Generally, the knowledge graphs used in the experiments are biased towards the North American cultural background, and so are evaluations and predictions made on them. As a consequence, the testing that we conducted in our experiments might not reflect the completion performance for other cultural backgrounds. Due to the high costs of additional oracle annotation, we could not conduct our analysis on more diverse knowledge graphs. However, we have used the most established and benchmark dataset with calibration annotations, CoDEx, which has been collected with significant human supervision. That gives us hope that our results will be as reliable and trustworthy as possible.
While our method can lead to better and more helpful predictions from knowledge graphs, we cannot guarantee that these predictions are perfect and can be trusted as the sole basis for decisionmaking, especially in life-critical applications (e.g. healthcare).

heading: Acknowledgement
text: This research has been funded by the Vienna Science and Technology Fund (WWTF)[10.47379/VRG19008] and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) RO 5127/2-1.

heading: B CoDEx datasets
text: In our experiments, we use benchmark CoDEx datasets (Safavi and Koutra, 2020). The datasets were collected based on the Wikidata in the following way: a seed set of entities and relations for 13 domains (medicine, science, sport, etc) was defined
and used as queries to Wikidata in order to retrieve the entities, relations, and triples. After additional postprocessing (e.g. removal of inverse relations), the retrieved data was used to construct 3 datasets: CoDEx-S, CoDEx-M, and CoDEx-L. For the first two datasets, the authors additionally constructed hard negative samples (by annotating manually the candidate triples which were generated using a pretrained embedding model), which allows us to use them in our experiments.
• An example of positive triple: (Senegal, part of, West Africa).
• An example of negative triple: (Senegal, part of, Middle East).

heading: C Embedding models
text: We use four knowledge graph embedding models. This section highlights their main properties and provides their scoring functions.
ComplEX (Trouillon et al., 2016) uses complexnumbered embeddings and diagonal relation embedding matrix to score triples; the scoring function is defined as s(h, r, t) = eTh diag (rr) et.
ConvE (Dettmers et al., 2017) represents a neural approach to KGE scoring and exploits the nonlinearities: s(h, r, t) = f(vec(f([eh; r] ∗ ω)(W)t.
TransE (Bordes et al., 2013) is an example of translation KGE models, where the relations are tackled as translations between entities; the embeddings are scored with s(h, r, t) = −∥eh + rr − et∥p.
RESCAL (Nickel et al., 2011) treats the entities as vectors and relation types as matrices and scores entities and relation embeddings with the following scoring function: s(h, r, t) = eThRret.
These models were selected, first, following the previous works (Safavi and Koutra, 2020; Speranskaya et al., 2020), and, second, to demonstrate the performance of our method using the different KGE approaches: linear (ComplEX and RESCAL), translational (TransE), and neural (ConvE).

heading: D Ablation Study
text: Optimization towards F1 score. Just as we converted the LocalOpt (Acc) baseline from Safavi and Koutra (2020) to a LocalOpt(F1) setting, we also converted ACTC into ACTC(F1). The only difference is the metric, which the thresholds maximize: instead of accuracy, the threshold that provides the best F1 scores are looked for. Table 3 is an extended result table, which provides the ACTC(F1) numbers together with the standard ACTC (optimizing towards accuracy) and baselines. As can be seen, there is no dramatic change in ACTC performance; naturally enough, the F1 test score for ACTC(F1) experiments is slightly better than the F1 test score
for experiments where thresholds were selected based on accuracy value.
Estimate All Samples Apart from the automatic labeling of additional samples discussed in Section 3 (i.e., the additional samples are labeled in case of insufficient manual annotations so that the size of the decision set built from manually annotated and automatically labeled samples equals n), we also experimented with annotating all samples. All samples that were not manually labeled are automatically labeled with a classifier. However, the performance was slightly better only for the middle budgets (i.e., for the settings with 5, 10, and 20 manually annotated samples) and became considerably worse for large budgets (i.e., 100, 200, etc), especially in the denstity_selection setting. Based on that, we can conclude that a lot of (automatically) labeled additional data is not what the model profits the most; the redundant labels (which are also not gold and potentially contain mistakes) only amplify the errors and lead to worse algorithm performance.
Hard VS Soft Labels. The classifier’s predictions can be either directly used as real-valued soft labels or transformed to the hard ones by selecting the class with the maximum probability. In most of our experiments, the performance of soft and hard labels was practically indistinguishable (yet with a slight advantage of the latter). All the results provided in this paper were obtained with hard automatic labels.

heading: E Experimental Setting
text: As no validation data is available in our setting, the ACTC method does not require any hyperparameter tuning. We did not use a GPU for our experiments; one ACTC run takes, on average, 2 minutes. All results are reproducible with a seed value 12345.
ACTC does not imply any restrictions on the classifier architecture. We experimented with two classifiers: Logistic Regression classifier and Gaussian Processes classifier. For both of them, we used a Scikit-learn implementation (Pedregosa et al., 2011). The Logistic Regression classifier was used in the default Scikit-learn setting, with L2 penalty term and inverse of regularization strength equals 100. In the Gaussian Processes classifier, we experimented with the following kernels:
• squared exponential RBF kernel with length_scale = 10
• its generalized and smoothed version, Matérn kernel, with length_scale = 0.1
• a mixture of different RBF kernels, RationalQuadratic kernel, with length_scale = 0.1
All the results for Gaussian Processes classifier provided in this paper are obtained with the Matérn kernel (Minasny and McBratney, 2005) using the following kernel function:
k (xi, xj) = 1
Γ(ν)2ν−1
(√ 2ν
l d (xi, xj)
)ν ∗
∗Kν (√ 2ν
l d (xi, xj)
)
where Kν is a Bessel function and Γ is a Gamma function.

heading: F Results for Different Annotation Budgets
text: Tables 4, 5, and 6 demonstrate the performance of the different ACTC settings for different annotation budgets (1, 10, and 50, respectively). The results are averaged over all settings; each setting was repeated 100 times. Table 4 demonstrates how useful and profitable the density-selection methods are in a lower budget setting. However, the nonbiased random selection works better with more manually annotated samples (e.g., 50).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1-6
3 B1. Did you cite the creators of artifacts you used? Sections 1, 2, 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 1 (footnote)
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 8
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 7, Appendix B
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 8
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix E
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Tables 1, 3, 4, 5, 6, Figures 2, 3, 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix E
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Moreover, we chose this more easy-to-implement setting to make our results easily reproducible and comparable with future work. Another limitation of experiments that use established data sets and focus on isolated aspects of knowledge-graph construction is their detachment from the real-world scenarios. Indeed, in reality knowledge graph completion is done in a much more complicated environment, that involves a variety of stakeholders and aspects, such as data verification, requirements consideration, user management and so on. Nevertheless, we do believe that our method, even if studied initially in isolation, can be useful as one component in real world knowledge graph construction