Probability Calibration for Knowledge Graph Embedding Models
Authors: Pedro Tabacof
Luca Costabello
Venue:  (2020)
Keywords: knowledge graph embeddings
probability calibration
calibration
graph representation learning
knowledge graphs

Abstract: Knowledge graph embedding research has overlooked the problem of probability calibration. We show popular embedding models are indeed uncalibrated. That means probability estimates associated to predicted triples are unreliable. We present a novel method to calibrate a model when ground truth negatives are not available, which is the usual case in knowledge graphs. We propose to use Platt scaling and isotonic regression alongside our method. Experiments on three datasets with ground truth negatives show our contribution leads to well calibrated models when compared to the gold standard of using negatives. We get significantly better results than the uncalibrated models from all calibration methods. We show isotonic regression offers the best the performance overall, not without trade-offs. We also show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds.

Published as a conference paper at ICLR 2020
PROBABILITY CALIBRATION GRAPH EMBEDDING MODELS
FOR
KNOWLEDGE
Pedro Tabacof, Luca Costabello
Accenture Labs
Dublin, Ireland (pedro. tabacof, luca.
luca.costabe www..to lucaa..cccabbllloocccetttte....
ABSTRACT
Knowledge graph embedding research has overlooked the problem of probabil- ity calibration. We show popular embedding models are indeed uncalibrated. That means probability estimates associated to predicted triples are unreliable. We present a novel method calibrate model when ground truth negatives are not available, which is the usual case in knowledge graphs. We propose to use Platt scaling and isotonic regression alongside our method. Experiments on three datasets with ground truth negatives show contribution leads to well calibrated models when compared to the gold standard of using negatives. We get signifi- cantly better results than the uncalibrated models from all calibration methods. We show isotonic regression offers the best the performance overall, not without trade-offs. We also show that calibrated models reach state-of-the accuracy without the need to define relation-specific decision thresholds.
INTRODUCTION
Knowledge graph embedding models are neural architectures that learn vector representations (i.e. embeddings) of nodes and edges of knowledge graph. Such knowledge graph embeddings have applications in knowledge graph completion, knowledge discovery, entity resolution, and link-based clustering, just to cite few (Nickel et al. 2016a)).
Despite burgeoning research, the problem of calibrating such models has been overlooked, and existing knowledge graph embedding models do not offer any guarantee the probability estimates they assign to predicted facts. Probability calibration is important whenever you need the predictions to make probabilistic sense, i.e., if the model predicts fact is true with 80% confidence, it should to be correct 80% of the times. Prior art suggests to use a sigmoid layer to turn logits returned by models into probabilities (Nickel al. 201 (also called the expit transform), but we show that this provides poor calibration. Figure I shows reliability diagrams for off-the-shelf TransE and ComplEx. The identity function represents perfect calibration. Both models are miscalibrated: all TransE combinations in Figure I under-forr mnder-forecast the probabilities (i.e. probabilities are too small), whereas ComplEx under-forecasts or over-forecasts according which loss is used (FigurdTp))
Calibration is crucial in high-stakes scenarios such as drug-target discovery from biological net- works, where end-users need trustworthy and interpretable decisions. Moreover, since probabilities are not calibrated, when classifying triples (i.e. facts) true or false, users must define relation- specific thresholds, which can awkward for graphs with great number of relation types.
To the best of our knowledge, this is the first work to focus calibration for knowledge embed- dings. Our contribution is two-fold: First, we use Platt Scaling and isotonic regression to calibrate knowledge graph embedding models feature of knowledge graphs is that they usually rely on the open world assumption (facts not present not necessarily false, they are simply unknown). This makes calibration troublesome because of the lack of ground truth negatives. For this reason, our second and main contribution a calibration heuristics that combines Platt-scaling isotonic regression with synthetically generated negatives.
datasets that include ground truth negatives. One peculiar
Experimental results show that we obtain calberecelibrated models and that it is possible to cali- brate knowledge graph embedding models even when ground truth negatives present. We

---

Published as a conference paper at ICLR 2020
WordNet11 with TransE
Freebase13 13 with ComplEx
Reliability diagram scores transform
Reliability diagram scores with expit transform
1.0
10
Fruction of postives
Fraction of positves
0.6
0.4
0.4
Perfectly calibrated Pairwise
Perfectly calibrated
Pairwise
Self adversarial
Multiciass NLL
NLL
NLL
0.2
0.4
0.6
0.8
1.0
1.0
Moan predicted value
Mean predicted value
Figure 1: Reliability diagrams of uncalibrated models. Probabilities are generated by a logistic sigmoid layer. The larger the deviation from the diagonal, the more uncalibrated is the model. We present four different common loss functions used to train knowledge graph embedding models. (a) Uncalibrated TransE on WN11. (b) Uncalibrated ComplEx on FB13. Best viewed in colors.
also experiment with triple classification, and show that calibrated models reach state-of- site-off-hhearrr accuracy without the need to define relation-specific decision thresholds.
2
RELATED WORK
comprehensive survey of knowledge graph embedding models is out of the scope of this paper. Recent surveys such (Nickel al. (016a) and (Cai et al. 2017) summarize recent literature.
TransE (Bordes et al. 2013) is the forerunner of distance- based methods, and spun a number of models commonly referred to as TransX. The intuition behind the symmetric bilinear-diagon model DistMult (Yang et al. 2015) paved the way for its asymmetric evolutions in the complex space, RotatE (Sun et 2019 and ComplEx (Trouillon uses hypercomplex representations (Zhang et al. 2019)). HolE relies instead on circular correla- tion (Nickel al.] 2016b). The recent TorusE (Ebisu Ichise
al. 2016) (a generalization of which
2018 operates on lie group and not in the Euclidean space. While the above models can be interpreted as multilayer perceptrons, others such as ConvE (Dettmers et al. 2018) or ConvKB (Nguyen et al. 2018) include convolutional layers. More recent works adopt capsule networks architectures (Nguyen et al.) 2019). Adversarial learning is used by KBGAN (Cai & Wang 2018), whereas attention mechanisms are instead used by (Nathani et al. 2019). Some models such RESCAL (Nickel et al. 2011). TuckER (Balazević et al. 2019), and SimplE (Kazemi & Poole 2018) rely on tensor decomposition techniques. More recently, ANALOGY adopts differentiable version of analogical reasoning (Liu et al., (017). In this paper we limit our analysis to four popular models: TransE, DistMult, ComplEx and HolE. They do not address the problem of assessing the reliability of predictions, leave aside calibrating probabilities.
Besides well-established techniques such as Platt scaling (Platt al. 1999) and isotonic regres- sion (Zadrozny & Elkan 2002). recent interest in neural architectures calibration show that modern poorly calibrated and that calibration can be improved with novel methods. For example, (Guo et al. 2017) successfully proposes to use temperature scaling for calibrating modern neural networks in classification problems. On the same line, (Kuleshov al.) 2018) pro- poses a procedure based on Platt scaling to calibrate deep neural networks in regression problems.
neural architectures
The Knowledge Vault pipeline in (Dong et al., 2014) extracts triples from unstructured knowledge and is equipped with Platt scaling calibration, but this is not applied to knowledge graph embedding models. KG2E (He et al., 2015) proposes to use normally-distribure the uncertainty, but their model does not provide the probability of a triple being true, KG2E would also benefit from the output calibration we propose here. To the best of our knowledge, the only work that adopts probability calibration to knowledge graph embedding models is Krompaß & Tresp (2015). The authors propose to use ensembles in order to improve the results of knowledge graph embedding tasks. For that, they propose calibrate the models with Platt scaling, they
embeddings to account for
2

---

Published as a conference paper at ICLR 2020
WNTI
FB13
YAGO39K
FB15K-237
WNI8RR
Training
112,581
316,232
354,996
272,115
86,835
Model
Scoring Function fm
Validation
5,218
11,816
18,682
17,535
3,034
Transt:
lle, + ealln
21.088
47,466
18,728
3,134
DistMult
(ea, Fp: no)
Entities
38,696
75,043
39,374
14,541
40,943
ComplEx
Re. Fp. (s))
Relations
13
39
237
HolE
(e..Fp @ eo)
(b)
Table I: (a) Triple classification datasets used in experiments (left); link prediction datasets used for positive base rate experiments (right); (b) Scoring functions of models used in experiments.
operate on the same scale. No further details on the calibration procedure are provided. Besides, there is explanation on how to handle the lack of negatives.
PRELIMINARIES
Knowledge Graph. Formally, a knowledge graph 9 = [(s,p. S x R x E is a set of triples t == (s, p, o) each including subject E È, predicate R, and an object E. E and R are the sets of all entities and relation types of g.
Triple Classification. Binary classification task where G (which includes only positive triples) is used as training set, and T = {(s, }}} S E x R x E is disjoint test set of labeled triples to classify. Note T includes positives and negatives. Since the learned models multiple decision thresholds т; must be picked, where O < i < [R], i.e. one for each relation type. This is done using a validation set (Bordes al. 2013). Classification metries apply (e.g. accuracy).
not calibrated,
Given a training set & that includes only positive triples, the goal is assigning a score f(1) E I proportional to the likelihood that each unlabeled triple included in held-out set S is true. Note S does not have ground truth positives or negatives. This task is cast as a learning to rank problem, and uses metrics such as mean rank (MR), mean reciprocal rank (MRR) or Hits @ N.
Link Prediction.
Knowledge Graph Embeddings. Knowledge graph embedding models are neural architectures that encode concepts from a knowledge graph G (i.e. entities and relation types R. into low- dimensional, continuous vectors E IR* (i.e, the embeddings). Embeddings are learned by training neural architecture over G. Although such architectures vary, the training phase always consists in minimizing a loss function C that includes scoring function fm (t), i.e. model-specific function that assigns score to a triple i (s, p, o) (more precisely, the input of f' are the embeddings the subject e,, the predicate pp. and the object e.). The goal of the optimization procedure is learning optimal embeddings, such that the scoring function fm assigns high scores to positive triples t* and low scores to triples unlikely be true t- Existing models propose scoring functions that combine the embeddings C,, Fp, e, E TR using different intuitions. Table 1b lists the scoring functions of the most common models. For example, the scoring function of TransE computes a similarity between the embedding of the subject e, translated by the embedding of the predicate e, and the embedding of the object e., using the L1 or L2 norm ||- Such scoring function then used on positive and negative triples + G.t E N in the loss function. This is usually pairwise margin- baaeed loss (Bordes et al. 2013), negative log- likelihoooo or multi-class log likelihood (Lacroix et al. 2018). Since the training set usually includes positive statements, we generate synthetic negatives t E N required for training. We so by corrupting one side of the triple at a time (i.e. either the subject or the object), following the protocol proposed by (Bordes al. 2013).
Calibration. Given a knowledge graph embedding model identified by its scoring function fms with fm (t) P. where ý is the estimated confidence level that triple t (s,p. 0) is true, we define fm to be calibrated if و represents a true probability. For example, if f.) predicts 100 triples all with confidence p 0.7, expect exactly 70 to be actually true. Calibrating model requires reliable metrics to detect miscalibration, and effective techniques to fix such distortion. Appendix A. incluue
includes definitions and background the calibration metrics adopted in the paper.
3

---

Published as a conference paper at ICLR 2020
4 CALIBRATING KNOWLEDGE GRAPH EMBEDDING MODELS PREDICTIONS
We propose two scenario-dependent calibration techniques: we first address the case with ground truth negatives f- W.. The second deals with the absence of ground truth negatives.
Calibration with Ground Truth Negatives. We propose to use off-the-shelf Platt scaling and iso- tonic regression, techniques proved to be effective in literature. is worth reiterating that to calibrate a model negative triples / are required from a held-out dataset (which could be the validation set). Such negatives usually available in triple classification datasets (FB13, WN11, YAGO39K)
Calibration with Synthetic Negatives. Our main contribution is for the case where no ground truth negatives are provided all, which is in fact the usual scenario for link prediction tasks.
We propose to adopt Platt scaling or isotonic regression and to synthetically generate corrupted triples negatives, while using sample weights to guarantee that the frequencies adhere to the base rate of the population (which is problem-dependent and must be user-specified), It is worth noting that it is not possible calibrate model without implicit or explicit base rate. If is not implicit on the dataset (the ratio of positives to totals), it must be explicitly provided.
We generate synthetic negatives N following the standard protocol proposed by (Bordes et al. 2013 for every positive triple t = (s. o), we corrupt one side of the triple at time (i.e. either the subject
the object o) by replacing it with other entities in E. The number of corruptions generated per positive is defined by the user-defined corruption rate
N. Since the number of negatives N [W] can be much greater than the number of positive triples P =G], when dealing with calibration with synthetically generated corruptions, we weigh the positive and negative triples to make the calibrated model match the population base rate a = P/(P + N) E [o, 1], otherwise the base rate would depend on the arbitrary choice of r7.
Given a positive base rate a, we propose the following weighting scheme:
W+ = 7 for positive triples G
1 for negative triples /
where wi+ IR is the weight associated to the positive triples and w_ RR to the negatives. The wt weight removes the imbalance determined by having a higher number of corruptions than positive triples each batch. The w_ weight guarantees that the given positive base rate a is respected.
The above can be verified as follows. For the unweighted problem, the positive base rate is simply the ratio of positive examples the total number of examples:
P
P+N
we add uniform weights to each class, we have:
W P
(3)
w W_N N W
By defining w+ 77, i.e. adopting the ratio of negatives to positives (corruption rate), then have:
N
1
P
(4)
Nw PN
W
W_
+1
Thus, the negative weights is:
w
(5)
RESULTS
We compute the calibration quality of our heuristics, showing that achieve calibrated predictions even when ground truth negative triples are not available. We then show the impact of calibrated predictions on the task of triple classification.
We also experimented with per-hatch entities only, without any significant changes to the results. Future work will experiments with additional techniques as proposed by Kotnis & Nastase (2017)
4

---

Published as a conference paper at ICLR 2020
Datasets, We run experiments triple classification datasets that include ground truth negatives (Table We train on the training set, calibrate on the validation set, and evaluate on the test set.
2013). A subset of Wordnet (Miller 1995). it includes a large number WN11 (Socher et al.
of hyponym and hypernym relations thus including hierarchical structures.
FB13 (Socher et al.
2013). A subset of Freebase (Bollacker et al. 2008), it includes facts on famous people (place of birth and/or death, profession, nationality, etc).
YAGO39K (Ly et al.] 2018). This recently released dataset has been carved out of YAGO3 (Mahdisoltani al. 2013), and includes mixture of facts about famous people. events, places, and sports teams.
We also use two standard link prediction benchmark datasets, WN18RR (Dettmers al. 2018 (a subset of Wordnet) and FB15K-237 (Toutanova et al. 2015) (a subset of Freebase). Their test sets do not include ground truth negatives.
Implementation Details. The knowledge graph embedding models are implemented with the AmpliGraph library (Costabello et al., 2019) version 1.1, using TensorFlow 1.13 (Abadi et al, 2016) and Python 3.6 on the backend. All experiments were run under Ubuntu 16.04 on an In- tel Xeon Gold 6142, 64 GB, equipped with a Tesla V100 16GB. Code and experiments are available at htttp::/ggitttite hub. com Accenturre/Arg
Graph
For each dataset in Table D. we train a TransE, DistMult, and a ComplEx knowledge graph embedding model. We rely typical hyperparameter values: we train the embed- dings with dimensionality k = 100, Adam optimizer, initial learning rate aq = 1c-4, negatives per positive ratio y 20, epochs 1000. We train all models on four different loss functions: Self- adversarial (Sun et al.) 2019), pairwise (Bordes et al., 2013), NLL, and Multiclass-NLL (Lacroix) al. 2018 Different losses are used in different experiments.
Hyperparameter Tuning.
5.1 CALIBRATION RESULTS
Calibration Success. Table 2 reports Brier scores and log losses for all our calibration methods, grouped by the type of negative triples they deal with (ground truth or synthetic). All calibration methods show better-calibrated results than the uncalibrated case, by considerable margin and for all datasets. particular, to put the results of the synthetic strategy to predict the positive base rate as baseline, for each of the cases in Table 2 (the three datasets share the same positive base rate a = 0.5), we would get Brier score B 0.25 and log loss Liog = 0.69, results that are always worse than methods. There is considerable variance of results between models given a dataset, which also happens when varying losses given a particular combination of model and dataset (Table 3). TransE provides the best results for WN11 and FB while DistMult works best for YAGO39K. We later propose that this variance comes from the quality of the embeddings themselves, that is, better embeddings allow for better calibration.
perspective, if we suppose
In Figure 2 we also evaluate just the frequencies themselves, ignoring sharpness (i.e. whether prob- abilities close to 0 or 1), using reliability diagrams for single model-loss combination, for all datasets (ComplEx+NLL)) Calibration plots show a remarkable difference between the uncalibrated baseline (s-shaped blue line on the left-hand side) and all calibrated models (curves closer to the identity function are better). A visual comparison of uncalibrated curves in Figure with those in Figure 2 also gives sense of the effectiveness of calibration.
Ground Truth vs Synthetic. As expected, the ground truth method generally performs better than the synthetic calibration, since it has more data in both quantity (twice as much) and quality (two classes instead of one). Even so, the synthetic method is much closer to the ground truth than to the uncalibrated scores, as highlighted by the calibration plots in Figure 2 For WN11, is actually as good as the calibration with the ground truth. This shows that our proposed method works as intended and could be used in situations where we do not have access to the ground truth, as is the case for most knowledge graph datasets.
Isotonic vs Platt. Isotonic regression performs better than Platt scaling in general, but in practice Isotonic regression has the disadvantage of not being a convex differentiable algorithm Zadrozny & Elkan (2002). This is particularly problematic for the synthetic calibration, as it requires the generation of the synthetic corrutiinns which can only be made scale via mini-batch based
5

---

Published as a conference paper at ICLR 2020
Reliability diagram: CoLp
1.0
Fraction positives
Perfectly calibrated
Uncalibrated scores: expit
Platt scaling: synthetic
Platt scaling: ground truth
Isotonic: synthetic
Isotonic: ground truth
0.0
Reliability diagram: WN11,ComplEx-NLL NLL
1.0
Fraction positives
Perfectly calibrated
Uncalibrated scores: expit
$
scaling: synthetic
Platt scaling: ground truth
Isotonic: synthetic
Isotonic: ground truth
0.0
Reliability diagram: YAGO39K,ComplEx-Ae NNL
1.0
Fraction of positives
0.6
Perfectly calibrated
Uncalibrated scores: expit
Platt scaling: synthetic
Platt scaling: ground truth
0.2
Isotonic: synthetic
Isotonic: ground truth
0.0
0.2
0.6 Mean predicted value
0.4
0.8
1.0
Figure/ 2: Calibration plots for the best calibrated model-loss combinations. Isotonic regression delivers the best results, getting very close the perfectly calibrated line, both when used with the ground truth method or our proposed synthetic method. Best viewed colors.
Brier Score
Log Loss
Ground Truth
Synthetic
Ground Truth
Synthetic
Uncalib
Platt
Iso
Platt
Iso
Uncalib
Platt
Platt
Iso
TransE
.443
.089
.087
.092
.088
1.959
.302
.295
.311
.296
DistMult
.88
213
208
214
.088
5.625
.618
.604
.618
.601
WN11
ComplEx
.490
.240
.227
.240
.228
6.061
.674
.651
.674
.650
HolE
.444
.355
.235
.235
2336
2.731
.663
.661
.663
.668
TransE
.466
.124
.124
.148
.141
1.534
.390
.391
.459
.442
DistMult
.473
.178
.170
.185
.192
2.177
.533
.518
.549
.567
FB13
ComplEx
.177
.170
.182
.189
2.393
.534
.516
.544
.565
HolE
.452
.229
.228
.242
.263
1.681
.650
.651
.677
.725
TransE
.363
.095
.093
.106
.110
1.062
.319
.309
.370
.376
YAGO 39K
DistMult
.284
.081
.079
.093
.089
1.043
.279
.266
.311
.308
ComplEx
.664
.089
.084
.097
.095
1.199
.305
.278
.323
.313
HolE .35
.141
.140
.166
.162
1.065
.444
.438
.581
.537
Table 2: Calibration test results (self-adversarial loss (Sun et al. results in bold for each combination of dataset and metric.
2019
Low score
better. Best
optimization procedure. Platt scaling, given that it convex and differentiable loss, be made part of a computational graph and optimized with mini-batches, thus it can rely on the modern computational infrastructure designed train deep neural networks.
Influence of Loss F Function. We experiment with different losses, to how calibration affeets each of them (Table 3). We choose to work with TransE, which is reported as a strong baseline in (Hamaguchi et al. 2017). Belf-adversarial loss obtains the best calibration results for all calibra-

---

Published as a conference paper at ICLR 2020
Brier Score
Log Loss
MRR
Ground
Truth
Synthetic
Ground
Truth
Synthetic
(filtered)
C
Platt
Iso
Platt
Iso
Platt
Iso
Platt
Iso
Pairwise
.202
.198
.209
.200
.591
.885
.606
.589
.058
NLL
.093
088
-094
.088
.322
.299
.34
.301
.134
Multiclass-NLL
-004
.189
.204
.189
.599
.550
.599
.551
.108
Self-adversarial
.089
.087
.092
.088
.302
.295
.311
.296
.155
(a) WN11
Brier Score
Log Loss
MRR
Ground Truth
Synthetic
Ground Truth
Synthetic
(filtered)
C
Platt
Iso
Platt
1so
Platt
Iso
Platt
Iso
Pairwise
.225
.203
.255
.208
.636
.582
.637
.594
.282
NLL
.209
.003
.240
.244
.614
.592
.676
.685
.202
Multiclass
NLL
.166
.146
.162
.159
.455
.454
.500
.490
.402
Self aele-aaveraarial
.124
.124
.142
.141
.390
.390
.446
.422
.296
(b) FB13
Brier Score
Log Loss
MRR
Ground
Truth
Synthetic
Ground Truth
Synthetic
(filtered)
C
Platt
Iso
Platt
1so
Platt
Iso
Platt
Iso
Pairwise
.123
.103
.147
.113
.445
.352
.477
.393
.371
NLL
.187
.170
.260
200
.577
.518
.756
.622
.063
Multiclass-NIL.
.104
.128
.116
.392
.350
.431
.325
Self-adversarial .095
.093
.113
.109
.319
.308
.399
.376
.169
(c) YAGO39K
Calibration test results using different losses L with TransE. Lower calibration metrics better. We compute MRR only on positive test triples. Self-adversarial loss achieves better calibrated results across the board. Results show correlation between MRR and calibration performance, iee. embeddings that bring higher MRR are not necessarily easier to calibrate. Best results in bold.
Table 3:
tion methods, across all datasets. Experiments also show the choice of the loss has a big impact, greater than the choice of calibration method or embedding model. We assess whether such vari- ability is determined by the quality of the embeddings. To verify whether better embeddings lead to sharper calibration, computes the (inverse) rank of the triple against synthetic corruptions, then averages the inverse rank (Table 3). In fact, we notice correlation between calibration results and MRR. In other words, embeddings that lead to the best predictive power are not necessary the best calibrated.
report the mean reciprocal rank (MRR), which, for each true test triple,
Positive Base Rate. We apply synthetic calibration method to two link prediction benchmark datasets, FB15K-237 and WN18RR. As they only provide positive examples, we apply our method with varying base rates ag, linearly spaced from 0.05 to 0.95. We evaluate results relying on the closed-worid assumption, i.c. triples not present in training, validation or test sets are considered negative. For each a we calibrate the model using the synthetic method with both isotonic regres- sion and Platt scaling. We sample negatives from the negative set under the implied negative rate, and calculate baseline which is simply having all probability predictions equal to a;, Figure 3 shows that isotonic regression and Platt scaling perform similarly and always considerably below baseline. As expected from the previous results, the uncalibrated scores perform poorly, only reaching acceptable levels around some particular base rates.
Triple Classification and Decision Threshold. To overcome the need to learn [R] decision thresh- olds 7. from the validation set, propose to rely on calibrated probabilities, and use the natural threshold of r = 0.5. Table 4 shows how calibration affects the triple classification task, comparing with the literature standard of per-relation thresholds (last column). For simplicity, note same self- adf-aaavesalial loss in Table 2 and Table We learn thresholds т; on validation sets, resulting in 11, 7, and 33 thresholds for WNT1, FB13 and YAGO39K respectively.
the
Using a single 7 0.5 and calibration provides competitive results compared to multiple learned thresholds (note uncalibrated results with 0.5 are poor, as expected). is worth mentioning that

---

Published as a conference paper at ICLR 2020
FB15K-237: Synthetic calibration
WN18RR: Synthetic calibration
Baseline
Baseline
Platt scaling
Platt scaling
0.225
Isotonic
Isotonic
Uncalibrated scores
Uncalibrated scores
score
0.150
0.075
0.000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
Positive base rate
Positive base
Figure 3: Synthetic calibration FB15K-237 and WN18RR, with varying positive base rates. The baseline stands for using the positive base rate as the probability prediction. Results are evaluated under the closed-world assumption, using the same positive base rate used to calibrate the models.
Ground Truth (T .5)
Synthetic (T .5)
Uncalib.
Uncalib, (Per-Relation r)
Platt
Iso
Platt
Iso
(T = .5)
Reproduced
Literature
TransE 88.8
88.9
88.9
50.7
DistMult 66.5
67.2
66.4
67.1
50.8
67.2
88.9
WN11
ComplEx 60.6
62.4
60.0
62.4
50.8
59.6
HolE
59.3
59.0
59.3
59.0
50.9
60.8
TransE 82.4
82.4
80.7
80.2
50.0
82.1
FB13
DistMult 72.5
73.2
72.1
70.2
50.1
80.8
89.1"
ComplEx 73.8
74.2
74.2
72.4
50.1
HolE 60.3
60.6
57.8
54.3
50.0
62.6
TransE 87.2
87.8
85.3
84.9
50.2
YAGO 39K
DistMult 88.9
89.3
88.1
88.5
56.7
90.2
93.8
ComplEx 87.3
88.2
86.9
87.2
61.1
89.4
HolE 80.4
80.4
78.4
78.5
50.6
81.5
Table 4: Effect calibration on triple classification accuracy. Best results in bold. For all calibration methods there is single threshold, - 0.5. For the per-relation т, learned multiple thresholds from validation sets (Appendix A.5 We did not carry out additional model selection, and used Table hyperparameters instead. Isotonic regression reaches state-of-the-ar results for WN11.
Results of * from (Zhang et al. 2018); from (Ji et al. 2016); i from (Lv et al. 2018).
are at par with state-of-the-art results for WN11. Isotonic regression is again the best method, there more variance in the model choice. Our proposed calibration method with synthetic neg- atives performs well overall, even though calibration is performed only using half of the validation set (negatives examples are replaced by synthetic negatives).
CONCLUSION
We propose method calibrate knowledge graph embedding models. We target datasets with and without ground truth negatives. We experiment on triple classification datasets and apply Platt scaling and isotonic regression with and without synthetic negatives controlled by our heuristics. All calibration methods perform significantly better than uncalibrated scores. We show that isotonic re- gression brings better calibration performance, but computationally more expensive. Additional experiments triple classification shows that calibration allows to single decision threshold, reaching state-of-the-an results without the need to learn per-relation thresholds.
Future work will evaluate additional calibration algorithms, such as beta calibration (Kull et al. 2017) or Bayesian binning (Naeini et al. 2013). We will also experiment on ensembling of knowl- edge graph embedding models, inspired b Krompaß & Tresp 2015). The rationale is that different models operate on different scales, but calibrating brings them all to the same probability scale, so their output be easily combined.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ( ({OSDI) 16), pp. 265-283, 2016.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. Tucker: Tensor factorization for knowl- edge graph completion. arXiv preprint arXiv: feti::/00100 000000990, 2019.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collab- oratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247-1250. AcM, 2008
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pp. 2787-2795, 2013.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. A comprehensive survey of graph embedding: Problems, techniques and applications. arXiv preprint arXiv:1709.07604 2017.
Liwei Cai and William Yang Wang. Kbgan: Adversarial learning for knowledge graph embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1470- 1480, 2018.
Luca Costabello, Sumit Pai, Chan Le Van, Rory McGrath, Nicholas McCarthy, and Pedro Tabacof. AmpliGraph: Library for Representation Learning Knowledge Graphs, March 2019. URL
https://dom weloorg//oo / 5281/222odd. 2595044
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22, 1983.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp. and Sebastian Riedel, Convolutional 2d knowledge graph embeddings. In Procs of AAAI, 2018 URL https: //www/ aaai. org/
ocs/ bess/index.ppp php AAAI/ GAAAI/AAAIO AAAI18/ http:/apee////// hatper/vieww111. 17366
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: A web-scale approach to proba- bilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 601-610. ACM, 2014.
Takuma Ebisu and Ryutaro Ichise. Toruse: Knowledge graph embedding lie group. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- -Volume 70, pp. 1321-1330. JMLR. org, 2017.
Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. Knowledge transfer for www-o---noolede--bas. entities: a graph neural network approach. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 1802-1808 AAAI Press, 2017.
Shizhu He, Kang Liu, Guoliang Ji, and Jun Zhao. Learning to represent knowledge graphs with gaussian embedding. Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp.
ACM, 2015.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. Knowledge graph completion with adaptive sparse transfer matrix. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.

---

Published as a conference paper at ICLR 2020
Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowl- edge graphs. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Caee--Ceanceaee Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, Pp. 4284 4295. Curran Associates, Inc., 2018. URL http: http:://pppprs....ae ps. cctps/pprrrppp.ppppppppppee pdf
Bhushan Kotnis and Vivi Nastase. Analysis of the impact of negative sampling on link prediction in knowledge graphs. arXiv preprint arXiv:1708.0000 06816, 2017.
Denis Krompaß and Volker Tresp. Ensemble solutions for link- ink-predictton in knowledge graphs. In 2nd Workshop, 2015.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint 187.00080 2018.
Meelis Kull, Telmo M Silva Filho, Peter Flach, et al. Beyond sigmoids: How to obtain well- calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statis- tics, 11(2):0052-500 5080, 2017.
Timothee Lacroix. Nicolas Usunier, and Guillaume Obozinski. Canonical tensor decomposition for knowledge base completion. In International Conference on Machine Learning, pp. 2869-2878, 2018.
Hanxiao Liu, Yuexin Wu, and Yiming Yang. Analogical inference for multi-relational embeddings. In Proceedings of the 34th International Conference on Machine Secarning-Vouume 70, pp. 2168- 2175 2178. JMLR. org, 2017.
Xin Lv, Lei Hou, Juanzi Li, and Zhiyuan Liu. Differentiating concepts and instances for knowledge graph embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1971-1979, 2018.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. Yago3: A knowledge base from multilingual wikipedias. 2013.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 1): 39-41, 1995.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. Learning attention-based em- beddings for relation prediction in knowledge graphs. arXiv preprint arXiv:1906.1 195, 2019.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoe Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 327-333, 2018
Dai Quoc Nguyen, Thanh Vu, Tu Dinh Nguyen, Dat Quoe Nguyen, and Dinh Phung. capsule network-based embedding model for knowledge graph completion and search personalization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 2180-2189, 2019.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. three-way model for collective learning on multi-relational data. In ICML, volume 11, pp. 809-816, 2011.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Procs of the IEEE, 104(1):11-33, 2016a.
Maximilian Nickel, Lorenzo Rosasco, Tomaso A Poggio, et al. Holographic embeddings of knowl- edge graphs. In AAAI, pp. 1955-1961, 2016b.
10

---

Published as a conference paper at ICLR 2020
Alexandru Niculescu-Min Mizil and Rich Caruana. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd international conference on Machine learning, pp. 625-632
ACM, 2005.
John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems (NIPS), 2013.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint 1rXiv:1902.11197 2019.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1499-1509, 2015.
Théo Trouillon, Johannes Welbl, Sehastian Riedel, fiii Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In procs of ICML, pp. 207 102000 2016.
Bishan Yang, Scott Wen-tau Yih, Xiaodong He. Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In Procs of ICLR, 2015.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. Proceedings of the eighth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pp. 694-699, ACM, 2002.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. Quaternion knowledge graph embedding. arXiv preprint arXiv:1904, 10281, 2019.
Zhao Zhang, Fuzhen Zhuang, Meng Qu, Fen Lin, and Qing He. Knowledge graph embedding with hierarchical relation structure. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3198-3207, 2018.
11

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 CALIBRATION METRICS
Reliability Diagram DDGGroot & Fienberg 1983
Niculescu-Mizii & Caruana, 2005). Also known
as calibration plot, this diagram visual depiction of the calibration of model (sce Figure for an example). shows the expected sample accuracy as a function of the estimated confidence. A hypothetical perfectly calibrated model is represented by the diagonal line (i.e. the identity function). Divergence from such diagonal indicates calibration issues (Guo et al. 2017).
Brier Score (Brier 1950). It is popular metric used to measure how well binary classifier calibrated. is defined as the mean squared error between n probability estimates f and the corresponding actual outcomes y 0, 1. The smaller the Brier score, the better calibrated is the model. Note that the Brier score B E [0,
7 B Σ
11 P.)²
Log Loss is another effective and popular metric to measure the reliability of the probabilities re- turned by a classifier. The logarithmic loss measures the relative uncertainty between the probability estimates produced by the model and the corresponding true labels.
Liog (y (og + (1 - y) log(1 D))
1999) for support vector machines, Platt scaling is a popular Platt Scaling. Proposed by (Platt et al.
parametric calibration techniques for binary classifiers. The method consists in fitting a logistic regression model to the scores returned by binary classifier, such that ğ sap b), where I E IR is the uncalibrated score of the classifier, a, E R are trained scalar weights. and q is the calibrated probability returned output. Such model can be trained be trained by optimizing the NLL loss with non-binary targets derived by the Bayes rule under an uninformative prior, resulting an Maximum Posteriori estimate.
Isotonic Regression (Zadrozny & Elkan 2002). This popular non-paramettrc calibration techniques consists in fitting non-decreasing piecewise constant function to the output of an uncalibrated classifier. As for Platt scaling, the goal is learning a function q = _()), such that is a calibrated probability. Isotonic regression learns g by minimizing the square loss Σi= (g. y1) under the constraint that must be piecewise constant (Guo et al. 2017
A.2 CALIBRATION DIAGRAMS: INSTANCES PER BIN
We present in Figure the total count of instances for each bin used in the calibration plots included in Figure 2 As expected, calibration considerably helps spreading out instances across bins, whereas in uncalibrated scenarios instances are squeezed the first or last bins.
A.3 IMPACT OF MODEL HYPERPARAMETERS: 7) AND EMBEDDING DIMENSIONALITY
In Figure we report the impact of negative/positive ratio n and the embedding dimensionality k. Results show that the embedding size k has higher impact than the negative/positive ratio n-- We observe that calibrated and uncalibrated low-dimensional embeddings have worse Brier score. Results also show that any >> 50 does not improve calibration anymore. The negative/positive ratio y follows a similar pattern: choosing 77 10 does not have any effect on the calibration score.
A.4 POSITIVE BASE RATE EXPERIMENTS: LINK PREDICTION PERFORMANCE
In Table 5 present the traditional knowledge graph embedding rank metrics: MRR (mean recip- rocal rank), MR (mean rank) and Hits 10 (precision at the top-10 results). We report the results for all datasets and models used in the main text, which appear in Table Table 4nd Figure
12

---

Published as a conference paper at ICLR 2020
Probability Distribution FB13,ComplEx
Probability Distribution WN11, CON1,.Co
Probability Distribution YAGO39K,ComplEX-N.
20K
40K
Uncalibrated: expit
Uncalibrated: expit
10K
Uncalibrated: expit
10K
20K
5K
0K
0.25
0.50
1.00
0.00
0.25
0.50
0.75
1.00
0K
0.00
0.25
0.50
0.75
1.00
Platt scaling: ground truth
AMA
Platt scaling: ground truth
6K
A64
Platt scaling: ground truth
10K
5K
2K
2K
24
0K
10,00.00 00 00700000000000000000000000000000
0K
0K
0.00
0.25
0.50
0.75
1.00
0.00
0.00
0.25
0.50
0.75
1.00
10K
20K
2..
Platt scaling: synthetic
scaling: synthetic
2.S
Platt scaling: synthetic
4K
5K
10K
2K:
0K
Ad
0K
336
05 mu 05A
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
Isotonic: ground truth
15K
MMM
Isotonic: ground truth
6K
Isotonic: ground truth
20K
10K
4K
10K
5K
2K
OK
0K
0K
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Isotonic: synthetic
15K
Isotonic: synthetic
Isotonic: synthetic
6K
20K
10K
4K
10K
09
2K
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Mean predicted value
Mean predicted value
Mean predicted value
Figure 4: Histograms show the total count instances for each bin used by calibration plots presented in Figure Best viewed in colors.
13

---

Published as a conference paper at ICLR 2020
WN11
WN11
0.50
0.45
0.40
Brier score
0.35
Platt scaling
scaling
0.30
Isotonic
Isotonic
0.25
Uncalibrated scores
Uncalibrated scores
0.20
0.15
0.10
0.05
20
40
60
80
100
100
200
300
400
500
YAGO39K
FB13
0.50
0.45
0.40
Brier score
0.35
+pe Platt scaling
+xee+ Platt scaling
0.30
Isotonic
Isotonic
0.25
Uncalibrated scores
Uncalibrated scores
0.20
0.15
0.10
0.05
100
100
300
YAGO39K
YAGO39K
0.50
0.45
0.40
Brier score
0.35
+ke Platt scaling
<Xk Platt scaling
0.30
Isotonic
Isotonic
0.25
Uncalibrated scores
Uncalibrated
0.20
0.15
0.10
0.05
100
200
300
400
500
Eta
Embedding size
Figure 5: Impact y (eta) and k (embedding size) on the Brier score. We used TransE and the Self-Adversarial loss for all datasets. Best viewed in colors.
MR
MRR
Hits@ 10
TransE
2289
.155
.309
DistMult
10000
.045
.081
WNLI
ComplEx
13815
.054
.094
HolE
13355
.017
.035
TransF
3431
.266
.394
DistMult
6667
.183
.337
FB13
ComplEx
8937
.018
.039
HolE
8937
.018
.039
TransE
244
.169
.319
DistMult
635
.306
.620
YAGO39K
ComplEx
1074
.531
.753
HolE
922
.101
.189
WN18RR
ComplEx 4111
.066
.583
FB15K-237
ComplEx 183
.320
.499
Table Standard filtered metrics for knowledge graph embeddings models. The models are imple- mented in the same codebase and share the same evaluation protocol. Note that we do not include results from reciprocal evaluation protocols.
14

---

Published as a conference paper at ICLR 2020
A.5 BER-RELATION DECISION THRESHOLDS
We report in Table 6 the per-relation decision thresholds used in Table under the 'Reproduced" column. Note that the thresholds reported here probabilities, as they have been applied the raw scores returned by the model-dependent scoring function Sm (t).
Relation
domainnregee region
6.0069733
_ooain instance has
5,5200339
-5.673306
Relation
membbe..holoym
-6.3117476
couse oc.ch
-3.5680597
membeer_meronym
-5,982978
ethnicity
-3,4997067
part of
-5.798244
gender
-3,4051323
_iiiilar_to
6.8552255
institution
-3,547462
subordinate_nstance..
_subordinate instance_off
-5.47502233
nationality
-3.8507419
-ynset topic
profession
type..of
-6.743014
religion
-3,5918012
WN11
FB13
Relation
Relation
339869606
1.8443029
-3.1118833
-3.4323683
2.9660777
-166325312
2.9241138
4.22111304
-386-10308
20
-4.101904
3,685308
3.
2,866 1993
22
1 832546
3.3280334
-2.0114485
3,0741243
3,1512089 24
3.1950998
2.4524217
11
18720441
29
1242291288
13
- 1,542841
32
-1.3310984
14
22,944544
-2.3231838
+3.384997
-2.0019974
36
-13955651
YAGO39K
Table 6: Relation- Sppccfin-ppecific decision thresholds learned on uncalibrated raw scores (See also Table for report on triple classification results.)
15


Reviews:
[{"review_id": "HkeLm0hpqH", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is the first work that studies probability calibration for knowledge graph embedding models. In the case where ground-truth negatives are available the authors directly use off-the-shelf established calibration techniques (Platt scaling, isotonic regression). When ground-truth negatives are not available they propose to synthetically generate corrupted triples as negatives and use sample weights to guarantee that the frequencies adhere to the base rate.\n\nIn general the paper is well-written and easy to follow. Given that the paper's major contribution is experimental insight, and there are no major technical contributions, I would have liked to see a more in-depth analysis of how some of the key hyper-parameters influence the calibration of a model beyond the type of the loss, and beyond the correlation with embedding quality. Overall, I would be willing to increase the score if the authors perform a more comprehensive experimental analysis.\n\nSuggestions to improve the paper:\n1) I would expect that especially the negatives per positive ratio \\eta, and the dimensionality of the embeddings have a significant impact on model calibration. It would be valuable to experimentally quantify the impact of these key hyper-parameters.\n2) It is currently difficult to judge how well-calibrated are the models from the reliability diagrams/calibration plots since the total counts are not shown (e.g. total number of instances with mean predicted value between 0.4 and 0.5). That is, it could be that deviation from identity is due to small sample effects, i.e. we are estimating the fraction of positives from a handful of instances. Showing the total counts for each bin will help the reader better understand the calibration of the models.\n3) Several questions can be clarified regarding the sample weights:\n3.1) How essential is the proposed weighting scheme? How do the calibration techniques perform when using synthetic negatives with uniform sample weights?\n3.2) How does the proposed weighting scheme relate to the the general problem of calibrating models that have class imbalance?\n4.1) Can we observe significant difference in terms of calibration between translational distance models and semantic matching models, i.e. using distance-based scoring functions vs. using similarity-based scoring functions. If so is there any reason for that? To help answer this question the authors could compare additional models from each group (beyond the three models used in the paper). \n4.2) Are methods that represent entities as random variable to capture uncertainties (e.g. KG2E) better calibrated?\n5) Platt scaling assumes that per-class probabilities are normally distributed, while isotonic regression makes no assumption about the input probabilities. Given that Platt scaling performs worse in the experiments it would be interesting to investigate whether this can be (partly) explained by a deviation from the above assumption.\n6) Results reported in Table 3 are for WN11. It would be valuable to report similar results for the other datasets in the appendix.\n7) it would be beneficial to explore the different procedures proposed in the literature for generating synthetic negatives and their impact on the calibration. \n\nSuggestions to improve the paper that did not impact the score:\n1) On the triple classification task in Table 4, there is a significant gap between the literature results and the reproduced results on FB13 and YAGO39K. Is there an explanation for this? Furthermore, it would be interesting to investigate how much do the per-relation \\tau_i's deviate from 0.5 when they are learned using both non-calibrated and calibrated probabilities.\n2) In Eq. 6 after the second equality shouldn't there be \"N/(w- + N)\" instead of \"N/(w_{-} + PN)\"?  Is the additional P a typo?\n3) It would be nice to make the figures more readable (e.g. when printed in black and white) by using different markers for each line. \n\nEdit: Rating updated to 6 after rebuttal.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxwSZbY9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the calibration of the knowledge graph embedding task with Platt scaling and isotonic regression. This paper is well-written, well-motivated and well-organized. However, my major concern is the novelty of this paper or the contribution.\n\nMajor Concerns:\n\n1. This paper lacks novelty. In this paper, the authors only apply the existing techniques (e.g. Platt Scaling, Isotonic Regression) to tackle the calibration issue, which makes a minor contribution. I suggest that the authors could provide their own method specified to knowledge graph tasks rather than leverage the off-shelf methods.\n\n2. The related work could be enhanced, while the preliminaries could be reduced. Actually, in the area of knowledge graph or natural language processing, the preliminary of this paper is a bit trivial. \n\nMinor Concerns:\n\n1. In Table 2, we can conclude that Iso will be better than Platt in general. However, in the case of FB13 (ComplEx) and YAGO (TransE), the results are again the conclusion. Is this because of the optimization issue? I suggest the authors clearly state the experimental analysis.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkgKWAKc9S", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n1. Summary\nThe paper studies probability calibration for three different knowledge graph embedding methods, with a focus on TransE evaluated on the task of knowledge graph triple classification. It studies Brier and log loss performance of Platt scaling and isotonic regression probability calibration on WN11 for TransE and claims that better calibration yields better performance as measured by mean reciprocal rank. Calibration plots for other datasets are also included as evidence. Furthermore, evidence is presented that probability calibration can lead to better performance for the task of triple classification. The main contributions of the paper also include the adaption of sampling techniques introduced by Bordes et al. (2013) adapted for estimating negatives for probability calibrations.\n\n2. Decision (See the updated decision in the comment below)\n\nProbability calibration is a very relevant issue, particularly in industry and when combining knowledge graph embedding models as external data in other models. Thus I see this work as a valuable contribution to the literature. In particular, I like the analysis from multiple views: Calibration plots, calibration metrics, and model performance. However, there is currently not enough evidence in the paper to make recommendations or judgments about when researchers and practitioners may want to use probability calibration. I also believe the datasets and models are not well tied into the literature, for example, in 2018/2019 I can find 3 papers for triple classification and 9 papers for link prediction as triple/entity ranking and from the data, it is not clear how probability calibration affects the latter. In the current state of the work, I recommend rejecting this work.\n\n3. Further supporting arguments\n\nAs a researcher and practitioner in this area, I know very well that predictions of most knowledge graph embedding models usually live near the decision boundary so that there is little difference between probability or score of a true positive and false positive. Also talking with people in industry, I heard that word embedding models are currently not that useful practically because they make too many useless predictions. This shows me that probability calibration is an important topic and I see this study as an important contribution to the field which is often mindlessly following evaluation metrics. \n\nHowever, from experience I also know that evaluation of knowledge graph embedding methods is not very reliable, that is people often get widely varying results and replication is difficult. Thus it is difficult to trust results if they are not well tied into the literature and compared against multiple datasets and models. This work focuses on three models TranseE, DistMult and ComplEx. DistMult and ComplEx have become models that are viewed as quite reliable to compare against. However, their performance is mostly studied on a learning-to-rank objective on datasets such as FB15k-237 and WN18RR. The authors report Brier score for their synthetic calibration method these datasets, but do not report any modeling results. Inclusions of results on these datasets would greatly improve this work. \n\nThe authors also currently focus on establishing that probability calibration improves the performance of the models. They claim that low Brier score or log loss are tied to good performance, but Pairwise and Multiclass-NLL loses achieve similar Brier/log loss performance while the MRR is double for Multiclass-NLL compared to the Pairwise loss. NLL and Multiclass NLL losses have similar MRR but very different Brier/log loss performance. As such I do not think this claim is sufficiently substantiated. I do not believe it is necessary to establish that better probability calibration is correlated with better model performance. I view the careful study of probability calibration and its effects per se as more useful.\nAs mentioned above, I also believe the results on WN11, FB13, and YAGO39k to not be sufficient to evaluate the effect of probability calibration.\n\n4. Additional feedback\n\nI really like this work. I think adding more results would make this paper great and I would be happy to change my acceptance decision.\n\nAs mentioned above I believe including results on FB15k-237 and WN18RR would make the results easier to interpret. Please also add more results to the table (no need to rerun those experiments, take them from other papers). I really like the analysis of Brier Score/Log loss and MRR. I think if you would extend this it would give very valuable insights into how probability calibration relates to performance.\n\nOne additional experiment which I do not deem critical, but which would improve your work further would be to tie probability calibration into a more practical setting. A setting that is also very interesting to researchers is if probability calibration would affect the results in tasks where you use word embedding models as an external \"knowledge source\". I really like  Kumar et al., 2019[1] since their word embedding model integrated into an entity linking model beats a strong BERT baseline. But I think a study of any task/model of your choice that integrates a knowledge embedding model would be a valuable addition to your work.\n\nAgain as I mentioned above, I do not believe it is critical to show improve performance on these tasks, a study of the effects of probability calibration is valuable in its own right. You might want to slightly pivot into this direction if you have sufficient evidence to make judgments about the effects of probability calibration.\n\nFurther small details: In the introduction, you make specific claims and justify them by citing a survey paper (Nickel et al., 2016). It would be easier for the reader to look up these claims in the source rather than in the survey paper. I believe there is a typo in your derivation in equation (6): the denominator of the second term should be just w-N + N or N(w- + 1).\n\n[1] Zero-shot Word Sense Disambiguation using Sense Definition Embeddings: https://www.aclweb.org/anthology/P19-1568/", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgfSVlTKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors deal with the calibration problem in graph embedding models. They used Platt scaling and isotonic regression in the situation when there is ground truth negatives. They also work when there is no ground truth negatives. In this situation, they proposed a calibration heuristics for synthetically generated negatives. Overall, the approach is not very innovative, but the problem they tackled is under studied. The presentation of the whole paper is ok, although it falls onto preliminary side. Since I did not identify any technical problem so far, I will vote for a weak acceptance, unless I observe more technical issues during the discussion.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method to calibrate a knowledge graph embedding method when ground truth negatives are not available. Essentially, the method relies on generating corrupted triples as negative examples to be used by known approaches (Platt scaling and isotonic regression). 

This is claimed as the first approach of probability calibration for knowledge graph embedding models, which is considered to be very relevant for practitioners working on knowledge graph embedding (although this is a narrow audience). The paper does not propose a wholly novel method for probability calibration. Instead, the value in experimental insights provided.

Some reviewers would have liked to see a more in-depth analysis, but reviewers appreciated the thoroughness of the results in the clear articulation of the findings and the fact that multiple datasets and models are studied. 

There was an animated discussion about this paper, but the paper seems a useful contribution to the ICLR community and I would like to recommend acceptance. 
Decision: Accept (Poster)