Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This is the first work that studies probability calibration for knowledge graph embedding models. In the case where ground-truth negatives are available the authors directly use off-the-shelf established calibration techniques (Platt scaling, isotonic regression). When ground-truth negatives are not available they propose to synthetically generate corrupted triples as negatives and use sample weights to guarantee that the frequencies adhere to the base rate.

In general the paper is well-written and easy to follow. Given that the paper's major contribution is experimental insight, and there are no major technical contributions, I would have liked to see a more in-depth analysis of how some of the key hyper-parameters influence the calibration of a model beyond the type of the loss, and beyond the correlation with embedding quality. Overall, I would be willing to increase the score if the authors perform a more comprehensive experimental analysis.

Suggestions to improve the paper:
1) I would expect that especially the negatives per positive ratio \eta, and the dimensionality of the embeddings have a significant impact on model calibration. It would be valuable to experimentally quantify the impact of these key hyper-parameters.
2) It is currently difficult to judge how well-calibrated are the models from the reliability diagrams/calibration plots since the total counts are not shown (e.g. total number of instances with mean predicted value between 0.4 and 0.5). That is, it could be that deviation from identity is due to small sample effects, i.e. we are estimating the fraction of positives from a handful of instances. Showing the total counts for each bin will help the reader better understand the calibration of the models.
3) Several questions can be clarified regarding the sample weights:
3.1) How essential is the proposed weighting scheme? How do the calibration techniques perform when using synthetic negatives with uniform sample weights?
3.2) How does the proposed weighting scheme relate to the the general problem of calibrating models that have class imbalance?
4.1) Can we observe significant difference in terms of calibration between translational distance models and semantic matching models, i.e. using distance-based scoring functions vs. using similarity-based scoring functions. If so is there any reason for that? To help answer this question the authors could compare additional models from each group (beyond the three models used in the paper). 
4.2) Are methods that represent entities as random variable to capture uncertainties (e.g. KG2E) better calibrated?
5) Platt scaling assumes that per-class probabilities are normally distributed, while isotonic regression makes no assumption about the input probabilities. Given that Platt scaling performs worse in the experiments it would be interesting to investigate whether this can be (partly) explained by a deviation from the above assumption.
6) Results reported in Table 3 are for WN11. It would be valuable to report similar results for the other datasets in the appendix.
7) it would be beneficial to explore the different procedures proposed in the literature for generating synthetic negatives and their impact on the calibration. 

Suggestions to improve the paper that did not impact the score:
1) On the triple classification task in Table 4, there is a significant gap between the literature results and the reproduced results on FB13 and YAGO39K. Is there an explanation for this? Furthermore, it would be interesting to investigate how much do the per-relation \tau_i's deviate from 0.5 when they are learned using both non-calibrated and calibrated probabilities.
2) In Eq. 6 after the second equality shouldn't there be "N/(w- + N)" instead of "N/(w_{-} + PN)"?  Is the additional P a typo?
3) It would be nice to make the figures more readable (e.g. when printed in black and white) by using different markers for each line. 

Edit: Rating updated to 6 after rebuttal.
review_title: Official Blind Review #6

final_score: rating: 8: Accept
review_content: 
1. Summary
The paper studies probability calibration for three different knowledge graph embedding methods, with a focus on TransE evaluated on the task of knowledge graph triple classification. It studies Brier and log loss performance of Platt scaling and isotonic regression probability calibration on WN11 for TransE and claims that better calibration yields better performance as measured by mean reciprocal rank. Calibration plots for other datasets are also included as evidence. Furthermore, evidence is presented that probability calibration can lead to better performance for the task of triple classification. The main contributions of the paper also include the adaption of sampling techniques introduced by Bordes et al. (2013) adapted for estimating negatives for probability calibrations.

2. Decision (See the updated decision in the comment below)

Probability calibration is a very relevant issue, particularly in industry and when combining knowledge graph embedding models as external data in other models. Thus I see this work as a valuable contribution to the literature. In particular, I like the analysis from multiple views: Calibration plots, calibration metrics, and model performance. However, there is currently not enough evidence in the paper to make recommendations or judgments about when researchers and practitioners may want to use probability calibration. I also believe the datasets and models are not well tied into the literature, for example, in 2018/2019 I can find 3 papers for triple classification and 9 papers for link prediction as triple/entity ranking and from the data, it is not clear how probability calibration affects the latter. In the current state of the work, I recommend rejecting this work.

3. Further supporting arguments

As a researcher and practitioner in this area, I know very well that predictions of most knowledge graph embedding models usually live near the decision boundary so that there is little difference between probability or score of a true positive and false positive. Also talking with people in industry, I heard that word embedding models are currently not that useful practically because they make too many useless predictions. This shows me that probability calibration is an important topic and I see this study as an important contribution to the field which is often mindlessly following evaluation metrics. 

However, from experience I also know that evaluation of knowledge graph embedding methods is not very reliable, that is people often get widely varying results and replication is difficult. Thus it is difficult to trust results if they are not well tied into the literature and compared against multiple datasets and models. This work focuses on three models TranseE, DistMult and ComplEx. DistMult and ComplEx have become models that are viewed as quite reliable to compare against. However, their performance is mostly studied on a learning-to-rank objective on datasets such as FB15k-237 and WN18RR. The authors report Brier score for their synthetic calibration method these datasets, but do not report any modeling results. Inclusions of results on these datasets would greatly improve this work. 

The authors also currently focus on establishing that probability calibration improves the performance of the models. They claim that low Brier score or log loss are tied to good performance, but Pairwise and Multiclass-NLL loses achieve similar Brier/log loss performance while the MRR is double for Multiclass-NLL compared to the Pairwise loss. NLL and Multiclass NLL losses have similar MRR but very different Brier/log loss performance. As such I do not think this claim is sufficiently substantiated. I do not believe it is necessary to establish that better probability calibration is correlated with better model performance. I view the careful study of probability calibration and its effects per se as more useful.
As mentioned above, I also believe the results on WN11, FB13, and YAGO39k to not be sufficient to evaluate the effect of probability calibration.

4. Additional feedback

I really like this work. I think adding more results would make this paper great and I would be happy to change my acceptance decision.

As mentioned above I believe including results on FB15k-237 and WN18RR would make the results easier to interpret. Please also add more results to the table (no need to rerun those experiments, take them from other papers). I really like the analysis of Brier Score/Log loss and MRR. I think if you would extend this it would give very valuable insights into how probability calibration relates to performance.

One additional experiment which I do not deem critical, but which would improve your work further would be to tie probability calibration into a more practical setting. A setting that is also very interesting to researchers is if probability calibration would affect the results in tasks where you use word embedding models as an external "knowledge source". I really like  Kumar et al., 2019[1] since their word embedding model integrated into an entity linking model beats a strong BERT baseline. But I think a study of any task/model of your choice that integrates a knowledge embedding model would be a valuable addition to your work.

Again as I mentioned above, I do not believe it is critical to show improve performance on these tasks, a study of the effects of probability calibration is valuable in its own right. You might want to slightly pivot into this direction if you have sufficient evidence to make judgments about the effects of probability calibration.

Further small details: In the introduction, you make specific claims and justify them by citing a survey paper (Nickel et al., 2016). It would be easier for the reader to look up these claims in the source rather than in the survey paper. I believe there is a typo in your derivation in equation (6): the denominator of the second term should be just w-N + N or N(w- + 1).

[1] Zero-shot Word Sense Disambiguation using Sense Definition Embeddings: https://www.aclweb.org/anthology/P19-1568/
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: This paper focuses on the calibration of the knowledge graph embedding task with Platt scaling and isotonic regression. This paper is well-written, well-motivated and well-organized. However, my major concern is the novelty of this paper or the contribution.

Major Concerns:

1. This paper lacks novelty. In this paper, the authors only apply the existing techniques (e.g. Platt Scaling, Isotonic Regression) to tackle the calibration issue, which makes a minor contribution. I suggest that the authors could provide their own method specified to knowledge graph tasks rather than leverage the off-shelf methods.

2. The related work could be enhanced, while the preliminaries could be reduced. Actually, in the area of knowledge graph or natural language processing, the preliminary of this paper is a bit trivial. 

Minor Concerns:

1. In Table 2, we can conclude that Iso will be better than Platt in general. However, in the case of FB13 (ComplEx) and YAGO (TransE), the results are again the conclusion. Is this because of the optimization issue? I suggest the authors clearly state the experimental analysis.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors deal with the calibration problem in graph embedding models. They used Platt scaling and isotonic regression in the situation when there is ground truth negatives. They also work when there is no ground truth negatives. In this situation, they proposed a calibration heuristics for synthetically generated negatives. Overall, the approach is not very innovative, but the problem they tackled is under studied. The presentation of the whole paper is ok, although it falls onto preliminary side. Since I did not identify any technical problem so far, I will vote for a weak acceptance, unless I observe more technical issues during the discussion.


review_title: Official Blind Review #2

Meta-review: The paper proposes a novel method to calibrate a knowledge graph embedding method when ground truth negatives are not available. Essentially, the method relies on generating corrupted triples as negative examples to be used by known approaches (Platt scaling and isotonic regression). 

This is claimed as the first approach of probability calibration for knowledge graph embedding models, which is considered to be very relevant for practitioners working on knowledge graph embedding (although this is a narrow audience). The paper does not propose a wholly novel method for probability calibration. Instead, the value in experimental insights provided.

Some reviewers would have liked to see a more in-depth analysis, but reviewers appreciated the thoroughness of the results in the clear articulation of the findings and the fact that multiple datasets and models are studied. 

There was an animated discussion about this paper, but the paper seems a useful contribution to the ICLR community and I would like to recommend acceptance. 
Decision: Accept (Poster)