Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper discusses the gap between close-world and open-world assumptions, which are two major assumptions for KGs. Even though the missing links in KGs are uncertain in advance, the current ranking-based evaluation metric such as MRR and Hits@K heavily relies on the close-world assumption. It may cause a wrong conclusion when comparing KG completion models. The theoretical analysis through the expectation for the rank of entities based on the model strength shows that when the links in a KG are sparse, the model gets stronger the increase of the metric MRR will be less and less significant. Thus, in this case, there is a possibility that the evaluation result is severely affected by its variance. It may cause inconsistent comparison results between models. Empirical analysis of the artificially created dataset shows that the theoretical analysis is consistent with the observed results.
strengths_and_weaknesses: Strengths:

- The stated problems are theoretically and empirically investigated.
- The information in this paper is essential for creating appropriate test sets in the future since sparse relationships in the test set may make it difficult to evaluate models correctly.

Weakness:

- Although this paper reveals the problem in the current raking-based evaluation metrics, its solution is not presented, excluding the increase of queries in testing.
questions: - Based on your conclusion that large $l$ can reduce the variance problem in the ranking-based metrics, Hits@K is more robust than MRR since it is the same as purging entities with low $l$ in predicted results. Is this understanding correct? Clearing this point is quite beneficial for readers comparing KG models.
- Related to the first question, it seems that just using Hits@1 can avoid the problem. If so, Is there any problem with this choice?
limitations: - This paper reveals the unreliable behavior of the current ranking-based evaluation metrics. However, only increasing test queries are presented as a solution.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper investigated the KGC evaluation framework under the open-world assumption. This is a case where unknown triplets are considered to include many missing facts not included in the training or test sets. The paper consider KGC as a positive-negative classification and deduce an approximation of the expectation of the ranking-based metrics with or without the independence assumption. 

The currently used metrics such as mean reciprocal rank (MRR) and Hits@K, show an unexpected behavior under the open-world assumption. 
Authors point out the focus-on-top property of ranking-based metrics worsen the degradation and inconsistency. 
Considering the variance, the paper shows that the degradation in the reported numbers result in incorrect comparisons between different models, where stronger models may have lower metric numbers. 

strengths_and_weaknesses: The paper discusses a very important research topic of big interest to the community. The comparison with current approach showd improvements point toward true north, which is nice to see. The paper consider KGC as a positive-negative classification and deduce an approximation of the expectation of the ranking-based metrics with or without the independence assumption. 

Opportunities:
The proposed evaluation is still not convincing yet, even though we see improvements to current techniques. The assumption of positive-negative classification model doesn’t seem ideal. Other assumptions doesn’t seems matching a real world problem. This include the ranking in positive and negative parts that is considered random uniformly. 
How to assume freshness, authority, accuracy when adding/detecting a triple when dealing with the open-world.  


questions: How to assume freshness, authority, accuracy when adding/detecting a triple when dealing with the open-world? 
How to detect the accuracy of a triplet when dealing with open domain?
How to deal with bias, etc.?


limitations: Please refer to comments above
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper revisits the rank based metrics used for evaluating knowledge base completion (KBC).

Large KBs, by definition, are incomplete. Models for KBC are evaluated based on their ranking of all entities given a query entity (head entity) and a query relation with the model being penalized for ranking entities that are not present in the held-out test set higher above.

However, the paradox lies in the fact that certain answers for which models are penalized are actually correct, since that fact might not just be present in the KB (open-world setting)

This paper shows that in the open-world setting, the metrics does not capture the true model improvement as the strength of the model increases. Specifically, as the model strength increases, the metrics show a logarithmic trend, i.e. it might be too flat to capture improvements of stronger models (gets worse as sparsity increases) (metric degradation problem)

Also, more importantly, the metrics might favor a model which would have lower performance than another model if evaluated on a KG with no additional missing edges other than the query (metric inconsistency problem).

Lastly, the paper proposes two metrics which focus less on “entities at top” and shows that they can reduce the degaradation and inconsistency problem.
strengths_and_weaknesses: Strengths:

Originality:

1. This paper studies in detail the efficacy of the ranking of evaluation of KBC models in open-world setting. The analysis is well done and highly novel. The paper shows that the evaluations are not perfect and also proposes couple of metrics which can fix the problems. These are important contributions.

Quality:

2. The paper is high quality work in general. All the claims are justified by both theory, simulation and experiments

Clarity:

3. Very clearly written. The paper was very easy to follow.

Significance:

4. The paper makes fundamental contributions in understanding of evaluation metrics and I think the contributions are significant
questions: The MRR in fig 1 looks surprisingly low. For example, for the most powerful model (strength=1), and with high density, the best MRR only reaches around 0.3? But in practice for KGs such as WN18RR, the MRR goes over 0.5. How do you explain the discrepancy?

In Line 13 of abstract, shouldnt it say “with many missing facts”, instead of “with not many missing facts”?

There are certain papers that model open-world setting as a continuously increasing KB where unknown entities and facts are added. Worth citing them for completion - e.g. Probabilistic CBR for Open-world KBC.
limitations: Yes, the authors have addressed the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the evaluation of KG completion under the open-world assumption. Its main contribution is to give a theoretical analysis of common evaluation metrics on the open-world knowledge graph and point out the shortcomings of focus-on-top metrics. In addition, experiments on a closed-world dataset validate the obtained conclusions. 
strengths_and_weaknesses: Pros：

- The paper is well written and the theoretical analysis is easy to understand.

- A fundamental problem of KGC is analyzed, which is beneficial for the development of the field.

Cons:

- There is a work that investigates a similar problem (https://arxiv.org/abs/2108.01387, although its main contribution is not theoretical analysis), but a related discussion is missing.

- Lack of discussion of specific relations. For example, 1-to-1 relations do not have the problem of degradation and inconsistency. However, some 1-to-n relations, especially when n is large, may have this problem. 
questions: If necessary, the authors can choose to discuss the proposed issues in the above *Strengths And Weaknesses* section to help me make a better assessment of this paper.
limitations: I would like this paper to be accepted, but there are several limitations of this article that need to be noted. This paper only analyzes several existing evaluation metrics, and does not propose a better solution. In addition, due to its rich set of high-quality rules, the relational pattern of Family Tree KG is much simpler than that of a large-scale knowledge graph, so some basic models can also get high scores. Experimental analysis on such toy KGs may be biased, although I understand that full labeling of large-scale knowledge graphs is impossible.
review_title: null

Meta-review: metareview: This paper studies the evaluation of knowledge graph completion under an open-world assumption, where the training or test sets may include many unknown missing facts. It shows that the currently most-used metrics may not sufficiently reflect the true model performance and suggests alternative metrics to address this issue.  The reviewers found the problem this paper studies is fundamental, and well investigated both theoretically and empirically, which should benefit the research community in this field, although reviewers still have concerns about the assumption in this study and the lack of experiments on more realistic KGs. Overall the merits of the paper outweigh the drawbacks and an acceptance is recommended.
confidence: Certain
Decision: Accept