Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion

Authors: Harry Shomer, Jay Revolinsky, Jiliang Tang
Primary area: datasets and benchmarks
Keywords: knowledge graphs, graphs, link prediction

Abstract:
Knowledge Graph Completion (KGC) attempts to predict missing facts in a Knowledge Graph (KG). Recently, there's been an increased focus on designing KGC methods that can excel in the *inductive setting*, where a portion or all of the entities and relations seen in inference are unobserved during training. Numerous benchmark datasets have been proposed for inductive KGC, all of which are subsets of existing KGs used for transductive KGC. However, we find that the current procedure for constructing inductive KGC datasets inadvertently creates a shortcut that can be exploited even while disregarding the relational information. Specifically, we observe that the Personalized PageRank (PPR) score can achieve strong or near SOTA performance on most inductive datasets. In this paper, we study the root cause of this problem. Using these insights, we propose an alternative strategy for constructing inductive KGC datasets that helps mitigate the PPR shortcut. We then benchmark multiple popular methods using the newly constructed datasets and analyze their performance. The new benchmark datasets help promote a better understanding of the capabilities and challenges of inductive KGC by removing any shortcuts that obfuscate performance.

Decision: reject
Reviewer scores: 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730204665871

thread_type:
official_review

<thread_object>
creator: reviewer_joyy

<content>
summary:
This paper studies the datasets for inductive KGC. First, it is observed that PPR can have hear SOTA performance on most inductive datasets. Then, some analytical experiments are conducted to study the correlation between PPR and shortest path distance. Based on the observations, the authors propose a new split method through graph participation algorithms.

strengths:
1. The logic in this paper is smooth and easy to capture.
2. The analysis of the relationship between PPR, SDP, and KGC performance is studied in detail.
3. The results of the split generated by the graph partition method are consistent with the detailed analysis.

weaknesses:
1. Even though the evaluation of PPR and SPD on current inductive benchmarks reals some problems and sounds reasonable, the match of these problems with realistic demands is not fully discussed. In particular, there lack of discussion and evaluation of datasets in real-world inductive settings. At the beginning of the introduction, KGC applications include drug discovery, personalized medicine, and recommendations. Is it possible to evaluate some cases of new drugs, new patients, and cold start recommendations? I want to know whether phenomena like the correlation between PPR and SPD exist in these applications.

2. Similar to problem 1, I think the main purpose of recreating the inductive dataset is to match it with its transductive counterparts. In my mind, the transductive datasets also have unrealistic problems.

3. Some details of the graph partition method in data creation are not clear. In particular, I care about how the graph partition method can guarantee that the training and inference datasets have different sets of entities or relations. Maybe I missed some important content, but I failed to find this point in both the main content and the appendix.

4. Another commonly used inductive dataset NELL is not studied in this paper.

questions:
1. Is it possible to evaluate some cases of new drugs, new patients, and cold start recommendations? 

2. Whether phenomena like the correlation between PPR and SPD exist in these applications?

3. How to guarantee the sets of entities or relations are different by graph partitioning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730452156853

thread_type:
official_review

<thread_object>
creator: reviewer_msv8

<content>
summary:
This paper identifies a shortcut in current inductive KGC datasets, where simple methods like Personalized PageRank can achieve strong performance without using relational information. The authors propose a new dataset construction strategy to eliminate this shortcut and benchmark popular KGC methods on these datasets.

strengths:
1. This work provides a clearer understanding of the capabilities and challenges in inductive KGC and the benchmark construction.

2. The manuscript is well-organized and easy to follow.

3. Experiments have verified the effectiveness of the proposed benchmarks.

weaknesses:
1. The technical contribution is kind of limited for a long paper. The applied graph partitioning strategy is straightforward and the research challenge in this paper is not significant.

2. There is a confusing point in the benchmark construction. Why the differences in SPD are beneficial from graph partitioning sampling? Since “the different in mean SPD” (also a typo in line 237) causes this shortcut, why not improve the negative sampling strategy directly? For example, select entities having a much shorter distance from the query entity as negative samples. It could be independent of the graph structure.

3. In my opinion, the shortcut proposed in this paper is mainly caused by the negative-sampled evaluation (distinguishing the positive entity from a fixed number of negative ones). Recent studies, including RED-GNN and NBFNet, already employ the full evaluation (finding the positive entity from the entire entity set). This might be the reason for the superior performance of the two models as well as ULTRA. From this point, have the authors compared the performance differences between the two evaluation settings on new benchmarks?

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703568463

thread_type:
official_review

<thread_object>
creator: reviewer_wyld

<content>
summary:
Inductive link prediction aims to predict missing links in test sets where entities, relations, or both have not been encountered during training. Numerous datasets have been developed as benchmarks for evaluating inductive link prediction methods. This paper identifies a flaw in existing inductive link prediction datasets: a simple, non-trainable method called personalized PageRank (PPR) can achieve performance comparable to complex machine learning approaches. This is largely because current datasets are structured such that the shortest path distance (SPD) between entities in positive test samples is considerably shorter than in negative samples. Consequently, a non-trainable algorithm like PPR, which disregards relation information, can still rank positive samples above negative ones, yielding high performance.
To address this issue, the authors propose new datasets for inductive link prediction, carefully partitioned from transductive datasets to prevent the PPR shortcut. The new benchmark reveals that models perform worse on these updated datasets compared to older ones, highlighting the need for more robust evaluation standards in inductive link prediction.

strengths:
-- Numerous machine learning and AI approaches have been introduced in the literature. To demonstrate the effectiveness of new models compared to older ones, several standard datasets have been established for evaluation and conclusion. However, issues with these datasets can lead to misleading conclusions within the community about model performance. Therefore, the paper's original motivation and the problem it addresses are both significant and valuable.

– Overall, the paper is written well and easy to follow and highlight different aspects. 

– The results are consistent with what claimed, i.e., the performance of existing models on new inductive datasets are lower than older ones and also PPR obtain lower results on new datasets.

weaknesses:
– Some part of the paper requires a better and more detailed explanation, e.g., the description related to page rank which is important to better understand the issue.

– that the deltaSPD of train and inference should be close is understandable. But it is not very clear why this partitioning solve the main raised issue. 

–  the analysis of the tables should be more detailed and cover interpretation of various observations.

questions:
Older datasets have simple patterns than new datasets. Why performance of NBFNet improves on new datasets while these datasets are more challenging?

Why the performance difference of PPR and NBFNet on WN18RR dataset in both old and new versions are the same, but this is not the case for FB237?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730837393826

thread_type:
official_review

<thread_object>
creator: reviewer_tapy

<content>
summary:
The main finding of the paper is that many existing inductive knowledge graph (KG) datasets are constructed in a way that creates significant differences in shortest path distances (SPD) between positive and negative samples. This discrepancy allows non-learning-based heuristics, like Personalized PageRank (PPR), which ignores relation types altogether and is unsuitable for knowledge graph completion (KGC), to achieve performance close to that of SOTA methods. This highlights that existing inductive KG datasets might be too simplistic, thereby failing to adequately test the full inductive capabilities of models.

The authors propose a novel dataset construction strategy using graph partitioning to ensure that the sampled subgraphs maintain the original KG structure while mitigating the introduction of SPD-based shortcuts. The proposed benchmark datasets are shown to be more challenging for inductive models, providing a better platform for evaluating their real capabilities.

strengths:
1. **Addressing a Fundamental Problem in Existing Inductive Datasets**: The paper identifies that the shortest path distance (SPD) shortcut is prevalent among several existing inductive KG datasets. Furthermore, the authors demonstrate that all datasets generated by varying configurations of the existing data generation algorithm suffer from this problem. This indicates that the current inductive data generation strategy is fundamentally flawed, and addressing these flaws is crucial to genuinely evaluate a model's inductive capabilities.

2. **Well-Written and Clear Presentation**: The paper is well written and easy to follow. The problem statement, methodology, and experimental findings are presented in a clear and structured manner.

3. **Effective Remedial Strategy**: The authors propose an intuitive and effective strategy for sampling inductive KGs based on graph partitioning, ensuring that entities within a partition are densely connected while inter-partition connections are minimized. This maintains the original graph's SPD distribution, mitigating shortcuts and providing a fairer benchmark.

4. **Extensive Empirical Study**: The authors conducted an extensive empirical evaluation, providing a thorough analysis of both the existing and newly proposed datasets. The experiments cover a wide range of inductive KG completion methods, adding significant value to the paper.

weaknesses:
**My main concern with this paper is that it misses key citations in Related Work**. Specifically, Gao et al. [1] was a concurrent work to Galkin et al. [2] that first provided a theoretical understanding of what is necessary for solving the (E, R) inductive KGC task. Gao et al. also introduced two new (E, R)-capable methods, ISDEA+ and DEq-InGram, with the latter being an improved version of the original InGram [3]. Additionally, two new (E, R) datasets, PediaTypes and WikiTopics, were proposed for empirical evaluation. Including discussions of all (E, R) capable methods, i.e. InGram (DEq-InGram), ULTRA, ISDEA+, in Section 2 (around lines 134-139) as well as including these datasets and methods in the empirical results would provide a more complete overview of recent advances in inductive KGC research and support a more comprehensive empirical comparison.

questions:
1. **Analysis of Other (E, R) Inductive Datasets**: As discussed in W1, PediaTypes and WikiTopics are two fully inductive cross-domain KG datasets proposed by [1] that were not studied in this paper. Could you consider applying the same analysis to these datasets to determine whether they also exhibit divergent SPD differences between positive and negative samples? Additionally, could you evaluate the performance of PPR on these datasets? Including results for these datasets would significantly enhance the paper's comprehensiveness, particularly regarding its coverage of (E, R) inductive datasets.

2. **Evaluation of Other (E, R) Inductive Methods on New Inductive Dataset**: As discussed in W1, Gao et al. [1] proposed the (E, R) inductive methods ISDEA+ and DEq-InGram. Including these methods in the baseline experiments for Section 5 would provide a more comprehensive comparison with the other methods tested in the paper. Of particular interest is that the paper notes that "InGram struggles in the (E, R) setting" on the newly created dataset based on graph partitioning; therefore, it would be valuable to determine whether DEq-InGram could overcome these struggles or if it still faces similar challenges.

3. **Question about the Inference Setting on the (E, R) Inductive Experiments**: It seems that the results of ULTRA [2] on the new inductive datasets are placed in Figure 6, separate from the main (E, R) results shown in Table 4. Could you clarify why these results are presented separately? Is the distinction due to different inference settings (e.g., zero-shot inference for ULTRA in Figure 6 versus few-shot fine-tuning or test-time adaptation for other methods in Table 4)? If this is the case, explicitly mentioning this distinction in the experimental setup section would help reduce confusion and improve clarity.

**References**

[1] Gao, Jianfei, et al. "Double equivariance for inductive link prediction for both new nodes and new relation types." (2023)

[2] Galkin, Mikhail, et al. "Towards foundation models for knowledge graph reasoning." (2023)

[3] Lee, Jaejun, Chanyoung Chung, and Joyce Jiyoung Whang. "InGram: Inductive knowledge graph embedding via relation graphs." (2023)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work finds that in terms of Hits@10, simply using Personalized PageRank (PPR) can achieve relatively high performance on existing inductive knowledge graph completion (KGC) datasets. To resolve this issue, the authors propose a new strategy for constructing inductive KGC datasets, which utilizes graph partitioning algorithms such as Spectral Clustering or Louvain method. The authors provide new experimental results on their proposed datasets for state-of-the-art inductive methods such as InGram, RED-GNN, and NBFNet.

Unfortunately, the authors violated the anonymity policy. In install.md, there is a code written: "git clone git@github.com:**HarryShomer**/Better-Inductive-KGC.git" (https://anonymous.4open.science/r/KG-PPR-ICLR-D0B7/install.md), which reveals one of the authors' GitHub repository: https://github.com/HarryShomer

Since this link was provided within the anonymized link, it wasn't easy to find this during the desk-reject period. When I tried to reproduce the results myself, I found the install.md file contains this identification information. Therefore, it is likely that the reviewers also know the author's information, which can have affected the review process.

I have also thoroughly read the manuscript and found some critical points described below.
1. Analysis only on Hits@10\
In KGC literature, several standard metrics exist: Mean Reciprocal Rank (MRR), Hits@10, (Hits@5, Hits@3,) and Hits@1. Each of these metrics shows a different perspective of a method. At least, MRR, Hits@10, and Hits@1 can be considered essential metrics to consider. However, this manuscript only mentions Hits@10 and ignores the other metrics. The authors should show their results using diverse evaluation metrics.

2. Insufficient Hyperparameter Tuning for RED-GNN, NodePiece, and InGram\
In the RED-GNN paper, it is mentioned that RED-GNN was tuned on 2880 hyperparameter sets (Section 5). Also, in the InGram paper, the authors mentioned 6912 hyperparameter sets (Appendix C). However, the authors of this manuscript tuned RED-GNN, NodePiece*, and InGram only using 4 hyperparameter sets. Therefore, it is likely that the unsatisfactory results of RED-GNN, NodePiece, and InGram reported in this manuscript can be due to insufficient hyperparameter tuning for these methods. Not using a reasonable number of hyperparameters (and thus potentially making the methods under-tuned) is a critical point and introduces risks of making wrong conclusions. (*NodePiece does not explicitly mention the range of hyperparameters in the original paper, but it is likely that more than 4 hyperparameter sets are considered in their experiments.)

3. Different Experimental Settings for NodePiece and InGram\
This manuscript's experimental setting assumes that the validation set is from the train graph. However, NodePiece and InGram assume a different experimental setting: the validation set is from the inference graph to reduce the gap between the validation and test performance (note that the validation and test sets are disjoint). Even though these different experimental settings can critically affect the models' performance, the authors have modified them without mentioning them in the manuscript. Due to these modifications, the performance of NodePiece and InGram reported in the manuscript is unreliable. The authors should correct the experimental settings for NodePiece and InGram to be identical to their original papers and re-do the experiments.

**ULTRA is in ICLR 2024, not 2023. Please check the reference.