Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a KG benchmark with subsets of different patterns (e.g., inductive, transductive). This is in contrast to existing benchmarks that do not separate the KG triples. While the dataset is built upon existing link prediction datasets, this benchmark can be useful to get new insights about existing link prediction methods. In additional, this work also includes baseline experiments and initial findings on different relational patterns, e.g, the distinctive performance between inductive and transductive subsets. 
strengths: 1. This is the first benchmark that highlights the different reasoning patterns in KG link prediction.
2. The benchmark and the results present here can help future identify the remaining challenges in KB link prediction
weaknesses: The benchmark is mostly based on existing dataset but only provides additional information for each instance (e.g., whether it's inductive).
correctness: The benchmark construction looks standard and sound. The baselines are standard and reasonable for provide some initial insights.
clarity: The paper is clearly written and easy to follow.
relation to prior work: The authors might consider discussing more work related to KB reasoning, e.g., methods than can handle more complex queries. 
additional feedback: N/A
review_title: A fine-grained benchmark for knowledge graph link prediction

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper introduces a new benchmark for knowledge base link prediction based on the FB15k and WN18 datasets. The authors identified several types of reasoning models must employ to solve these datasets, and selected subsets that represent these separate problems. Each subset is to be used in an inductive and a transductive setting. In the inductive setting, the entities of the training and test sets are kept entirely separate; in the transductive setting, entities from the training set can reoccur in the test set. The authors evaluate several models from the literature on each subset.
strengths: The paper introduces several subtasks of two common datasets, that could potentially be highly useful in identifying the specific strengths and weaknesses of different models. The authors evaluate several common as well as recent models from the literature on each subtask. This leads to interesting observations, e.g. a surprising resilience of DistMult compared to other embedding-based approaches (RotatE, TransE) to the move to an inductive setting in Figure 3.
weaknesses: This benchmark uses subsets of the FB15k and WN18 datasets. There are well-known issues of data leakage in these, discussed e.g. for FB15k in [1]. Essentially, some relations are encoded multiple times through e.g. explicit inverses (such as (x, father_of, y) and (y, inverse-father_of, x), with occurrences of the same relation in train and test sets because of this. Corrected versions of the datasets are available in the form of FB15k-237 [1], and WN18-RR [2]. The authors should either use the corrected datasets, or ensure the removal of any potential leakage from their dataset -- the latter can be tested through the NodeFeat and LinkFeat baselines from [1].

A second, lesser issue is the choice of evaluation metric. FB15k and WN18 are typically evaluated through ranking metrics (MRR, hit ratios), while this paper employs AUC-PR. This makes comparisons to older numbers on the full datasets difficult.

[1] Toutanova and Chen, 2015. Observed versus latent features for knowledge base and text inference. CVSC.

[2] Dettmers et al., 2018. Convolutional 2D Knowledge Graph Embeddings. AAAI.
correctness: Unfortunately, the dataset construction is somewhat problematic; see my comments on weaknesses.
clarity: The paper is moderately well-written, but there are issues of clarity when it comes to the reporting of numbers. Some baseline performances are presented through bar diagrams, which are difficult to read -- the numbers are small, and the scale of the axes varies from diagram to diagram. This makes it difficult to understand the relative performance of each strategy in each setting. As far as I can see the aggregate performances for the inductive/transductive settings are presented as bar diagrams, with individual performances on each task presented in a table; it is not clear why this choice was made, and clarity would benefit immensely from moving the aggregates to a table format as well.
relation to prior work: The paper should discuss and relate to work identifying problems with the underlying data; see my comments on weaknesses.
additional feedback: In Table 7 and section 4.3, a comparison between two models on the introduced subsets of FB15k is made. Given that this benchmark includes both FB15k and WN18, it is a bit strange that an equivalent comparison for WN18 is not included.
review_title: Benchmarking dataset based on FB15k and WN18

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper builds several specific Link Prediction datasets and group them into different categories based on the relation patterns and properties. The paper evaluates  state-of-the-art KG embedding models and the results are informative and directive. 
strengths: Adequate properties considerations. The author constructed four categories:  Fully Inductive, Fully Transductive, CountBased Inductive, and either Head or Tail Inductive. Each category is further divided into patterns of Symmetry, Anti-symmetry, Inverse and Inductive, making the evaluation comprehensive.
weaknesses: Some patterns have very little data. I'm afraid the results may be influenced by the data bias.
correctness: In section 3.1.4, the example is not Figure2d and should be Figure 2e. Meanwhile, the figure2d seems to be not a composition relation? 
clarity: From a personal point of view, I suggest the author to refer Table 5 in Page 6 Inductive part or put Table 5 before the Figure 3.  It will make the results clearer. It is also better to use a new section to illustrate the results rather than put them in Section 4.1.
relation to prior work: Previous work to evaluate the LP models' ability are usually too general and cannot analyze the advantages and disadvantages of the model in more detail . This work explores specific patterns and types of dataset to better analyze the performance of the model.
additional feedback: No
review_title: A dataset exploring LP model's ability of different relation patterns.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work creates two datasets for link prediction on KG with novel and motivating pattern formulations. With the defined patterns, each dataset could be divided into 16 sub-datasets. The evaluation strategy is clearly stated and convincing. With the benchmarking results comes detailed analysis.
strengths: 1. A novel perspective of formatting link prediction tasks on KG;
2. Evaluation pipeline is well-defined;
3. The comparisons are rigorous with detailed analysis;
weaknesses: 1. The sub-datasets are evaluated, however, the evaluation of the whole KG might also be interested. more importantly, what's the relation between the performance of the subsets and the whole KG?;
correctness: The data collection process and the evaluation seem correct to me.
clarity: The paper is well written and clearly structured.
relation to prior work: Existing datasets and their limitations are discussed. To the best of my knowledge, the pattern formulation is novel.
additional feedback: 1. Will the pattern formulations of links in KG motivate new prospectives of solving reasoning tasks (and other tasks) on KG?
review_title: Benchmarking link prediction on KG with novel formulation

Meta-review: This paper proposes to split existing datasets to subsets such that we can test whether models for knowledge prediction behave in a way that respects various reasoning properties. All reviewers agree this dataset is useful and the results are interesting and illuminating. Several concerns have been raised that must be addressed in the final version: (a) using versions of the datasets where there is no leakage from train to test, though authors explain this might affect just a small portion of the data (b) improve clarity (c) improve related work and refer to models that try to specifically look at properties such as symmetry and explain how the evaluation here is more comprehensive than prior work.
Decision: Accept