A Benchmark for Semi-Inductive Link Prediction in Knowledge Graphs
Authors: Adrian Kochsiek
Rainer Gemulla
Venue: EMNLP 2023 Findings (2023)
Keywords: semi-inductive
link prediction
knowledge graph
unseen entity
TL;DR: We describe a large-scale benchmark for semi-induct link prediction study the performance of recent approaches.

Abstract: Semi-inductive link prediction (LP) in knowledge graphs (KG) is the task of predicting facts for new, previously unseen entities based on context information. Although new entities can be integrated by retraining the model from scratch in principle, such an approach is infeasible for large-scale KGs, where retraining is expensive and new entities may arise frequently. In this paper, we propose and describe a large-scale benchmark to evaluate semi-inductive LP models. The benchmark is based on and extends Wikidata5M: It provides transductive, k-shot, and 0-shot LP tasks, each varying the available information from (i) only KG structure, to (ii) including textual mentions, and (iii) detailed descriptions of the entities. We report on a small study of recent approaches and found that semi-inductive LP performance is far from transductive performance on long-tail entities throughout all experiments. The benchmark provides a test bed for further research into integrating context and textual information in semi-inductive LP models.

A Benchmark for Semi-Inductive Link Prediction in Knowledge Graphs
Adrian Kochsiek
Rainer Gemulla
University of Mannheim
University of Mannheim Germany rgemmlle@unimmmmheii.. de
Germany
akochsiek@uni-mannhe im. de
Abstract
handled by retraining in principle. For large-scale KGs, however, retraining is inherently expensive and new entities may arise frequently. Therefore, the goal of SI-LP is to avoid retraining and perform LP directly, i.e., to generalize beyond the entities seen during training.
Semi-inductive link prediction (LP) in knowl- edge graphs (KG) is the task of predicting facts new, previously unseen entities based context information. Although new enti- ties can be integrated by retraining the model from scratch principle, such an approach is infeasible for large-scale KGs, where retrain- ing is expensive and new entities may arise frequently. In this paper, we propose and describe a large-scale benchmark to evaluate semi-inductive LP models. The benchmark is based on and extends Wikidata5M: It pro- vides transductive, k-shot, and O-shot LP tasks, each varying the available information from (i) only KG structure, to (ii) including textual mentions, and (iii) detailed descriptions of the entities. We report on small study of re- cent approaches and found that semi-inductive I.P performance is far from transductive per- formance on long-tail tail entities throughout all experiments. The benchmark provides test for further research into integrating con- text and textual information in semi-inductive LP models.
To perform LP for unseen entities, context infor- mation about these entities is needed. The amount and form of context information varies widely and may take the form of facts and/or textual informa- tion, such as an entity mention and/or its descrip- tion. For example, a new user in a social network may provide a name, basic facts such as gender or country of origin, and perhaps a textual self- description.
In this paper, introduce the Wikidata5M-S) benchmark for SI-LP. Our benchmark is based the popular Wikidata5M (Wang al., 2021) bench- mark and has four major design goals: (G1) It ensures that unseen entitics are long tail entities since popular entities (such as, say, Foo Fighters) and/or types and taxons (such as human and organi- zation) are unlikely to be unseen. (G2) It allows to evaluate each model with varying amounts of con- textual facts (O-shot, few-shot, transductive), i.e., explore individual models across range of tasks. (G3) provides controlled amount of textual in- formation (none, mention, full description), where each setting demands different modeling capabil- ities. Finally, (G4) the benchmark is large-scale so that retraining is not suitable approach. All prior SI-LP benchmarks violate at least one of these criteria.
1 Introduction
A knowledge graph (KG) is a collection of facts describing relations between real-world entities, Facts are represented in the form of subject- relation-object triples such (Dave Grohl, mem- berOf, Foo Fighters). In this paper, we consider link prediction (L.P) tasks, i.e., the problem of infer- ring missing facts the KG. LP can transductive (TD; all entities known a priori), semi-inductive (SI; some entities known a priori), and inductive (no entities known priori). We concentrate on semi-inductive and transductive LP.
We report on small experimental study with recent LP approaches. In general, we found that
SI performance was far behind TD perfor- mance in all experiments for long-tail entities,
SI-LP focuses on modeling entities that known or unseen during LP, such out-of-KG entities (not part or not yet part of the KG) or newly created entities, c.g., a new user, product, or event. Such previously unknown entities can be
2. there was generally trade-off between TD and SI performance,
textual information was highly valuable,

---

4. proper integration of context and textual infor- mation needs further exploration, and
Transductive
Semi-inductive
Train
Valid
Valid
Test
Triples
20,600,187
4,983
4,977
5,500
5,500
5. facts involving less common relations pro- vided more useful context.
Entitics
4,593,103
7,768
7,760
3.722
3,793
Entitics unseon Relations
0
0
500
500
822
217
211
126
115
Our benchmark provides directions and a test bed for further research into SI-LP.
Table
I: Statistics of the Wikidata5M-SI splits.
2 Related Work
did that by simply removing all triples involving unseen entities.
Multiple SI-LP datascts have been proposed in the literature. The benchmarks of Daza et al. (2021), Albooych et al. (2020), and Galkin et al. (2021) are obtained by first merging the splits of smaller trans- ductive LP datasets and subsequently sampling un- seen entities uniformly construct validation and test splits. These benchmarks do not satisfy goals G1-G4. Shi and Weninger (2018) follow a similar approach but focus on only 0-shot evaluation based on textual features. Xie et al. (2016) and Shah et al. (2019) select entities from Freebase with connec- tion to entities in 151 (Bordes et al., 2013), also focussing on O-shot evaluation using rich textual descriptions. These approaches do not satisfy G2 and G3. Finally, Wang et al. (2019) and Hamaguchi et al. (2017) uniformly sample lest triples and mark occurring entities as unseen. These approaches do not focus on long-tail entities (and, in fact, the ac- cumulated context of unseen entities may be larger than the training graph itself) and they do not sat- isfy G1-G3.
Unseen entities. To ensure that unseen entities in the semi-inductive splits are from the long tail (G1), we only considered entities of degree 20 or less. To be able to provide sufficient context for few-shot tasks (G2), we further did not consider entities of degree 10 or less. In more detail, we sam- pled 500 entities of degrees 11 11-20 (stratified sam- pling grouped by degree) for each semi-inductive split. All sampled entities, along with their facts, were removed from the train split. Note that these entities (naturally) have different class distribu- tion than all entities; see Sec. A. for details.
Tasks and metrics. For TD tasks, we follow the standard protocol of Wikidata5M. To construct SI tasks, we include 11 of the original facts of each unseen entity into its SI split; each split thus con- tains 5,500 triples. This enables up to 10-shot SI tasks (1 fact to test, up to 10 facts for context). For entities of degree larger than 11, we select the 11 facts with the most frequent relations; see Tab. 2 for example. The rationale is that more common relations (such as instanceOf country) may be considered more likely to be provided for unseen entities than rare ones (such as militaryBranch or publisher). We then construct a single k-shot task for each triple (s, p, o) in the SI split as follows. When, say, 8 is the unseen entity, we consider the LP task (s, p, ?) and provide is additional facts of form (s,p', of) as context. Context facts are se- lected by frequency above, but we also explored random and infrequent-relation context in our study. Models are asked provide a ranking of predicted answers, and we determine the filtered mean re- ciprocal rank (MRR) and Hits@K K of the correct answer (o).
There are also several of fully-inductive LP benchmarks (Teru et al., 2020; Wang et al., 2021) involving KGs. While SI-LP aims to connect un- seen entities to an existing KG, fully-inductive LP reasons about a new KG with completely separate entities (but shared relations). We do not consider this task in this work.
3 The Wikidata5M-SI Benchmark
Wikiduta5M-SI based on the popular Wiki- data5M (Wang et al., 2021) benchmark, which is induced by the 5M most common entities of Wikidata. Our benchmark contains transductive and semi-inductive valid/test splits; see Tab. 1 for an overview. Generally, we aimed to keep Wikidata5M-SI as close as possible to Wikidata5M. We did need to modify the original transductive valid and test splits, however, because they unin- tentionally contained both seen and unseen entities (i.e., these splits were not fully transductive). We
Textual information. For each entity, we pro- vide its principal mention and a detailed description (both directly from Wikidata5M); see Tab. 2. This allows to differentiate model evaluation with vary- ing amounts of textual information per entity (G3); (A) atomic, i.e., no textual information, (M) men-

---

ID
Q18918
Mention
Sam Witwer
Description
Samuel Stewart Witwer (born October 20, 1977) is an American actor and mu- sician. He is known for portraying Crashdown in Battlestar Galactica, Davis Bloome in Smallville, Aidan Waite in Being Human, and Ben Lockwood in Supergirl. He voiced the protagonist Galen Marek / Starkiller in Star Wars: The Force Unleashed, the Son in Star Wars: The Clone Wars and Emperor Palpatine in Star Wars Rebels, both of which he has also voiced Darth Maul.
Context triples
instance of human country of citizenship United States of America occupation musician
M: D:
M:
D:
M:
D: V
occupation actor
M:
D: V
place of birth Glenview
M: X
D: X
given name Samuel given name Sam
M:
D: V
M: V
D: O
cast member Battlestar Galactica
M: D:
cast member Being Human superntturl drama television series cast member Star Wars: The Force Unleashed II
M: D.
M: X D:
cast member The Mist
M: D:
Table 2: Example of entity from the semi-inductive validation set of WiKidata5M-SI. For each triple, we anno- tated whether the answer contained in (V), deducible from (o), or not contained in (x) mention (M) or description (D).
tions only, and (D) detailed textual deseriptions as in (Kochsiek et al., 2023). This differentiation is especially important in the SI setting, as detailed text descriptions might not be provided for unseen entities and each setting demands different model- ing capabilities. In fact, (A) performs reasoning only using graph structure, whereas (D) also bene- fits from information extraction to some extent. We discuss this further in Sec. 5.
approach taken by oDistMult-ERAgg (Albooyeh et al., 2020), which represents unseen entities by aggregating the embeddings of the relations and entities in the context.' A more direct approach is taken by HiuER (Chen et al., 2021), which con- textualizes the query entity with its neighborhood for TD-LP. The approach can be used for SI-LP directly by using masking token (akin to the global bias above) for an unseen entity. We origi- nally planned to consider NodePiece (Galkin et al., 2021) (entity represented by combination of an- chor embeddings) and NBFNet (Zhu et al., 2021) (a GNN-hased LP model); both support SI-LP di- rectly. However, the available implementations did not scale Wikidata5M-SI (out of memory).
4 Semi-Inductive Link Prediction Models
We briefly summarize recent models for SI-LP; considered these models in our experimental study.
Graph-only models. ComplEx (Trouillon et al., 2016) the best-performing transductive KGE model on Wikidata5M (Kochsiek et al., 2022). To use ComplEx for SI-LP, we follow an approach e
Text-based models. As baseline approach to integrate textual information directly into KGE models, we consider the approach explored the
explored by Jambor et al. (2021). In particular, we represent each entity as the sum of a local em- bedding (one per entity) and global bias embed- ding. For 0-shot, solely use the global bias for the unseen entity. For k-shot, we obtain the local embedding for the unseen entity by performing a single training step on the context triples (keep- ing all other embeddings fixed). An alternative
"To address high memory footprint (Galkin 2021) of oDistMult-ERA we extend it with neighborhood sampling.
'For NBENet (Zhu et al., 2021), the large memory foot- print is inherent to the model: it is full-graph GNN and hard to scale. For NodePiece (Galkin et al., 2021). however, the problem mainly lies in the expensive evaluation. All inter- mediate representations are precomputed, leading to a large memory overhead.

---

WikiKG90M benchmark (Hu al., 2021); see Sec. A.2 for details. The remaining approaches are purely textual. SimKGC (Wang et al., 2022) utilizes two pretrained BERT Transformers: one to embed query entities (and relations) based on their mention or description, and one for tail en- tities. Using a contrastive learning approach, it measures cosine similarity between both represen- tations for ranking. KGT5 (Saxena al., 2022) is a seuuuence-t--equeeee which is trained to generate the mention of the an- swer entity using the mention or description of the query entity and relation as input. Both approaches support 0-shot SI-LP when textual information is provided for the query entity. They do not utilize additional context, however, i.e., do not support s-otot SI-LP. KGTS-context (Kochsiek et al., 2023) is an extension of KGT5, which extends the in- put of KGT5 by the one-hop neighborhood of the query entity and consequently supports k-shot LP directly.
formation was provided for unseen entities, O-shot was not reasonably possible. Without text-based information, context was a necessity. simple enntt--rrtttt average neighborhood the best integration of context.
Mentions. Integrating mentions did not improve performance on its own, as provided text informa- tion was still limited. However, additionally provid- ing context information during inference (KGT5- context) simplified the learning problem and im- proved TD performance significantly. But for O- shot, the limited text information provided with mentions allowed for reasonable performance. To analyze what information conveyed for O-shot, we annotated 100 valid triples; see Tab. 4. In 10% of cases, the answer was already contained in the mention, and it was deducible in at least 7%. This enabled basic reasoning without any further infor- mation. In contrast to the TD setting, KGTS outper- formed its context extension. KGT5-context was reliant on context which was lacking especially dur- ing O-shot. This showed a trade-off between best performance in the SI and TD setting. This trade- off could be mitigated by applying (full and partial) context hiding. With such adapted training, KGT5- context reached middle ground with a transduc- tive MRR of 0.366 and O-shot MRR of 0.283.* 4 However, even with full context (10-shot), perfor- mance still only par with KGT5. Therefore, context information did not bring any further bene- fits when text was provided.
link prediction approach,
5 Experimental Study
We evaluated all presented baseline models in the TD and SI setting on the atomic, mentions, and descriptions dataset. Further, evaluated in de- tail which context was most useful and what in- formation was conveyed by textual mentions and descriptions.
Setup. Source code, configuration, and the benchmark itself are available at https://gi thu.. com/uma-pia pin //wk/iik wwwiikidaatamssi.... For further de- tails hyperparameter tuning and training see Sec. A.3.
Descriptions. Further, integrating descriptions improved performance for both settings, TD and SI, considerably; see Tab. 3. Similar to the mentions- only setting, KGT5-context performed best in TD and KGT5 in the SI setting. Applying the same trade-off with context-hidinn hiding reached a middle ground with 0,418 TD-MRR and 0.449 SI-MRR.
Main results. Transductive and SI performance in terms of MRR of all models is presented in Tab. 3; Hits@K K in Tab. 7-9 (Sec. A). Note that overall transductive performance was oftentimes below best reported SI performance. This is due to varying degrees of query entities between both set- tings. Typically, models perform better predicting new relations for an entity (e.g., the birthplace) than predicting additional objects for a known relation (e.g., additional awards won by a person) (Saxena et al., 2022; Kochsiek et al., 2023). For a direct comparison between both settings, we additionally report TD performance on long tail query entities. Atomic. TD performance on the long tail was considerably higher than SI performance. As no in-
Descriptions were very detailed and partially contained the correct answer as well as the same in- formation as contained in context triples; see Tab. 4. Therefore, performance did not further improve with context size. In such cases, models mainly benefit from information extraction capabilities. To judge how much information extraction helps, we grouped performance of KGT5+description in the O-shot setting on validation data into the groups contained, deducible and not contained in descrip-
'In 25%/25%/50% of cases, we hid the full con- text/sampled between 1-10 neighbors/used the full context, respoctively.
'ee define long tail query entities as entities with degree < 10 as in the SI setting.

---

Transductive
Semi-inductive (num. shots)
Pre-
trained
Model
All
Long tail
0
1
3
5
10
ComplEx + Bias Fold in (Jambor et al., 2021)
0.308
0.523
0.124
0,151
0.176
0.190
0.206
no
DistMult+ Avg (Albooyeh al., 2020)
0.294
0.512
0.171
0.246
0.295
0.333
no
HinER (Chen et al., 2021)
0.284
0.512
0.019
0.105
0.153
0.179
0.221
no
DistMult + ERAvg + Mentions
0.299
0.535
0.187
0.235
0.258
0.280
yes
SimKGC (mentions only)
0.212
0.361
0.220
yes
KGT5 (Saxena et al., 2022)
0.281
0.542
0.310
KGTS-context (Kochsick et al., 2023)
0.374
0.678
0.220
0,217
0.236
0.259
0.311
no
DistMult ERAve Descriptions
0.313
0,585
0,278
0.281
0.285
0.292
yes
SimKGC + Descriptions (Wang et al., 2022)
0.353
0.663
0.403
yes
KGT5 Descriptions (Knchsick et al., 2023)
0.364
0.728
0.470
no
KGT5-context Descriptions (Kochsiek et al., 2023)
0.420
0.777
0.417
0,420
0.416
0.420
0.437
n10
Table 3: Transductive and semi-inductive link prediction results in terms of MRR on the dataset Wikidata5M-SL The first group presets results on the atomic, the second on the mentions and the third on the descriptions dataset. Best per TD/SI in bold. Best per group underlined.
Mention
Description
Conclusion
Contained
10%
44%
We proposed the new WikiData5M-SI large-scale benchmark for semi-supervised link prediction. The benchmark focuses on unseen entities from the long tail and allows to evaluate models with vary- ing and controlled amounts of factual and textual context information. In our experimental evalua- tion, we found that semi-inductive LP performance fell behind transductive performance for long-tail entities in general, and that detailed textual informa- tion was often more valuable than factual context information. Moreover, current models did not in- tegrate these two types of information adequately, suggesting a direction for future research.
Deducible
7%
10%
Not contained 83%
46%
Table 4: Information about query answer contained in mentions and descriptions. Annotated for 100 sampled triples from 0-shot valid. For an example, see Tab. 2.
Context selection
5
Most common
0.217
0.236
0.259
Least common
0.253
0.273
0.290
Random
0.237
0.260
0.281
Table 5: Influence of context selection. Semi-inductiiv test MRR of KonSec.
Limitations
This study was performed on Wikidata5M-SL. i.c.. subset of single knowledge graph. Model per- formance and insights may vary if graph structure and/or availability and usefulness of mentions and description is different. In particular, the entity descriptions provided with Wikidata5M-SI partly contained information relevant for link prediction so that models benefited from information extrac- tion capabilities.
tion; see Fig. in Sec. A. When contained, the correct answer was extracted 70% of cases. Context selection. We selected the most com- mon relations context triples so far, as this may be a more realistic setting. To investigate the ef-
feet of this selection upproach, we compared the default selection of choosing most common rela- tions to least common and random. Results for KGT5-context are shown in Tab. 5; for all other models in Tab. 10 in Sec. A. We found that the less common the relations of the provided context, the better the SI performance. More common context relations often described high-level concepts, while less common provided further detail; see the exam- ple in Tab. 2. While more common context may be more readily available, less common context was more helpful to describe a new entity.
Ethics Statement
This research adapts publicly available data, bench- marks, and codebases for evaluation. We believe that this research was conducted in an ethical man- ner in compliance with all relevant laws and regu- lations.

---

References
19-23, 2022, Proceedings, Part II, pages 138-154. Springer.
Marjan Albooyeh, Rishab Goel, and Seyed Mehran Kazemi. 2020. Out-of-sample representation learn- ing for knowledge graphs. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 2657-2666.
Adrian Kochsiek, Apoorv Saxena, Inderjeet Nair, and Rainer Gemulla. 2023. Friendly neighbors: Con- textualized
link prediction. In Proceedings of the 8th Workshop on Representation Learning for NLP.
Antoine Bordes, Nicolas Usumier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Neural Information Processing Systems (NIPS), pages 1-9.
Apoorv Saxena, Adrian Kochsick, and Rainer Gemulla. 2022. grquence-to-sequenee knowledge graph com- pletion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 2814-2828.
Samuel Broscheit, Daniel Ruffinelli, Adrian Kochsiek, Patrick Betz, and Rainer Gemulla. 2020. LibKGE A knowledge graph embedding library for repro- ducible research. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 165- 174.
Haseeb Shah. Johannes Villmow, Adrian Ulges, UI- rich Schwanecke, and Faisal Shafait. 2019. An open-world extension to knowledge graph comple- tion models. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3044- 3051.
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. 2021. Hitter: Hi- erarchical transformers for knowledge graph embed- dings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10455100477
Baoxu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32.
Daniel Daza, Michael Cochez, and Paul Groth. 2021. Inductive entity representations from text via link prediction. In Proceedings of the Web Conference 2021, pages 798-808.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33:16857- 16867.
Mikhail Galkin, Etienne Denis Jiapeng Wu, and William Hamilton. 2021. Nodepiece: Compo- sitional and parameter-efficient representations of large knowledge graphs. In International Confer- ence on Learning Representations.
Komal Teru, Etienne Denis, and Will Hamilton. 2020. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, pages 9448-9448999 -957-997
Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. 2017. Knowledge transfer for out-of- kew.oo-knowledge-baeee entities: a graph neural net- work approach. In Proceedings of the 26th Inter- national Joint Conference on Artificial Intelligence, pages 1802-1808.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Interna- tional conference on machine learning, pages 2071- 2080.
I. Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. Simkge: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4281 4294.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. 2021. Ogb-Ise: A large-scale challenge for machine learning on graphs. Advances in Neural Information Processing Systems, 35.
Dora Jambor, Komal Teru, Joelle Pineau, and William I. Hamilton. 2021. Exploring the limits of few-shot link prediction in knowledge graphs. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2816-2822.
Peifeng Wang. Jialong Han, Chenliang Li, and Rong Pan. 2019. Logic attention based neighborhood ag- gregation for inductive knowledge graph embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7152-7159.
Adrian Kochsiek, Fritz Niesel, and Rainer Gemulla. 2022. Start small, think big: On hyperparameter optimization for large-scale knowledge graph em- beddings. In Machine Learning and Knowledge Discovery Databases: European Conference. ECML PKDD 2022, Grenoble, France, September
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transac- tions of the Association for Computational Linguis- tics, 9:176-194

---

Ruobing Xie. Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016. Representation learning of
A.2 Integrating Text into KGE Models
To integrate text into traditional KGE models, we follow the baseline models of the WikiKG90M link prediction challenge (Hu et al., 2021). We embed mentions combined with descriptions us- ing MPNet (Song et al., 2020), concatenate the resulting descriptions embedding with the entity embedding. and project with linear layer for the final representation of the entity. combination with oDistMult-ERAvg (Albooyeh al., 2020), we apply the aggregation of neighboring entities and relations on the entity embedding part only. The resulting aggregation is then concatenated with its description and finally projected.
knowledge graphs with entity descriptions. In Pro- ceedings of the AAA/ Conference on Artificial Intel- ligence, volume 30.
Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jian- feng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowl- edge bases. In Proceedings of the International Con- ference Learning Representations (ICLR) 2015.
Zhaocheng Zhu, Zuobai Zhang. Louis-Pascal Xhon- neux, and Jian Tang. 2021. Neural bellman- ford net- works: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems, 34.
Appendix
This approach is closely related BLP (Daza et al., 2021). The main differences BLP are:
A.1 Distribution of Unseen Entities
1. Hu et al. (2021) use MPNet, BLP uses BERT,
Long-tail entities have a different distribution than entities from the whole KG; see Tab. for an overview of the distribution shift for the top 10 entity types. This difference is natural. In particu- lar, high-degree entities in KG such as Wikidata often refer to types/taxons (c.g, human, organiza- tion, ...) as well as popular named entities (e.g.. Albert Einstein, Germany, ...). These entities are fundamental to the KG and/or of high interest and have many facts associated with them. For this reason, they do not form suitable candidates for benchmarking unseen or new entities. In addition, removing high-degree entities for the purpose of evaluating SI-LP is likely to distort the KG (e.g.. consider removing type "human" or "Germany"). In contrast, Wikidata5M-SI focuses on entities for which knowledge is not yet abundant: long-tail tail tities are accompanied by no or few facts (at least initially) and our SI-LP benchmark tests reasoning capabilities with this limited information.
2. In combination with DistMult-ERAvg, we concatenate learnable "structural embed- ding" to the CLS embedding of the language model, whereas BLP does not.
A.3 Experimental Setup
For hyperparameter optimization for Com- plEx (Trouillon al., 2016), DistMult (Yang et al., 2015), and HitER (Chen et al., 2021), we used the multi-fidelity approach GraSH (Kochsiek et al., 2022) implemented in LibKGE (Broscheit et al., 2020) with 64 initial trials and trained for up to 64 epochs. For fold-in, we reused training hyperparameters and trained for single epoch the provided context. For text-based approaches, we used the hyperparameters and architectures proposed by the authors for the transductive split of Wikidata5M. We trained on up to 5 A6000-GPUs with 49GB of VRAM.

---

WikidataID
Mention
commune of Italy All entities
Long-tail entities
Q5
human
39%
61% %
Q11424
film
3%
8%
Q484170
commune of France
1%
Q482994
album
3%
1%
Q16521
taxon
9%
1
Q134556
single
1%
1%
Q747074
0%
1
Q2074737
municipality of Spain 0%
1%
Q571
book
1%
1%
Q7889
video game
1
1%
Table 6: Distribution of top 10 entity types over long-tail entities with degree between 11 and 20 compared to all entities.
Correct prediction
Incorrect prediction
35
30
Percent
25
15
10
5
Contained
Deducible
Not contained
Figure I: Number of correct (rank=l) l) and incorrect predictions by KGTS+dessriptions on annotated examples per annotation label.
Semi-inductive (num. shots)
Model
Trans.
10
Complex Bias + Fold in (Jambor et al., 2021)
0.260
0.058
0.097
0.118
0.124
0.132
DistMult ERAvg (Albooych et al., 2020)
0.237
0.115
0.151
0.185
0.209
HitER (Chen et al., 2021)
0.234
0.005
0.076
0.115
0.132
0.153
DistMult ERAvg Mentions
0.239
0.106
0.142
0.153
0.167
SimKGC (mentions only)
0.182
0.187
KGT5 (Saxena al., 2022)
0.249
0.263
KGT5-context (Kochsiek et al., 2023)
0.347
0.184
0.177
0.195
0.218
0.263
DistMult ERAvg Descriptions
0.252
0.152
0.153
0.153
0.161
SimKGC Descriptions (Wang et al., 2022)
0.311
0.349
KGT5 + Descriptions
0.332
0.430
KGTS-context Descriptions
0.400
0.379
0.382
0.373
0.378
0.393
Table Transductive and semi-inductive link prediction results in of H the dataset WiKidataSM-SI.

---

Semi-inductive (num. shots)
Model
Trans.
3
5
10
ComplEx + Bias + Fold in (Jambor et al., 2021)
0.337
0.165
0.180
0.202
0.219
0.242
DistMult + ERAvg (Albooych et al., 2020)
0.328
0.190
0.292
0.352
0.401
HinER (Chen et al., 2021)
0.309
0.013
0.109
0.158
0.188
0.242
DistMult + ERAvg + Mentions
0.332
0.239
0.289
0.314
0.340
SimKGC (mentions only)
0.223
0.227
KGTS (Saxena et al., 2022)
0.296
0.332
KGT5-context (Kochsiek et al., 2023)
0.390
0.236
0.234
0.257
0.278
0.335
DistMult + ER + Descriptions
0.344
0.368
0.373
0.378
0.380
SimKGC (Wang et al., 2022)
0.367
0.421
KGT5 Descriptions
0.385
0.490
KGT5-context Descriptions
0.432
0.441
0,443
0.443
0.447
0.463
Table 8: Transductive and semi-inductive link prediction results in terms of H@3 the dataset Wikidata5MSSI.
Semi-inductive (num. shots)
Model
Trans.
5
10
ComplEx Bias Fold in (Jambor et al., 2021)
0.387
0.231
0.245
0.282
0.309
0.336
DistMult + ERAvg (Albooyeh al., 2020)
0.389
0.270
0.409
0.493
0.564
HinER (Chen al., 2021)
0.376
0.050
0.157
0.226
0.270
0.359
DistMult + ERAvg + Mentions
0.411
0.320
0.392
0.440
0.478
SimKGC (mentions only)
0.266
0.283
KGT5 (Saxena et al., 2022)
0.344
0.398
KGTS-context (Kochsiek et al., 2023)
0.423
0.293
0.295
0.310
0.336
0.400
DistMult + ERAvg + Descriptions
0.425
0.465
0.472
0.484
0.491
SimKGC (Wang et al., 2022)
0.432
0.504
KGT5 Descriptions
0,416
0.544
KGT5-context + Descriptions
0.455
0.484
0.489
0.489
0.495
0.516
Table 9: Transductive and semi-inductive link prediction results in terms of H@ 10 on the dataset Wikidata5M-SI.

---

Model
Context selection
Most common
0.151
0.161
0.168
ComplEx + fold-in
Least common
0.166
0.185
0.195
Random
0.164
0.187
0.196
Most common
0.171
0.246
0.295
DistMult ERAvg
Least common
0.217
0.299
0.323
Random
0.215
0.303
0.318
Most common
0.187
0,235
0.258
oDistMult + ERAvg + Mentions
Least common
0.237
0.274
0.279
Random
0.232
0.265
0.272
Most common
0.105
0.153
0.179
HittR
Least common
0.151
0.195
0.216
Random
0.136
0.190
0.206
Most common
0.217
0.236
0.259
KGT5-context
Least common
0.253
0.273
0.290
Random
0.237
0.260
0.281
Most common
0.420
0.416
0.420
KGT5-context Desc.
Least common
0.423
0,424
0.430
Random
0.422
0.430
0.430
Table 10: Influence context selection. Semi- indiiinductive test MRR. Best per model in bold.


Reviews:
[{"review_id": "aQvext1Kz0", "reviewer": "Reviewer_NEnw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and constructive criticism. We provide thoughts on each point below, and we will update the paper accordingly.\n\n> The paper lacks a clear motivation for why semi-inductive link prediction is an important problem and how it contributes to the broader field of knowledge graph research.\n\nSemi-inductve link prediction (SI-LP) focuses on modelling entities that are unknown or unseen during link prediction, such as out-of-KG entities (not part or not yet part of the KG) or newly created entities (e.g., a new user/product/event). A straightforward solution to handle unseen entities is to retrain the model and apply transductive inference. Unseen entities may arise frequently, however, and retraining the model is expensive for large KGs. The goal of SI-LP is to avoid retraining and perform LP directly, i.e., generalize beyond the entities seen during training. The goal of our benchmark is to provide a testbed for SI-LP methods. In addition, the benchmark clearly separates and evaluates different scenarios w.r.t. the information provided for unseen entities at test time: from zero- to few shot on the one hand and from no text to full textual descriptions on the other hand.\n\n> The construction of the benchmark is simple and does not require any new techniques or methods.\n\nWe agree.\n\n> The paper does not provide a strong justification for why the benchmark is novel or important.\n\nExisting benchmarks lack one or more of the characteristics outlined in our paper: sensible selection of unseen entities, varying amount of contextual facts, varying amount of textual information, and large scale. The goal of our benchmark is to satisfy these critera and provide baseline results across the different settings; it is the first benchmark that does so. An additional goal was to stay close to the Wikidata5M transductive benchmark so that the performance of transductive evaluation is comparable.\n\n> The abstract is brief and does not provide a strong overview of the contributions, methodology, or results of the paper.\n\nWe will expand the abstract accordingly.\n\n> The experimental study focuses on a single knowledge graph (Wikidata5M), which may limit the generalizability of the findings to other knowledge graphs.\n\nWe agree; this is a limitation of our work. We will provide scripts that allow for creation of similar benchmarks for other knowlege graphs, which facilitates the creation of benchmarks for other KGs in future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a new benchmark, Wikidata5M-SI, for semi-inductive link prediction in knowledge graphs. It addresses several important design goals, including focusing on long-tail entities, evaluating models with varying amounts of context and textual information, and providing a controlled setting for experimentation.\n- The paper presents a thorough evaluation of various state-of-the-art models on the benchmark, providing insights into their performance in different scenarios.\n- The study analyzes the impact of different types of contextual information (mentions, descriptions) and their integration into models for link prediction.\n- The paper discusses trade-offs between transductive and semi-inductive performance and highlights the potential of leveraging textual information for improving link prediction in knowledge graphs.\n", "reasons_to_reject": "\n- The paper lacks a clear motivation for why semi-inductive link prediction is an important problem and how it contributes to the broader field of knowledge graph research.\n- The construction of the benchmark is simple and does not require any new techniques or methods. The paper does not provide a strong justification for why the benchmark is novel or important.\n- The abstract is brief and does not provide a strong overview of the contributions, methodology, or results of the paper.\n- The experimental study focuses on a single knowledge graph (Wikidata5M), which may limit the generalizability of the findings to other knowledge graphs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a benchmark named Wikidata5M-SI for semi-inductive link prediction (LP) in knowledge graphs (KGs), specifically focusing on predicting facts for new, unseen entities. The benchmark provides a versatile evaluation framework for transductive, k-shot, and 0-shot LP tasks with varying levels of available information, including KG structure, textual mentions, and detailed entity descriptions. The study evaluates several recent approaches using this benchmark and finds that semi-inductive LP performance lags behind transductive LP across all experiments. The benchmark highlights the need for further research in integrating context and textual information for improving semi-inductive LP models. The authors compare different context selection strategies and explore the value of textual information in aiding LP predictions. They also identify trade-offs between TD and SI performance and discuss directions for future research.\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jC4ZDsdzVA", "reviewer": "Reviewer_WJjs", "summary": "", "questions": "1. Why not also contain the inductive learning setting?\n\n2. I am also wondering the performance of LLM in directly predicting the link with proper prompts.  I suppose LLMs will not be perfect but have a decent performance. If an LLM can already do this task well, the task will be not so meaningful and the authors have to think about how to make the dataset more useful.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and constructive criticism. We provide thoughts on each point below, and we will update the paper accordingly.\n\n> Why not also contain the inductive learning setting?\n\nThe original Wikidata5M benchmark already contains a fully-inductive split, so there is no need. Moreover, the fully-inductive setting is very different to both transductive and semi-inductive LP; methods and applications differ considerably. In particular, fully-inductive LP evaluates the transfer of a trained model to a new KG that does not share *any* of the entities of the training KG (but only the relation types). Our focus, in contrast, is on providing a benchmark for new entities, i.e., entities that are unknown or unseen during link prediction. Examples include out-of-KG entities (not part or not yet part of the KG) or newly created entities (e.g., a new user/product/event).\n\n> I am also wondering the performance of LLM in directly predicting the link with proper prompts.\n\nThe KGT5 and SimKGC models are SOTA approaches based on language modeling, and KGT5-context can be seen as a method to generate suitable prompts. We did not evaluate pretrained LLMs such as GPT-4 as these models are likely to have seen test data during training.\n\n> Why are the unseen entities long-tail?\n\nHigh-degree entities in a KG such as Wikidata often refer to types/taxons (e.g, human, organization, ...) as well as popular named entities (e.g., Albert Einstein, Germany, ...). These entities are fundamental to the KG and/or of high interest and have many facts associated with them. For this reason, they do not form suitable candidates for unseen or new entities. In addition, removing high-degree entities for the purpose of evaluating SI-LP is likely to distort the KG (e.g., consider removing type \"human\" or \"Germany\"). In contrast, our benchmark focuses on entities for which knowledge is not yet abundant: long-tail entities are accompanied by no or few facts (at least initially) and our SI-LP benchmark tests reasoning capabilities with this limited information.\n\n> Entities with degree less than 10 are not considered. But it is quite practical to see such cases in realworld scenario.\n\nWe do study LP for entities with (known) degree less than 10 in our evaluation, in which we consider 0-shot up to 10-shot settings. The reason we selected entities with degree at least 10 in the original KG to form the test set is that we needed to have ground truth available. I.e., we can only evaluate 10-shot LP if 10 facts about the entity are actually known.\n\n> The title is confusing. The dataset is not only for semi-inductive but also contains transductive.\n\nThe benchmark focuses on the semi-inductive setting. As you say, it can also be used to evaluate the transductive setting, but this setting can already be evaluated using the original Wikidata5M dataset. In fact, our benchmark aims to stay close to the original Wikidata5M transductive benchmark so that the performance of transductive evaluation is comparable across these two benchmarks.\n\n> It is good to add a figure with examples to compare different settings.\n\nTable 5 in the appendix shows an example of the varying amount of information (atomic, mention, description + a few facts). Is this what you had in mind? If not, additional suggestions are welcome!\n\n> Reference: Ali et al. Improving Inductive Link Prediction Using Hyper-Relational Facts. In ISWC2021.\n\nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Despite only a refinement of exsiting Wikidata5M, the proposed data did contribute to the large-scale KG reasoning community by providing another choice of dataset and settings.\n\n2. The comparison of various settings is good for validating the usefulness of each type of information. It maybe inspires some future research.", "reasons_to_reject": "1. The proposed benchmark is only a refinement of an existing data without much annotation effort. I admit it is useful for some specific tasks, but it is hard to evaluate its “novelty” as a dataset paper. \n\n2. Some choices of building the dataset is not convincing enough. (1) why are the unseen entities long-tail? I believe if we just want to test the inductive learning ability of a model, it does not matter whether the unseen entity is long-tail or not.(2) entities with degree less than 10 are not considered. But it is quite practical to see such cases in real-world scenario. \n\n3. The title is confusing. The dataset is not only for semi-inductive but also contains transductive.\n\n4. I understant the space limit as a short paper. But if the authors have extra space, it is good to add a figure with examples to compare different settings.", "questions_to_authors": "1. Why not also contain the inductive learning setting?\n\n2. I am also wondering the performance of LLM in directly predicting the link with proper prompts.  I suppose LLMs will not be perfect but have a decent performance. If an LLM can already do this task well, the task will be not so meaningful and the authors have to think about how to make the dataset more useful.", "paper_topic_and_main_contributions": "This paper proposes a new large-scale benchmark for link prediction based on Wikidata5M. Specifically, the proposed Wikidata5M-SI focuses on semi-inductive and transductive link prediction, and ensures the unseen entities are long-tail entities. It allows to evaluate models with varying amount of context and provides controlled amount of textual information. It evaluates baselines with various settings and obtains some general conclusions of usefulness of each piece of information.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "This reference may be a good add to explain different transductive/inductive settings.\n\nAli et al. Improving Inductive Link Prediction Using Hyper-Relational Facts. In ISWC2021."}, "extra_text": {}}, {"review_id": "63f4Gstvhi", "reviewer": "Reviewer_jB8R", "summary": "", "questions": "1. Does semi-inductive mean exactly one entity (either head or tail) is unseen during training in the zero-shot setting? Similarly, in few-shot, does exactly one entity is a low-frequency entity?\n2. What is the average degree or degree distribution of the entities in the dataset?\n3. Removing triples from a split can lead to a cascading effect, i.e., a reduction in other entities' frequencies. Was it one-shot removal or any measures were taken to handle the cascading effect?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and constructive criticism. We provide thoughts on each point below, and we will update the paper accordingly.\n\n> The paper does not provide enough motivation for why the SI-LP task is important.\n\nSI-LP focuses on modelling entities that are unknown or unseen during link prediction, such as out-of-KG entities (not part or not yet part of the KG) or newly created entities (e.g., a new user/product/event). A straightforward solution to handle unseen entities is to retrain the model and apply transductive inference. Unseen entities may arise frequently, however, and retraining the model is expensive for large KGs. The goal of SI-LP is to avoid retraining and perform LP directly, i.e., generalize beyond the entities seen during training. The goal of our benchmark is to provide a testbed for SI-LP methods. In addition, the benchmark clearly separates and evaluates different scenarios w.r.t. the information provided for unseen entities at test time: from zero- to few shot on the one hand and from no text to full textual descriptions on the other hand.\n\n> Does semi-inductive mean exactly one entity (either head or tail) is unseen during training in the zero-shot setting?\n\nYes. The goal of semi-inductive LP is to reason how unseen entities connect to (the entities in) an existing KG.\n\n> Similarly, in few-shot, does exactly one entity is a low-frequency entity?\n\nNo. In our benchmark, the unseen entity have low degree by construction. The other (seen) entitiy, can have low, mid, or high frequency. Some examples are shown in Tab. 5 (appendix) under \"context triples\", each of which is used as a prediction target in our evaluation.\n\n> Removing triples from a split can lead to a cascading effect, i.e., a reduction in other entities' frequencies. Was it one-shot removal or any measures were taken to handle the cascading effect?\n\nSince we focus on long-tail entities, the cascading effect is negligible and we did not take any particular countermeasure. Some detail: We removed 0.03% of entities and 0.07% triples from the train split. Moreover, long-tail entities are often connected to entities with a higher degree, and the resulting change of degree of these entities is relatively small.\n\n> Reproducibility: 2\n\nWe will ensure reproducibility and publicly provide scripts, datasets, model implementations and hyperparameters of the presented baselines. (An excerpt of the benchmark dataset with a script for task generation is provided as additional material with this submission.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a new benchmark for zero-shot and few-shot link prediction tasks that can promote further research in this direction.\n2. The paper makes interesting observations relevant to the literature.\n", "reasons_to_reject": "1. While the dataset proposed is a valuable resource to the community, the need for a semi-inductive link prediction setting is not well motivated. Also, it is not very well distinguished from existing transductive and inductive settings. \n", "questions_to_authors": "1. Does semi-inductive mean exactly one entity (either head or tail) is unseen during training in the zero-shot setting? Similarly, in few-shot, does exactly one entity is a low-frequency entity?\n2. What is the average degree or degree distribution of the entities in the dataset?\n3. Removing triples from a split can lead to a cascading effect, i.e., a reduction in other entities' frequencies. Was it one-shot removal or any measures were taken to handle the cascading effect?\n", "paper_topic_and_main_contributions": "Summary:\n\nThe paper introduces a benchmark for semi-inductive link prediction (SI-LP) in knowledge graphs (KG). The proposed dataset is based on an existing (transductive and inductive) link prediction dataset, Wikidata5M. Unlike the inductive setting where all the test entities are unseen during training, the semi-inductive setting addresses the case where test entities contain a mix of seen and unseen entities, e.g., new entities joining the KG. The benchmark also enables evaluating models on zero-shot and k-shot link prediction (0<=k<=10). \n\nThe paper also evaluates existing models (possibly adapted for SI-LP) on the benchmark and presents their findings, such as the importance of textual information for the task, compared with the context information.\n\nContributions:\n1. A benchmark for semi-inductive link prediction\n2. A detailed comparison of existing models on the benchmark\n3. A comparison of the impact of textual description vs context information on the SI-LP task\n4. A finding that less common relations are more informative as context information\n\nStrength:\n1. The paper makes interesting observations about the importance of various sources of information (i.e., context, mention, description) for the link prediction task.\n2. The experiments provide sufficient justifications for the observations made in the paper.\n\nWeakness:\n1. The paper does not provide enough motivation for why the SI-LP task is important, and why we need a new benchmark for it when transductive and inductive benchmarks already exist in the literature. Also, the description of semi-inductive is a bit ambiguous. It is described with \"some entities known apriori\", but it is not very clear whether each test triple has a k-shot (possibly zero) entity and a known entity (frequency > 20). Also, it is unclear what advantage we get from analyzing few-shot (k>0) link prediction compared to the transductive setting.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AERgfcWRja", "reviewer": "Reviewer_onsE", "summary": "", "questions": "A. Table 2: Do you have any intuition/explanation as to why KGT5 + Descriptions works better (by far) than KGT5 + context + Descriptions? Does the encoder struggle with long/complex inputs? Is there any ablation how context length (descriptions + verbalized context) affects performance?\nB. I have not understood \"dropout\" and the \"trade-off\" described (e.g. in 266). Dropout as a regularization technique? Can you elaborate on that?\nC. Do you have any insights into whether any measurable biases exist when sampling only from the 11-20 degree entities? Are some entity types over- or under-represented? Is this in any way problematic?\nD. You mentioned that most of the required information is already present in the descriptions and IR approaches are required for accessing that. Do you have any concrete ideas about how to tackle this? If it is “easy” to do - should there be a baseline in your paper showing how hard accessing this information is?\nE. Talking about \"realistic\" for the selection of the k-shot triples: I think this is a problematic word in general because what \"realistic\" constitutes always depends on the context (for example, you assume the availability of high quality text to be \"realistic\"). Maybe it is simply the \"hardest\" setting (as shown in Table 3)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and constructive criticism. We provide thoughts on each point below, and we will update the paper accordingly.\n\n> I have not understood \"dropout\" and the \"trade-off\" described (e.g. in 266). Dropout as a regularization technique? Can you elaborate on that?\n\nThank you for pointing this out. Our description is indeed misleading, a better term may be \"neighborhood hiding\" (we'll change that in the paper).\n\nWhen \"neighborhood hiding\" is enabled, we (i) remove the complete neighborhood from the prompt (w.p. 25%), (ii) sample the neighborhood down to 1-10 triples (25\\%), or (iii) leave the neighborhood untouched (50%). The idea is to provide the model with less neighborhood information during training by modifying the prompt. This makes the prompts closer to semi-inductive prompts, but takes them further away from transductive prompts.\n\nThe described tradeoff is between the increased semi-inductive performance of KGT5-context and the (slightly) reduced transductive performance when using neighborhood hiding.\n\n> Table 2: Do you have any intuition/explanation as to why KGT5 + Descriptions works better (by far) than KGT5 + context + Descriptions?\n\nKGT5-context seems to be heavily reliant on the provided neighborhood (up to 100), i.e., its focus on neighborhood seems to be larger than the focus on the description. Since neighborhood is missing or reduced (up to 10) in k-shot LP, performance drops considerably. The neighborhood hiding technique described above is a simple baseline approach that aims to mitigate this effect, but it's clearly not particularly effective and should be explored further in future work.\n\n>  Is there any ablation how context length (descriptions + verbalized context) affects performance?\n\nIn the context of few-shot prediction, both full neighborhood and full description typically fit into the input length. For this reason, \"k\" can be seen as a measure of context length. We did not perform any further ablation.\n\nThe authors of KGT5-context provided anecdotal results with larger context sizes, however, reporting better performance in general. In our study, we used the default settings of KGT5-context.\n\n> Do you have any insights into whether any measurable biases exist when sampling only from the 11-20 degree entities? Are some entity types over- or under-represented? \n\nYes, they are; this bias is by design. In particular, high-degree entities in a KG such as Wikidata often refer to types/taxons (e.g, human, organization, ...) as well as popular named entities (e.g., Albert Einstein, Germany, ...). These entities are fundamental to the KG and/or of high interest and have many facts associated with them. For this reason, they do not form suitable candidates for unseen or new entities. In addition, removing high-degree entities for the purpose of evaluating SI-LP is likely to distort the KG (e.g., consider removing type \"human\" or \"Germany\"). In contrast, our benchmark focuses on entities for which knowledge is not yet abundant: long-tail entities are accompanied by no or few facts (at least initially) and our SI-LP benchmark tests reasoning capabilities with this limited information.\n\nThe table below shows the fraction of entities of a certain type across all entities and across entities with degree 11-20; the shift in distribution is apparent (e.g., more humans, less taxons). If deemed useful, we will add the table to the appendix of the paper. \n\n|   WikidataID  | Mention   | Pct full graph    | Pct degree 11-20  |\n|   ------  |   ------  |   -------:    |   ------: |\n| Q5 | human | 36%   | 61% |\n| Q16521 | taxon | 9%   | 1% |\n| Q13406463 | Wikimedia list article | 3%   | 0% |\n| Q11424 | film | 3%   | 8% |\n| Q486972 | human settlement | 3%   | 0% |\n\n\n> If it \\[accessing information from descriptions\\] is “easy” to do - should there be a baseline in your paper showing how hard accessing this information is? \n\nKGT5 + description in the zero-shot setting essentially serves as an LLM baseline. The paper currently does not spell this out / provide more insight. \n\nFor the annotated examples presented in Tab. 5, we grouped the zero-shot performance by information in the description and obtained: \n\n| Setting       | Answer on rank 1 | \n|---------------|   ------: |\n| contained     |   75% |\n| deducible     |   50% |\n| not contained |   0%  |\n\nWe will extend this analysis to all annotated triples mentioned Tab. 3 and include it in the paper.\n\n>  Selection of k-shot triples: \"realistic\" constitutes always depends on the context. Maybe it is simply the \"hardest\" setting?\n\nYes, good point!\n\n> Computational constraints may prevent some models from working out-of-the-box, but it was not made clear if these are inherent problems concerning the run-time (algorithmically). \n\nFor NBFNet \\[1\\] the large memory footprint is inherent to the model; it is a full-graph GNN and hard to scale. For NodePiece \\[2\\], however, the problem mainly lies in the expensive evaluation code that is available; it is more likely to be addressable. We will clarify this in the paper.\n\n> BLP, for example, should work - at least offline.\n\nBLP is closely related to the DistMult+Description approach presented in the paper; we chose the latter as it is also used as a baseline in \\[3\\]. The main differences to BLP are:\n\n(i) \\[3\\] uses MPNet, BLP uses BERT.\n\n(ii) \\[3\\] concatenates a learnable \"structural embedding\" to the CLS embedding of the language model, whereas BLP does not.\n\nWe will clarify this in the paper.\n\n> 163: All employed models are \"embedding\" based - maybe better \"structural\", \"graph-based\", or \"graph-only\"?\n\nIndeed, thank you!\n\n> 231: This contradicts finding (1)?\n\nThank you, again! Our presentation is misleading indeed. Finding (1) shoud read \"semi-inductive performance is far behind transductive performance **for long-tail entities**.\"\n\nTransductive evaluation is performed on query entities with varying degree, while SI evaluation is only performed on low-degree query entities. The finding is supposed to say that transductive performance on low-degree query entities is considerably higher than SI performance on low-degree entties (see column long-tail in Tab. 2).\n\n> You might want to add https://arxiv.org/pdf/2301.00716.pdf\n\nThank you!\n\n\\[1\\] Zhu, Zhaocheng, et al. \"Neural bellman-ford networks: A general graph neural network framework for link prediction.\" Advances in Neural Information Processing Systems 34 (2021): 29476-29490\n\n\\[2\\] Galkin, Mikhail, et al. \"NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs.\" International Conference on Learning Representations. 2021.\n\n\\[3\\] Hu, Weihua, et al. \"OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs.\" Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed benchmark fills a gap which is not suitably filled by current benchmarks, which study semi-inductive link prediction with text. The closest semi-inductive benchmark which offers multiple text modes by Daza et al. is neither as large-scale nor offers a few-shot evaluation. The study of hybrid models which combine text (calling for NLP) and knowledge graphs is a prevailing topic and fits with the EMNLP. The paper is well written and easy to follow.", "reasons_to_reject": "The models provide an insight into the expected SOTA performance; however, there could have been more. It is understandable that computational constraints may prevent some models from working out-of-the-box, but it was not made clear if these are inherent problems concerning the run-time (algorithmically) or just poor implementations. BLP, for example, should work - at least offline.\n", "questions_to_authors": "A. Table 2: Do you have any intuition/explanation as to why KGT5 + Descriptions works better (by far) than KGT5 + context + Descriptions? Does the encoder struggle with long/complex inputs? Is there any ablation how context length (descriptions + verbalized context) affects performance?\nB. I have not understood \"dropout\" and the \"trade-off\" described (e.g. in 266). Dropout as a regularization technique? Can you elaborate on that?\nC. Do you have any insights into whether any measurable biases exist when sampling only from the 11-20 degree entities? Are some entity types over- or under-represented? Is this in any way problematic?\nD. You mentioned that most of the required information is already present in the descriptions and IR approaches are required for accessing that. Do you have any concrete ideas about how to tackle this? If it is “easy” to do - should there be a baseline in your paper showing how hard accessing this information is?\nE. Talking about \"realistic\" for the selection of the k-shot triples: I think this is a problematic word in general because what \"realistic\" constitutes always depends on the context (for example, you assume the availability of high quality text to be \"realistic\"). Maybe it is simply the \"hardest\" setting (as shown in Table 3)?", "paper_topic_and_main_contributions": "The paper proposes a new benchmark to study transductive and semi-inductive link prediction models. In a semi-inductive setting, links to a known graph for new unseen entities must be predicted. The dataset is based on Wikidata5m and offers a transductive (known entities), 0-shot (only textual features), and k-shot (text + k 1-hop neighbourhood triples) settings. The text is either only a mention (single phrase, usually the name of the entity) or a verbose description. The training/test-split is created by sampling from all 11-20-degree entities and setting them aside. Model performance is measured using MRR/Hits@k. The authors test multiple models to set up the SOTA baseline performance on the new benchmark. The main findings are: transductive is easier than semi-inductive, a trade-off exists between transductive and semi-inductive performance, high-quality text directly influences model performance, more study of hybrid models is required, spurious relations offer better information than general ones.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "You might want to add https://arxiv.org/pdf/2301.00716.pdf as a complementary approach: they study scarcity, noisy text, 0-shot only SI-LP, also using a biased sampling approach for the training/test split.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Other remarks are marked by the line number or table/figure reference:\n\n29: Are they really the most relevant? Transductive is definitely the most prevalent. Such statements, without any justification at least, are always risky.\n\n43: Long-tail: its definition first appears on pg. 3 - maybe add a (sparsely connected) to clarify.\nUp to 156: - A clarification about what \"mention\" means would have been nice for people unfamiliar with the internal naming of Wikidata properties (a simple example completely suffices). Mentions (e.g. in entity-linking) are usually multiple phrases associated to a single entity.\n\n163: All employed models are \"embedding\" based - maybe better \"structural\", \"graph-based\", or \"graph-only\"?\n\n189: (As previously mentioned) Are only their implementations running out of memory or is this a theoretical constraint inherent to their methods? If it is not a theoretical constraint, then it is a bit strange to say \"We would have, but it would have caused additional work\". Apart from that: wouldn't it be a nice way to underline the computational challenge of (your) large-scale dataset(s)\n\n195: I don't like the explanation: It is using two BERTs, for contrastive learning: one encodes \"h [SEP ] r\" and one encodes tail entities; it measures cosine similarity (the dot-product calculation is a computational trick)\n\n221: Wording unclear: Any textual information? I would guess that verbose, curated, high quality text is highly valuable in contrast to the simple mention phrase?\n\n231: This contradicts finding (1)?\n\nTable 4: Small typo: \"inductve\"\n\n313: Isn't this just saying: add more data to be better (which is never surprising for deep learning models)? My point being: to reach transductive setting performance, we only need to add quasi-transductive data (triples) to the data instead of working on the models? This falls a bit short, IMHO.\n\n\n"}}]

Author rebuttal:

Meta-review: This paper introduced a new large-scale benchmark for link prediction derived from Wikidata5M. This dataset focuses on fact prediction for unseen entities given a variety of contextual information. 

Reviewers appreciated the importance of the problem space, and detailed evaluation presented in this paper, but they concerned about the motivation of semi-inductive link prediction and dataset construction.
Decision: Accept-Findings