Better Together: Enhancing Generative Knowledge Graph Completion with Language Models and Neighborhood Information
Authors: Alla Chepurova
Aydar Bulatov
Yuri Kuratov
Mikhail Burtsev
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge graphs
Knowledge graph completion
Language models
Graph neighborhood
Transformer
TL;DR: The addition of neighborhood information improved KGC with language models and outperformed KGT5 and conventional KGC approaches.

Abstract: Real-world Knowledge Graphs (KGs) often suffer from incompleteness, which limits their potential performance. Knowledge Graph Completion (KGC) techniques aim to address this issue. However, traditional KGC methods are computationally intensive and impractical for large-scale KGs, necessitating the learning of dense node embeddings and computing pairwise distances. Generative transformer-based language models (e.g., T5 and recent KGT5) offer a promising solution as they can predict the tail nodes directly. In this study, we propose to include node neighborhoods as additional information to improve KGC methods based on language models. We examine the effects of this imputation and show that, on both inductive and transductive Wikidata subsets, our method outperforms KGT5 and conventional KGC approaches. We also provide an extensive analysis of the impact of neighborhood on model prediction and show its importance. Furthermore, we point the way to significantly improve KGC through more effective neighborhood selection.

Better Together:
Enhancing Generative Knowledge Graph Completion with Language Models and Neighborhood Information
Alla Chepurova' Aydar Bulatov!
Yuri Kuratov
Mikhail Burtsev
Neural Networks and Deep Learning Lab, MIPT, Dolgoprudny, Russia
AIRRI Moscow, Russia
London Institute for Mathematical Sciences, London, UK chepurova@deeppallov..o ai, (bulatov.. as yurii. kuratov)@physsech edu, mb@l ims. ac uk
Abstract
Input
Predict
(Seee place bont, W
Real-world Knowledge Graphs (KGs) often suf- fer from incompleteness, which limits their po- tential performance. Knowledge Graph Com- pletion (KGC) techniques aim address this issue. However, traditional KGC methods are computationally intensive and impractical large-scale KGs, necessitating the learning of dense node embeddings and computing pair- wise distances. Generative transformer-he based language models (e.g., T5 and recent KGT5) of- fer a promising solution as they can predict the tail nodes directly. In this study, we propose include node neighborhoods information to improve KGC methods based language models. We examine the effects of this imputation and show that, on both in- duetive and transductive Wikidata subsets, our method outperforms KGT5 and conventional KGC approaches. We also provide an exten- sive analysis of the impact of neighborhood on model prediction and show its importance. Furthermore, we point the way to significantly improve KGC through more effective neighbor- hood selection.
Knowledge Graph
Verbalized neighbohood
Model inference
protee Nlave pare
both (SEP)
LM
place dart Baleitto (SEP)
country calaaraii
Ameenca (SPP)
Figure 1: Graph completion pipeline. We extract and verbalize neighborhood of a triplet head node from the Knowledge Graph (KG). Language Model uses verbalized input triplet and neighborhood to make predictions Predictions can be used for question an- swering KG completion.
additional
KG link prediction commonly involves learn- ing knowledge graph embeddings (KGE), which capture structural and semantic information about KG nodes, relations, and their neighborhoods in way that simple distance- based function would be able distinguish real triplets from false ones. However, building unique embedding for each node and relation leads to a linear increase in model size and inference time as the KG size grows. An- other drawback that KG updating requires full re-training of KGEs, rather than simply learning new embeddings for updated entities or relations.
1 Introduction
Knowledge graphs (KG) represent structured infor- mation in the form of directed multi-relational graph with entities the graph's 's nodes and rela- tions between entities as the graph's edges. Entities in such graphs could be both real-world objects or abstract concepts. KGs are represented as set of triplets with representation entities and relation between them, i.e. (head entity, relation, tail entity) or (h, r, t) for short. Due to KGs' recent rapid de- velopment, they are now widely used in variety of upplications, including data mining, information retrieval, question answering, and recommendation systems. However, most real-world knowledge graphs suffer from incompleteness. Knowledge graph completion (KGC) algorithms attempt to fill in the gaps in the KG order to solve this problem.
Application of language models (LMs) KGC tasks promising direction (Pan et al., 2023). State-of-the-arr performance in most natural lan- guage processing (NIP) tasks has recently been demonstrated by pre-trained language models like BERT (Devlin et al., 2019) and more recent large or foundational language models (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023). Language models-based approaches, such as KG-BERT (Yao et al., 2019), BERTRL (Zha et al., 2022), KGT5 (Saxena et al., 2022) and GenKGC (Xie et al., 2022) consider a triplet the KG as a text sequence. While distance-

---

methods incorporate more structural information in dense embeddings, language models tend to cap- ture more semantic information contained in both relations and entities of KG, enabling such models to emhed previously unseen entities and operate on new facts based on semantic information contained in triplets textual representation. Thus, KGTS posi- tions KG link prediction as sennence-to-sequeee tasks. KGT5 model trained to generate target node using textually verbalized triplets achieves compet- itive performance and offers a scalable and simple framework for KGC and QA tasks.
2 Methods
One can represent KG more formally as G (T R), where a set of entities, R is a set of relations and T is set of triplets T E E x R x E. Therefore, the formulation of the KGC task is the following: for each triplet Ti = (h, r. f), where h E E, E, E R, answer both (h, r,,r) ?) and (2, 1) queries on the graph g/T,. The KGC tasks can have different variations depending on the KG type: inductive and transductive. In a transduc- tive setting the KGC task is formulated as follows: given a train KG Gtrain = (Ttrains &, R) train the model to predict on the inference KG Ginference = (Tinffrencee E, R). therefore in the train and infer- ence KG there is same set of entities and relations. In inductive KGC the model is trained on the KG Girain = (Tirain: R, Etrain) and predicts the infer- ence KG gin finference = (Tinference: R, finference). so the trained model observed only on the entities from train subgraph of KG.
Current methods based on language models (KG- BERT, BERTRL, KGTS, and GenKGC) rely only on information that can be extracted from LMs pa- rameters. However, since KG itself is available, build a method that relies on both internal language model knowledge and external information from KG. For the link prediction task (h, r, ?), we pro- pose to extract nodes and relations adjacent to from the KG, i.e. the neighborhood. Node / with its neighborhood better together represents h itself: neighborhood may contain clues such as Palo Alto and United States of America for the triplet (Steve Jobs, place of birth, ?) on Figure
We followed the KGT5 (Saxena et al., 2022) approach to the KGC task using an encoder-decoder Transformer based on T5 (Raffel et al., 2020). The link prediction task was posed as follows: for each triplct (entilyn, mentilyn,elation relation, entilyz) from KG we had to
Though some efforts have been made to incorpo- rate neighborhood into the KGC algorithms (Chen et al., 2020), our proposed pipeline inherits the benefits of generative LLMs compared to KGE approaches: (1) scalability and size of the model, (2) applicability to both transductive and induc- tive KGs due to an ability to generalize unseen entities, (3) no need to rank all possible candidate triplets due to direct generation of tail entity. Also, a concurrent work by the KGT5 authors, KGT5- context (Kochsiek et al., 2023). proposed a similar idea of integrating node neighborhood in the con- text of the generative LM model, supporting the main concerns and results of our study.
predict the tail in the queries (entity), relation, ?) and (entityz, relation ?, To generate input for the model, we converted such queries into a text form. Apart from the query verbalization only as in KGT5, we formed a neighborhood for the node and relations for each triplet in KG. Thus, verbalized queries and their neighborhoods in the form of text were fed to the input of the model for training (see Figure 1) and later to predict incomplete triplets. We form a neighborhood of nodes that have an edge with the triplet head node (I-hop neighbors). The number of adjacent nodes may be too large to fit into the context size of selected language models. Therefore, we sort them based relation semantic similarity. We take only those verbalized nodes and relations that fit into the max sequence length. We provide more details on neighborhood formation in Appendix and verbalization in Appendix C.
Our contributions are as follows: (1) we pro- pose a generative graph completion method which effectively integrates language models with graph neighborhood information. (2) We show that incor- porating graph information into multiple generative language models achieve strong performance im- provement of link prediction in both transductive and inductive settings. Moreover, our approach achieves state-of-the-arr results on the inductive ILPC dataset. (3) We analyze and evaluate the im- pact of adding neighborhood information on the performance of the models.
To evaluate our approach to link prediction tasks we used publically available transductive and in- ductive KGs. Specifically, we used the Wiki- dataSM (Wang et al., 2021) dataset, which is one of the largest publicly available transductive KG up-to-date and large and small versions of ILPC

---

dataset (Galkin et al., 2022), the largest fully in- ductive benchmark. ILPC inference graph size is challenging for modern GNN methods with induc- tive capabilities (Galkin et al., 2022). Apart from being challenging to solve by common approaches, the ILPC KGs are much more sparse than the trans- ductive Wikidata5M. The graph size and spanseness of the data make the task hard for existing methods and result in modest performance metries. Still, there were no existing solutions that beat the base- line provided in the ILPC paper. Both used datasets are the subsets of the real-world KG Wikidata. Datasets' statistics are provided in the Table 4).
CE loss. Such an approach is principally different from distance-basee methods as it does not require generating negative samples to make them further from the target prediction for each training query. In this way, training iteration becomes independent of the KG size.
To form predictions, first verbalize the query (head, relation, ?) and pass it as an input to the generative model to produce an output sequence. which will be a verbalization of the target entity id. For inference, we employed two strategies. The first one was sampling k most probable sequences from the decoder. We follow the KGT5 (Saxena et al., 2022) setting and use sample size of 50. We apply this approach to Wikidata5M datasets is transductive. The second option for inference was to search for the closest entities in KG the generated one. We apply the second approach to ILPC inductive datasets as language models have to generate previously unseen entities. More details on inference procedure are in Appendix D.
We compared models with and without neigh- borhoods to evaluate our hypothesis that neigh- borhoods help to better together represent graph node. In addition, unlike the KGT5, we experi- mented with different model configurations, both fine-tuning an existing pre-trained model and train- ing from scratch as common knowledge obtained during language models pre-traing me-training might be use- ful on real-world knowledge graphs like Wikidata.
We used the T5-small with 60M parameters our base model with both pretrained and ran- domly initialized weights to evaluate the effect of pretraining of the language model on the KG task. We added additional "[SEP]" tokens to the tokenizer with 30K tokens vocabulary to separate query and neighborhood. The model was trained using Adam optimizer (Loshchilov and Hutter, 2019), le-5 learning rate, 10% dropout, and batch size equals 320. The number of training steps was equal to 5M for the Wikidata5M dataset and 2M for all others. For multi-gpu training, we used the Horovod parallelization framework (Sergeev and Balso, 2018). In cases where a batch size equal to 320 did not fit GPU, we used gradient accumulation steps. We also benchmarked Ope- nAT 3...t..---tt 5-turbo zero-shot link prediction and evaluate the benefits of neighborhood LLM (see Appendix G). The code for all the described experiments is available at the link?.
3 Results
The hypothesis behind approach that neigh- borhood information incorporated in context fed to the language model should enhance link predic- tion of generative LM. Tables 1 and 2 demonstrate the performance of our technique on Wikidata5M and ILPC datasets under various conditions, includ- ing training a T5 model with and without neighbors, using pre-trained model or training a model from scratch, and zero-shot with gpt-3. 5.5turbb..
These results support the hypothesis that the quality of link prediction increases with the use of neighbors in the context of the generative model in both transductive and inductive settings. Moreover, our approach demonstrated SOTA performance on the ILPC dataset, while Wikidata achieved the best result among generative link prediction models, which is still comparable to scores of SOTA graph and LM methods with many more parameters. Relatively small absolute values of scores on the ILPC dataset can be explained by its fully inductive setup and larger sparseness com- pared to Wikidata5M. Still, there are no publicly available solutions that exceed the baseline score provided by ILPC authors. Our solution overper- forms the existing baseline, is elegantly simple, and more interpretable than GNN-based solutions. Experiments on pre-trained models revealed that pre-training speeds up convergence but degrades
As KGT5, our model produces probability dis- tribution over all possible tokens in the vocabulary at each step of decoding. Such distribution is penal- ized for not matching the actual distribution, i.c., for the difference between the real target token at position and the generated token step using
https://pplatform.penntt
con/ /ttp//ooo//ooeesggtt------
hhttps://github com/screemix/ Agc-tS-wiith-nnighbbrs

---

Table Adding a neighborhood persistently improves results on the link prediction task. We report Hits k metrics Wikidata5M, ILPlC, and ILPCee test sets. Arrows 14 show the effect of the neighborhood on language models. Neighbors allow to achieve the best performance considering the number of model parameters.
WIIDATAAM
ILPC-SMALL
ILPC-LARGE
MODEL
H@l
H
H@8 10
H
H#3
He 10
H@1
H88
H@10
#PARAMS
KGTS (SAXENA ET Al.,, 2022)
0.267
0.318
0.365
60M
TRANSE (BORDES ET 2013)
0.170
0.311
0.392
2.400M
COMPLEX (TROLILLOON
2016)
0.255
614M
KEPLER (WANG ET AL., 2021)
0.173
0.224
0.277
125M
SIMKGC (WANG ET 2022)
0.313
0.376
0.441
220M
INDNODEPIICE (GALKIN Al.,, 2022)
0.007
0.022
0.092
0.037
0.054
0.125
15.5K
INDNGDEPECEEINN (GALKIN ET AL., 2022)
0.076
0.140
0.251
0.032
0.073
0.146
24K
T5-SMALL NOT PRE-TRAINDD
0.240
0.286
0.523
60M
TS. SMALL NOT PRE- TREANER NEICHBORS
0.327
0.3631
0.386'
60M
TS-SMALI,
0.205
0.248
0.282
0.073
0.117
0.177
0.112
60M
TS SMALL PREEPRERAAAEEE TRAINED NEIGHBORS
0.306]
0.3421
0.3611
0.076
0.1371
0.2091
0.1141
0.1391
0,200
60M
TS. BASE PRE-TRAINED
0.073
0.130
0.199
0.110
0.136
0.198
220M
BA-BABB PRE-TRAINED NEIGHBORS
0.0821
0.142
0.1167
0.1411
0.215
Table 2: Neighbor information improves zero-shot link prediction with OpenAl gpt-3. 3.5--.5u 5-turbo-0613, "ILPC- Large results were obtained on 5000 random samples from the test set.
or as a substring of another entity in neighbor- hood or even the task itself (Table 3). Despite the tail appearing in input only in a small portion of the test set, the neighborhood model learns to make use of such hints, outperforming the baseline T5 in all selected cases. The high solution rate of more than 4 out of samples with hinted targets supports the importance of properly selected neighborhoods.
GPT
GPT+NEIGH
DATASET
H@
H@3
H@1
H@3
WIKIDATA5M
0.098
0.131
0.1131
0.1571
ILPC-SMALL
0.079
0.107
0.0841
0.1181
ILPG-LARGE*
0.103
0.134
0.114f
0.1501
0.31
Removing Important triplets
the quality of the model by few points compared to the one trained from scratch.
Removing random triplets
0.25
0.22
Also, we demonstrated that neighbors' informa-
Hits
0.19
0.16
tion enhances performance across different gen- erative models. This includes open-source mod- els such as T5, as well as proprictary models like OpenAl GPTs that are distributed as model-as-a- service. Although both types of models have their own advantages and limitations, it should be noted that OpenAI GPTs are black box models that can- not be modified or interpreted further. Usage of OpenAl GPTs is restricted in the case of more spe-
0.13
0.10
0.07
0.04
0.01
3
# dropped triplets
Figure KG neighborhood is critical for the model performance. The plot shows how the quality of link prediction depends on the number of neighborhood triplets removed from the input context Wikidata5M T5-smmll not pre- pre-trained model. The blue solid line cor- responds to removing the most relevant triplets and the dashed red line represents random selection.
cialized or proprietary KGs that can store secure or non-public information. In the context of our study, OpenAl's gpt-3. 3.5 5--5-rtur has much larger size and cost, compared to smaller and transparent T5, still our approach improves the performance of both models.
To evaluate the impact of neighborhoods, we analyzed triplets' importance by sorting them based on how they affect the probability of the target node prediction. As Figure 2 shows, performance of the model quickly deteriorates with the removal of important triplets from the context. This implies potential avenues for improvement by prioritizing significant triplets selection prior model training. Moreover, we analyzed attention maps of the model train on Wikidata5M and found that indeed pays attention to relevant neighbors, e.g. Appendix The analysis shows that adding relevant graph
To get an understanding of how proposed models learn to use neighboring triplets we compared pre- dictions of models trained with and without graph information in the input. One possible case when neighbors help in link prediction is when a target entity is somehow mentioned in the model input. Because of neighborhood filtering, the target triplet never appears in the model input, however in some cases target entity can occur with different relations

---

Table 3: Adding relevant neighbors improves link pre- diction for T5-small the Wikidata test set. The table shows the exact match depending on a hint of the target entity's appearance as a part of the model input.
Limitations
Limitations of this study include the training of only relatively small models, with only T5 being employed. Further study of the benefits of larger models applied to KGC tasks should be performed. Additionally, each knowledge graph in this study was limited to only one model, which may not accurately represent more complex graphs that in- volve multiple datasets. Another limitation is that language models are better suited for real-world textual data and may not work well for KGC rep- resenting more specialized concepts, such as bi- ology or medicine. Also, although neighborhood selection was performed, we cannot guarantee that the selected neighborhoods are optimal. Further works should include the way of extending the abil- ity of models to capture larger parts of a graph or selecting the most relevant neighbors. Moreover, constructing prompts for testing GPT performance may not be optimal, and the model's maximal per- formance cannot be guaranteed without knowledge of its training data.
Target position
T5
T5 Neighbors
frequency
In task
0.93
0.94
5 %
In neighborhood
0.34
0.82
8 %
Not in input
0.18
0.24
87 %
Total
0.23
0.32 1
100 %
context information greatly enhances link predic- tion. Hence one way of improving performance is increasing neighborhood size by adding k-hop neighbors. Due to the quadratic complexity of at- tention and the limited input size of pretrained transformers, processing exponentially growing neighborhoods would be computationally expen- sive. To deal with large input sizes one can use efficient T5 architectures with input sizes up 32k (Guo et al., 2022) and 64k (Ainslie et al., 2023) to- kens or recurrent approaches (Bulatov et al., 2022; Hutchins et al., 2022) that can sequentially process effectively unlimited neighborhood sizes. Another potential way of subgraph selection is to add rele- vant but more distant knowledge graph information, such random walks more sophisticated meth- ods (Das et al., 2018).
Ethics Statement
As field of study in the intersection of natural lan- guage processing and knowledge graphs, this work inherits the main ethical considerations of both of these domains. Automated graph completion tasks may produce various types of mistakes, including but not limited to factual and grammatical errors. Consequently, any application based on this work should employ robust error detection mechanisms prior deployment. Additionally, given that our system employs a generative language model, there exists a risk of generating false or misleading con- tent such as hallucinations and biases existing in pre-training and knowledge graph data. Users of the proposed method or any application based on it should be warned about the potential risks of using generative language models. Potential mis- uses may include purposeful disinformation and mislcading language or factual information. Tak- ing into account the aforementioned considerations, we believe that under ethically responsible usage this work will have a positive impact on artificial intelligence research and applications.
4 Conclusions and Future work
Our study found that incorporating KG neighbor- hood information into a generative model's con- text can enhance the quality of the link prediction task. We trained the T5-small model on transduc- tive Wikidata5M and inductive ILPC datasets with and showed that coupling the node and its neighborhood information together better represents the node. Our approach signifi- cantly outperformed baselines the largest real- world KGCs, with a comparable or fewer number of parameters. We also demonstrated that Ope- nAI's 3 performance on the KGC task also benefits from neighborhood information without prior fine-tuning. In addition to the quality improvement, we analyzed the impact of neighbor- hood imputation, highlighting the significance of selecting relevant triplets extending the model's context in future works. Furthermore, future stud- ies could explore the impact of model size, the effect of neighborhood imputation on the datasets from other domains, and neighborhood selection strategies.
various configuration
Acknowledgements
We are grateful to Mikhail Galkin for valuable ad- vice on knowledge graphs and introduction to the

---

ILPC dataset. We are thankful to SberDevices for granting us access to additional computational re- sources. A.C., A.B., and Y.K.'s 's work was supported by a grant for research centers in the field of artifi- cial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D703211500000 ment with the Moscow Institute of Physics and Technology dated November 1, 2021 No. 70-2021- 00138.
language modeling with pathways, arXiv preprint arXiv:2204.02311.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy. Alex Smola, and Andrew McCallum. 2018. Go for walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In International Conference on Learning Representa- tions.
and the agrec-
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistic..
References
Joshua Ainslic, Tao Lei, Michiel de Jong, Santiago On- tañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Tee-Thorp, Yi Tay, Yun- Hsuan Sung, and Sumit Sanghai. 2023. Colt5: Faster rangerange transformers with conditional computa- tion.
Mikhail Galkin, Max Berrendorf, and Charles Tapley Hoyt. 2022. An Open Challenge for Inductive Link Prediction on Knowledge Graphs. arXiv preprint arXiv:220.001520.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the associa- tion for computational linguistics, 5:135-146 146.
Mandy Guo. Joshua Ainslic, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsual Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguisties: NAACL2022, pages 724- 736, Seattle, United States. Association for Compu- tational Linguistics.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in neural information pro- cessing systems, 26.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
DeL Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Bloukereeurrent transformers. In Advances in Neural Information Processing Systems.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):355 547.
Adrian Kochsick, Apoorv Saxena, Inderjeet Nair, and Rainer Gemulla. 2023. Friendly neighbors: Contex- tualized sequence-to-sequeecc link prediction. arXiv preprint 0rxiv:2305./3055
Ilya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization. In International Confer- ence Learning Representations.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. In Ad- vances in Neural Information Processing Systems. volume 35, pages 11079-11091, Curran Associates, Inc.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Ji- apu Wang, and Xindong Wu. 2023. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arXiv:2306.00 08302.
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. 2020. Hitter: Hierar- chical transformers for kndwledge graph embeddings. arXiv preprint ar0iv:2008. /2813.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5455-551)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling

---

Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNEL pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
A Input data pipeline
Hput query
-ou Dniyy and vilaton rapp
Qanr vataliztion
The Negh
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sequees knowledge graph com- Proceedings
vapper
pletion and question answering. of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 281 4-2828, Dublin, Ireland. Association for Computational Linguistics.
Figure 3: Visualization of the complete pipeline for the preparation the input data of the language model: 0) pre-calculate FastText embeddings for each relation in KG and form a static similarity matrix out of them; 1) form a neighborhood for the input triplet: 2) sort the neighborhood by relation similarity to the relation in the input query; 3) verbalize and concatenate query triplet and its neighborhood.
Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.0 8002.5599.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
B Neighborhood formation
To augment the semantic information contained in the entity and relationship names with the structural one, we used triplet neighborhoods. A triplet neighborhood is formed as follows: for a query (head, relation, ?) search for 1-hop neighbors associated with head entity, while excluding from the formed neighborhood target tail entity, which is associated with head by relation; head can be both head or tail entity in triplets that form this 1-hop neighborhood. In this manner, we obtain the context neighborhood of the graph, avoiding the leakage of the target entity. The process of neighborhood formation is illustrated on Figure 4.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, foric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. Interna- tional conference on machine learning. pages 2071- 2080. PMLR.
Liang Wang. Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. Simkge: Simple contrastive knowledge graph completion with pre-trained language models. arXiv preprint 2xXx:2203.0216:
Xiaozhi Wang. Tianyu Gao. Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transac- tions of the Association for Computational Linguis- tics, 9:176-194.
(Phww Joo8, plice oiimm 1%
Faxiido www.g/otttottoeoooooooooooo Widdee/ooo
legat- Francaco
Xin Xie, Ningyu Zhang, Zhoubo Li, Shumin Deng, Hui Chen, Feiyu Xiong, Mosha Chen, and Huajun Chen. 2022. From discrimination to generation: Knowl- edge graph completion with generative transformer. In Companion Proceedings of the Web Conference 2022, pages 162-165.
Fid detipatt easkou tipal ottiriation and foom roat
Frandiice
Unied
bunded
(Draw Joos oxneutioon axteppassa))
(Ereo 20i5 courtty
miihin ln Strwe of Aftenta))
(Tewe Joos ploca
Deed
(Bave Joza educatudle flad Co (Acale hunded Slen..bbe Jobe)
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg- bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909 1999.03193
Figure 4: Neighborhood formation taking the input triplet, search for the triplets in which the head from the input triplet either head or tail, then exclude the triplet with relation and tail from the input. As result, there is a list of triplets that form the neighborhood for a given triplet. The represented neighborhood from the WikidataSm KG is incomplete and textually verbalized for ease of understanding.
Hanwen Zha, Zhiyu Chen, and Xifeng Yan. 2022. In- ductive relation prediction by bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5923-5931.

---

However, some of the triplets from KGs result in extremely large neighborhoods, e.g. hub nodes such as "human". Therefore, we needed to tackle the problem of large neighborhoods to fit it into the model's context, which is limited to 512 tokens for most language models including T5. We propose to do it as follows: 1) sort the neighborhood by se- mantic similarity of relations from its triplets to the relation from the input triplet in order to prioritize more important information the context; 2) limit the sorted neighborhood to 512 triplets, since this will always be at least as big as the size of the al- lowed context, and, after verbalization, specify the maximum length of 512 for the model tokenizer to fit the resulting verbalized neighborhood represen- tation into the language model context. Neighbor- hood sorting by semantic proximity was performed using a pre-calculated matrix of cosine similarity of relations in KG, for similarity calculation the relations were embedded by the fasttext model (Bo- janowski et al., 2017). Figure displays a pipeline for neighborhood sorting.
MongoDB collection too, and retrieved by batches from the database during training.
C Triplets verbalization
To pass the resulting representations of queries the imput of the language model, we had perform a transformation of these queries and their neigh- borhoods into textual form. In order to do this, we first needed to generate a mapping of entity and relationship Wikidata IDs into corresponding texts. For all the datasets we performed our experiments on there is such textual mapping. However, often there is unambiguous encoding in such map- pings because many entities have the same name. The problem non-unique textual representations concerns Wikidata KG entities, not relations, c.g. apple as a fruit of the apple tree and Apple as an American multinational technology company. To tackle this issue we used the same method KGT5 authors proposed: 1) concatenate non-unique en- tity names with corresponding descriptions if ones exist; 2) in case there is no description or after de- scription, concatenation text representations cannot be disambiguated, add unique numeric ids. For all other smaller datasets we only added a unique numeric identifier at the end of the name of non- unique entities. The full disambiguation pipeline is displayed in Figure 6. No such transformations were required for KG relations, they were fully distinguishable by their text representation.
Input query
Sorting by ralstion simiantty
Nelghbourhood
Sorted neighbournod
Relation similanty matrix
FastText ambeddings
Relations
Figure 5: Neighborhood sorting taking the input triplet, its neighborhood, and pre-calculated similarity matrix for verbalized relation embedded by the FastText model, sort the neighborhood by the similarity of rela- tions in its triplets to the relation in the input query.
ently id
entiyy id
eniiy id
entiy id
Add descripton pton if there
HS metadata
Neighborhood composition was relatively straightforward task for a smaller graph like ILPC, but for Wikidata KG with a large number of triplets in the graph, neighborhood formation took a significant amount of time and RAM. To solve this problem, we used the MongoDB database?, which stored knowledge graph triplets in its collection in JSON format and contained indices for 'head' and 'tail' fields to speed up retrieving. In order not store verbalized data in memory either during the verbalization process or during model training. verbalized representations were outputted to the
Apple
Apple American muliinttinall sechnology company
tf acpie tree
Apple
Apple
Enumerate entilee without descriptions
Apple 1
Apple
Figure 6: Disambiguation schema for wikidata5m en-
tities: 1) concatenate entities' textual representation with corresponding description if there is one in the KG metadata; 2) if there is no description for some entities concatenate their names with unique numerical id.
The next step after obtaining toll mapping of en- tities and relation IDs to their text representations
"htns//wwww.onggdhh.omm

---

was verbalization of the resulting queries and corre- sponding neighborhoods in the KG. Textual repre- sentations of the query (headg: relationg: ?) from KG were formed as follows:
D Link prediction inference
As in KGTS (Saxena et al., 2022), to evaluate top-k possible answers, we sample most probable se- quences from the decoder and assign them their corresponding probabilities, which is a sum of log probabilities of each token in the sequence. Then, assign -00 score for all the nodes that were not generated by the sampling procedure. In this way, we are producing results comparable with tra- ditional KG models as we obtain plausibility scores for all possible triplcts in KG. By such procedure, we can obtain whether a real score of the model (if the target sequence persists in k generated se- quences) or the score that will be worse than real (if the target sequence was not generated in the top-k probable sequences), thus not overes- timating the resulting metric even though we do not score all the possible triplets. Additionally, to exclude both entities that do not belong to KG and known links from the train and valid graph in a transductive setting, we performed filtering procedure specific to KGC evaluation protocol as described in the previous studies (Bordes et al.. 2013).
1. first part of query verbalization was formed via concatenation of heady's 's and relation, 's text mappings with "predict" string, e.g. "pre- dict Steve Jobs place of birth":
2. for each i-th triplet (headg, relation, taill) or (head,, relation;, headg) from the sorted query's neighborhood we formed string consisting of relation; and entity not belonging to the input query, i.c. for query (Steve Jobs, place of birth,?, ?) triplet
the (Steve Jobs, occupation, entrepreneur) from its neighborhood will be "occupation en- trepreneur": in case of so-called inverse triplet, e.g. in triplet (head,, relation, headg), where headg is a tail of the neighbor- hood triplet, we add "inverse of to the textual mapping of relation,, i.e. for (Apple, founded by, Steve Jobs) verbal- ization will be "inverse of founded by Apple":
verbalized
As a main metric for choosing the best model during evaluation, we used Exact Match (EM) or in other words Hits@1 1, which returns in case the generated and target sequences are exactly the same or 0 otherwise. However, an exact match be- tween the target and the generated sequence can be rather strict metric for long target entity names, and for previously unseen entities. Thus, we came up with a new approach for scoring entities pro- duced by generative models such as T5 for induc- tive KGC. In this approach, we first embedded verbalized entities from the inference part of KG by Sbert model (Reimers and Gurevych, 2019) and then indexed the resulted vectors using FAISS in- dex (Johnson et al., 2019). In such a way, we used simple flat FAISS index to perform lookup for the k most similar entities from inference KG to the generated one. Using the index for the inductive ILPC dataset, we retrieve entities from the KG that are semantically closest to the generated sequence even if the generated string is imperfect, but con- tains the right tokens referring to the target entity. In this manner, we could calculate Hits @k retriev- ing top-k closest to the generated entity names in similar way as distance-based methods. Overall, in the case of inductive setup, we used a combina- tion of exact match (EM) and FAISS-based met-
3. at the end we concatenate the verbalization obtained in stage (1) and all the verbaliza- tions from stage (2) using the" [SEP] string inserted between all the above-mentioned strings.
(1130 PTA.
Soried nei ploourtood near moce pranepa) 1oree PST a001 ra1sest roR G4720) (oreear pos 0mN50711 (0312 P112 (10637)
(lou of ond
masite
of
Zaras dffimkk
entrone rerroneneeeee woeeeeeeeeeeeeeeeeeeeeeeeeeeeeee (nav)
Figure 7: Query verbalization map each entity and re- lation id from query triplet and its neighborhood triplets with corresponding textual representations; concatenate "predict" string, obtained query verbalization excluding target and the sorted triplets' verbalization without head entity from the query, separating each verbalized triplet by "[SEP]" token.
Example of query and its neighborhood verbal- ization is represented on a Figure 7.

---

rics Hits @k calculated by the following formula: 1000EM + Hits@l. In this way, we give more weight to the exact match metrie, but the case of equal EM in different checkpoints, the model with the best Hits@1 will be chosen.
G.1 Zero-shot link prediction without neighborhood
You will be provided with a incom- plete triplet from the Wikidata knowledge graph. Your task is to complete the triplet with a tail (sub- ject) based on the given triplet head (object) and relation. Your answer must only include the tail of the triplet with prefix 'Tail:". Do not include relation into the answer.
System prompt:
E Datasets statisties
Table 4: Statistics of used KGs.
User prompt: Triplet to complete: <verbalized triplet>.
Dataset
Entities
Relations
Triplets
WikidataSM
4.8M
828
21M
G.2 Zero-shot link prediction with neighborhood
ILPC-small
10,230
96
78,616
ILPC-laare
46,626
130
202,446
System prompt: You will be provided with a incom- plete triplet from the Wikidata knowledge graph. Your task is to complete the triplet with tail (sub- jeci) based on the given triplet head (object) and relation. Triplet complete IS NOT in the list of related nodes, but it may contain helpful clues. Your answer must only include the tail of the triplet with prefix "Tail:". Do not include relation into the answer.
F Attention map interpretation
attzenship SSP)) Bacc
occupation diplemat
[SEP] accupation Nistorian (SER
istance
gredict
105
san
country
brth
ven
ISEP ]
human
[SEP]
echnic
Armenia </$>>
User prompt: Adjacent relations: <verbalized neighbborkoood>>/Triippee to complete: <verbalized triplet>.
Figure Example of cross-encoder attention map
highlighting the importance of neighborhood triplets from the input
H Overview of concurrent work
While our work was under review, the concurrent work KGT5-context (Kochsiek et al., 2023) was published. Following the same idea, KGT5-context -onttext uses verbalized node neighborhood as additional context for KGC with the KGT5 model. How- ever, KGT5-context explores the applicability of neighborhood information for KGC only in trans- ductive setup. Also, neighborhood sampling KGT5-context was performed randomly instead of prioritizing the neighbors, which makes the re- sults less reproducible and does not consider the importance of neighbors in the context of query. Furthermore, show that this approach gener- alizes well other pre-trained language models like g.t-3.5- 5-trbbb in a zero-shot setting without pre- traininiin on KGC datasets.
Figure 8 demonstrates cross-encoder attention map highlighting the input neighborhood's signifi- cance for the target prediction. The query (Arman Kirakossian, country of citizenship, ?) identifies Yerevan, Armenia's 's capital, as the birthplace of the target entity, thereby amplifying its role predict- ing the right country citizenship.
G Zero-shot link prediction with
Since OpenAl API only returns generated text, cannot get its probabilities. Therefore, slightly different evaluation pipeline from other models. We generate up to three predictions with 3. If any of the randomly se- lected & predictions match the target entity after cleaning and normalization, we set that model pre- diction as correct (Hits @ in Table 2).
We have made the effort and tried our best to reproduce the results of the KGT5 model (Saxena et al.. 2022) on Wikidata5M. We report the results of T5-small not pre-trained (Table 1), which is our version of the KGT5 model. Therefore, all our other models were trained and evaluated
We use the following prompts to control the 50t-3..-tuubb-0611

---

the same setting and with the same pipeline as for TS-small not pre-trained model.


Reviews:
[{"review_id": "RSc2Hx9Asc", "reviewer": "Reviewer_12WZ", "summary": "", "questions": "1. Why is T5-SMALL PRE-TRAINED  worse than not pretrained? Any explanation? \n2. Why does the paper show the performance of dropping triplets in Figure2 instead of adding neighbors from zero?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate your comprehensive review and comments. We have thoroughly examined your remarks and would be happy to clarify each of them.** \n\n- Thanks for noting that the approach is simple and straightforward. Indeed, we were very surprised when we did not find works based on such an apparent idea. Moreover,  this unexplored but direct approach can yield state-of-the-art results. We believe that simple methods can serve as better building blocks for stronger pipelines. Akin to Tf-IDF, simple ideas can provide large benefits when combined with other methods. It would be great if you are aware of works that use the same neighborhood techniques and could share them with us, especially ones dedicated to the retrieval of KG in conjunction with generative models.\n- The main goal of this work is to inform the research community with results on how a simple and practical idea can enhance application of generative LMs to KG related tasks. As the idea is straightforward, however powerful as our experiments demonstrated. Therefore, study is submitted as a short paper. \n- The performance concerns you raise are related to the complexity of fully inductive ILPC tasks. We agree that absolute values of metrics are modest but to our best knowledge  **it is state of the art on the ILPC dataset**.\n- We believe that our submission fully complies with reproducibility standards. We provided all the code which reuse the open-source HuggingFace library for the model training in the supplementary materials. All the hyperparameters are described in the paper. All the datasets are available online, moreover, they can be easily accessed via open source pykeen [1] python library.\n\n**Additionally, each of your concerns has been thoroughly addressed below in more detail.**\n\n**On lack of novelty:**\n\nWe agree that the method is simple to implement and similar to retrieval from a structured data store. However, the main contribution of the paper is **the first generative pipeline with explicit graph neighborhood information**. And we believe that simplicity and interpretability is its strength.\n\nThis approach can be used as a building block to integrate more complex graph structures. \nMoreover, using structural KG data combined with LM is a relatively new direction in KGC tasks. Recent works [2] showed the strength of this approach. Apart from increased performance, such an approach provides (1) flexibility in combining different LM models and retriever methods, (2) dynamic learning of contextualized representations for queries instead of using static embeddings for entities and relations, and (3) the possibility to be upgraded to a scalable framework that can provide KG completion over the large KGs. \nWhile the method from the above-mentioned work [2] incorporates several blocks including different types of attention layers (GNN-based attention layer and classical Cross-Attention) and different types of positional embeddings, our approach is much simpler and more interpretable. It can be easily used for different KGs underway and without specific alterations. \n\nAlso, we provided a simple and promising way to rank neighbors by relevance to the target query, instead of just using all possible options. As recent work [3] showed, it is important for language models to put the most important information at the start or the end of its context. Relevant neighbors can be simply lost in the middle and the proposed step of sorting triplets in our work aims to address this issue\n\nApart from the mentioned contributions, adding neighborhoods to a generative language model opens a whole new space for experimentation - in addition to searching for better retrieval methods, we plan to study application of memory augmented transformers to KG-related tasks with the goal to retain important structural information for long documents that can’t be fit in the context of standard LMs. \n\n**On the performance:**\n\nOur approach demonstrated SOTA performance on the ILPC dataset, while on Wikidata we achieved the best result among generative link prediction models, which is still comparable to scores of SOTA graph and LM methods with many more parameters.\n\nThe ILPC dataset is a relatively novel benchmark, which is challenging for conventional models for the following reasons:\n- fully inductive setting, where all the node features are not given during training and links that have to be predicted fall into the unseen ↔ unseen category. Because the presence of unseen entities in the inference graph is fundamentally incompatible with the shallow embedding paradigm [4].\n- ILPC is the largest graph and is almost an order of magnitude larger than all the existing fully inductive benchmarks ( FB15k-237, WN18RR, and NELL-995) and its inference graph size is challenging for modern GNN methods with inductive capabilities [4].\n\nApart from being challenging to be solved by common approaches, the ILPC KGs are much more sparse than the transductive wikidata dataset mentioned in the paper. The mean and median neighborhood sizes of the test set are 9.5 and 4 in the small subset of ILPC, and 9.6 and 3.0 in the small large of ILPC respectively. On Wikidata the mean neighborhood size is 1403 and the median neighborhood size is 10. The graph size and sparseness of the data make the task hard for existing methods and result in modest performance metrics. Still, there are no existing solutions that beat the baseline provided in the ILPC paper. Our solution  overperforms the existing baseline, elegantly simple and more interpretable than GNN-based solutions.\n\n**On the higher performance of learning from scratch:**\n\nIt was one of the reasons why in the KGT5 paper the authors trained the model from scratch. Our initial intuition was that a pre-trained model should perform better too. One plausible explanation is that the capacity of T5-small (with its 60M parameters) might not be sufficient to retain the knowledge from its pre-training tasks and simultaneously optimize effectively for KGC tasks. Additionally, the T5 training corpora (C4) spans a wide array of facts accumulated since the early 2000s. Given that some of these facts may have grown obsolete relative to the more recently collected 2019 Wikidata dataset, this might introduce discrepancies and sub-optimal performance.\n\n**On the value of the T5 model compared to GPT-3.5:**\n\nWe would like to emphasize that our contribution is not a new model, but **demonstrating that neighbors' information enhances performance across different generative models**. Open-source models like T5 and proprietary ones like OpenAI GPT distributed as model-as-a-service each have their own advantages and limitations but our method works with both. Thus, OpenAI models make users dependent on them. Moreover, Open AI’s GPT is a black box model that is not available for further alteration and interpretation. Using such black boxes is maybe good for KGs like Wikidata as it includes well-known facts about real-world entities. It is highly possible that such facts could be incorporated into the training corpora of the GPT model since Wikipedia contains large amounts of open-source information about the real world. However, we cannot guarantee that it would be true for other subsets of Wikidata and other datasets that include not widely spread information as we do not know how GPT training was performed. Additionally, this fact restricts the usage of GPT to more specialized or proprietary KGs that can store secure or non-common information. Another argument for the value of a trained T5 model with neighbors is that it can be run locally at a much lower cost. In the context of our study GPT has much larger size and cost, compared to smaller and transparent T5, still our approach improves performance of the both models.\n\n**Clarifications on neighborhood interpretation:**\n\nThank you for your suggestion. This is indeed another appropriate way to demonstrate the importance of neighborhood.\n\nHowever, dropping triplets is a more natural way because we do not know the importance of the triplets in advance. We form the neighborhood and the most important triplets are not necessarily the first ones (as it would be if we added them from zero). Once we have the neighborhood, we can modify it slightly by dropping up to nine triplets. We show that this reduces the score to zero for important triplets.\n\n**On reproducibility:**\n\nWe provide all our code with training, evaluation, and a full set of hyperparameters as supplementary materials. We also provide all the prompts we used to evaluate the OpenAI GPT3.5 model. The datasets we used are all publicly available. We are committed to ensuring that our work is as reproducible as possible. If you have specific concerns about reproducibility, we would be happy to address them.\n\n**We sincerely appreciate the time you took to review our work and offer constructive feedback. We made every effort to clarify the significance of our results in an unambiguous way, and we hope that our explanations were helpful. We look forward to making the modifications that are required to our manuscript.**\n\n**Links**\n\n[1] https://pykeen.readthedocs.io/en/stable/\n\n[2] Pahuja, V., Wang, B., Latapie, H., Srinivasa, J., & Su, Y. (2022). A Retrieve-and-Read Framework for Knowledge Graph Link Prediction. arXiv preprint arXiv:2212.09724.\n\n[3] Liu N. F. et al. Lost in the middle: How language models use long contexts //arXiv preprint arXiv:2307.03172. – 2023.\n\n[4] Galkin, Mikhail, Max Berrendorf, and Charles Tapley Hoyt. \"An open challenge for inductive link prediction on knowledge graphs.\" arXiv preprint arXiv:2203.01520 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Using neighborhood as additional context is intuitive useful for KGC. The technical part in the paper is rational.\n2. It is good to show the neighborhood information is not only useful in T5 but also useful as prompts in GPT3.5, which makes it more generalizable.\n3. Analyzing the impact of selecting relevant neighbor triplets is also useful.", "reasons_to_reject": "1. The idea of adding negihbor in contexts for KGC is very straightforward. It is a little bit similar to retrieval-enhanced models, if we consider the neighbors as retrieved information. So, it is hard to evaluate its novelty. For me it is not an exciting idea.\n2. Some performance is not as good as it claims. Especially in the inductive setting, it seems the performance gain from adding neighborhood is trivial (Table 1). \n3. Why is T5-SMALL PRE-TRAINED  worse than not pretrained? Any explanation? Also, GPT+NEIGH is already comparable to the trained T5 model on the inductive learning setting, which also reduces its value of the model.", "questions_to_authors": "1. Why is T5-SMALL PRE-TRAINED  worse than not pretrained? Any explanation? \n2. Why does the paper show the performance of dropping triplets in Figure2 instead of adding neighbors from zero?", "paper_topic_and_main_contributions": "The paper proposes to integrate neighborhood information in a generative transformer-based model (e.g. KGT5) to enhance the knowledge graph completion performance. Experiments on transductive and inductive KGC datasets demonstrate the usefulness of adding neighborhood information.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SI2LEpFvJk", "reviewer": "Reviewer_ht6W", "summary": "", "questions": "Question A: Why by having 220M parameters, still the results are very low in the ILPC dataset?\nQuestion B: page 3, lines 235, 237, should it be target triplet or target entity?\nQuestion C: page 2, line 125, should the query be (entity2, relation^-1, ?)?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your extensive review and thoughtful feedback on our work. We highly appreciate the insights you've provided, and have addressed each of your comments.**\n\n- You have found our contribution to be a small extension and limited. The main goal of this work is to inform the research community with results on how a simple and practical idea can enhance application of generative LMs to KG related tasks. As the idea is straightforward, however powerful as our experiments demonstrated, the study is submitted as a short paper.  That is why we emphasize future works dedicated to generative LM in the KG field and propose additional improvements in the generative LM+KG pipeline. \n- Thank you for your comment about the research line of KG and text. Indeed, we were very surprised when we did not find works based on such an apparent idea. Moreover, this unexplored but direct approach can yield state-of-the-art results. We believe that simple methods can serve as better building blocks for stronger pipelines. Akin to Tf-IDF, simple ideas can provide large benefits when combined with other methods. It would be great if you are aware of works that use the same neighborhood techniques and could share them with us, especially ones dedicated to the retrieval of KG in conjunction with generative models.\n- The performance concerns you raise are related to the complexity of fully inductive ILPC tasks. We agree that absolute values of metrics are modest but to our best knowledge **it is state of the art on the ILPC dataset**.\n\n**Additionally, each of your concerns has been thoroughly addressed below in more detail.**\n\n**On contribution:**\n\nYou correctly noted that the procedure of neighborhood integration is simple and easy to implement. It was also surprising to us that there had been no work that implemented this straightforward pipeline to boost the quality of existing models.\n\nThe main contribution of the paper is that we introduce **the first generative pipeline with explicit graph neighborhood information**. It achieves SOTA on ILPC and on Wikidata our model gets the best result among generative link prediction models and is comparable to SOTA methods with many more parameters. \n\nWe do not fully agree that our paper provides a small extension to KGT5. While KGT5 was only applied to inductive KGs, we extended the approach of using generative models on inductive KGs too. We demonstrated that such an approach can be generalized to different generative models apart from T5, even without prior fine-tuning. Also, we provided a simple and promising way to rank neighbors by relevance to the target query, instead of just using all possible options. As recent work [1] showed, it is important for language models to put the most important information at the start or the end of its context. Important neighbors can be simply lost in the middle and the proposed step of sorting triplets by their relevance in our work aims to address this issue. We also studied how application of the same method to pre-trained black-box models like ChatGPT improves performance in KG related tasks. \n\nApart from the mentioned contributions, adding neighborhoods to a generative language model gives a start to a whole new space for experimentation - in addition to searching for better retrieval methods, we propose to investigate the suitability of integration of memory in LM for KG-related tasks to retain important structural information for querying in larger quantities than the limited context of standard LMs. \n\n**On existing competitors:**\n\nThe paper emphasizes the importance of neighborhoods for generative models. On the ILPC dataset, our approach shows state-of-the-art results outperforming IndNodePiece(GNN). On WikiData5M, our results are close to SOTA methods, and we show that generative models are a promising new approach to KGC. We would appreciate it if you could share other methods that outperform our approach significantly on the Wikidata5M and which are better on the ILPC at all.\n\n**On Graph+LM line of work:**\n\nText and knowledge graphs might be combined in several ways (like enriching graph embedding methods with text information as it was done in [2]). Combining text and KGs in the most efficient way is not our research goal. Our work is focused on a promising new direction of applying generative language models to graph representation. Generative language models have benefits compared to embedding methods such as TransE, ComplEx, KEPLER, and GNN-based models while having less number of parameters and achieving performance gain. We demonstrated that this approach could be further improved with information about neighborhood nodes.\n\nAlso, link prediction is commonly performed by classification models that use textual and graph information. However such models cannot predict an unseen entity or use a new relation. As it is shown, generative models can overcome this limitation, but there still are very few successful generative LP models. \n\n**Answers to the questions**:\n\n**Question A**: Why by having 220M parameters, still the results are very low in the ILPC dataset?\n\n**Answer**: Our approach demonstrated SOTA performance on the ILPC dataset.\nMoreover, the ILPC dataset is a relatively novel benchmark, which is challenging for conventional models for the following reasons:\n- fully inductive setting, where all the node features are not given during training and links that have to be predicted fall into the unseen ↔ unseen category. Because the presence of unseen entities in the inference graph is fundamentally incompatible with the shallow embedding paradigm [3].\n- ILPC is the largest graph and is almost an order of magnitude larger than all the existing fully inductive benchmarks ( FB15k-237, WN18RR, and NELL-995) and its inference graph size is challenging for modern GNN methods with inductive capabilities [3].\n\nApart from being challenging to be solved by common approaches, the ILPC KGs are much more sparse than the transductive wikidata dataset mentioned in the paper. The mean and median neighborhood sizes of the test set are 9.5 and 4 in the small subset of ILPC, and 9.6 and 3.0 in the small large of ILPC respectively. On Wikidata the mean neighborhood size is 1403 and the median neighborhood size is 10. The sparseness of the data can lead to such modest performance gain. Still, there are no existing solutions that beat the baseline provided in the ILPC paper. We provided an approach that can overperform the existing baseline and which is elegantly simple and more interpretable than GNN-based solutions.\n\n**Question B**: page 3, lines 235, 237, should it be target triplet or target entity?\n\n**Answer:** We filter target triplet, which is a combination of target relation and tail that refer to the head from the query,to prevent leakage of the answer to the model input. However, the target tail can be incorporated through other neighbors without explicitly providing the answer in the form of a target triplet in the model context. \n\nFor example, for a triplet (Michael Jackson, field of work, music industry) we form a query (Michael Jackson, field of work, ?) and filter out the target triple (Michael Jackson, field of work, music industry) from the neighborhood. However, in the formed neighborhood there is a triplet (Michael Jackson, occupation, singer-songwriter), which can provide a useful hint about the query to the model.\n\n**Question C:** page 2, line 125, should the query be (entity2, relation^-1, ?)?\n\n**Answer:**Thank you. It indeed would be more clear if we included such a notation of an inverse relation. We will include it in a further version of the manuscript.\n\n**On missing references:**\n\nThank you for providing relevant references. The paper you denoted as [1] was uploaded to arXiv just 10 days before the EMNLP deadline, we were unable to mention it in our paper. We will reference suggested papers in the final version of our work.\n\n**On typos and grammar:**\n\nThank you for your attention, we will finalize the text in the next version of the manuscript according to your comments.\n\nWe sincerely hope that our clarifications provide a comprehensive insight of our work and its significance to the field. Once again, we express our deep appreciation for your feedback and look forward to revising our manuscript accordingly.\n\n**Links**\n\n[1] Liu N. F. et al. Lost in the middle: How language models use long contexts //arXiv preprint arXiv:2307.03172. – 2023.\n\n[2] Wang, Xiaozhi, et al. \"KEPLER: A unified model for knowledge embedding and pre-trained language representation.\" Transactions of the Association for Computational Linguistics 9 (2021): 176-194.\n\n[3]  Galkin, Mikhail, Max Berrendorf, and Charles Tapley Hoyt. \"An open challenge for inductive link prediction on knowledge graphs.\" arXiv preprint arXiv:2203.01520 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- Using generative models for KGC tasks has been a relatively less explored topic, and it is worth to further investigating such successful architectures for KGC tasks. \n\n-- The incorporation of neighbors presents a further context for the model and significantly enhances the model performance compared to ignoring neighbors.\n", "reasons_to_reject": "-- The contribution of the paper is limited as it is only a small extension of KGT5 (neighbor verbalization). \n\n-- While the results show that the proposed method can enhance the performance by using neighbors, the proposed method is underperformed by other existing competitors.\n\n-- As the model uses graph and text, and there is a popular line of work that use KG and text, it is expected to cite further works from this direction and possibly compare with them. \n", "questions_to_authors": "Question A: Why by having 220M parameters, still the results are very low in the ILPC dataset?\nQuestion B: page 3, lines 235, 237, should it be target triplet or target entity?\nQuestion C: page 2, line 125, should the query be (entity2, relation^-1, ?)?\n", "paper_topic_and_main_contributions": "Embedding-based link predictions for knowledge graph completion are computationally expensive and lack scalability. Generative transformer language models, e.g., KGT5 are promising approaches that can perform link prediction for KGC. This paper further enhances this line of work by including neighboring nodes as further information to be introduced to LMs. In particular, the paper converts queries (entity, relation, ?) and neighbor nodes to text (verbalization), and then feeds them to LM for training to generate the verbalized target entity id. The experimental results show that the utilization of neighboring nodes can enhance the performance of generative models for KGC. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "— Some methods in Table 1 have wrong references e.g., both KEPLER and SimKGC have the same references.\n— Page 3, the caption of Figure 2, tripletsand —> triplets and (space)\n"}, "extra_text": {"Missing_References": "[1] Pan, Shirui, et al. \"Unifying Large Language Models and Knowledge Graphs: A Roadmap.\" arXiv preprint arXiv:2306.08302 (2023).\n[2] Alam, Mirza Mohtashim, et al. \"Language model guided knowledge graph embeddings.\" IEEE Access 10 (2022): 76008-76020.\n"}}, {"review_id": "dRmRBV9TgK", "reviewer": "Reviewer_Xgyv", "summary": "", "questions": "1. Could you give more information on what \"teacher forcing\" is (line 153)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your comprehensive feedback. We value your insights and try to do our best to address all your concerns.**\n\n- The main goal of this work is to inform the research community with results on how a simple and practical idea can enhance application of generative LMs to KG related tasks. As the idea is straightforward, however powerful as our experiments demonstrated. Therefore, the study is submitted as a short paper. However, the capacity of the works related to the short paper section is limited to 4 pages and we could not include all the background information at first. Nevertheless, we cited all the resources that our work is based on. We will extend our appendix and main section according to your comments to provide more background information about the LM field. \n- The performance concerns you raise are related to the complexity of fully inductive ILPC task. We agree that absolute values of metrics are modest but to our best knowledge  **it is state of the art on the ILPC dataset**.\n\n**Additionally, each of your concerns has been thoroughly addressed below in detail in separate sections:**\n\n**On background information:**\n\nThank you for your feedback. The T5 model is an encoder-decoder transformer pre-trained to predict masked spans. The KGT5 model uses the same architecture, but solves a different pre-training task: the input to the encoder is the triplet head and relation, and the target is the tail of the triplet. We will add more details about these models to the Appendix. For now, you can find more information in the original work [1].\n\n**On complexity of the ILPC task:**\n\nOur approach demonstrated SOTA performance on the ILPC dataset, while on Wikidata we achieved the best result among generative link prediction models, which is still comparable to scores of SOTA graph and LM methods with many more parameters.\n\nThe ILPC dataset is a relatively novel benchmark, which is challenging for conventional models for the following reasons:\nfully inductive setting, where all the node features are not given during training and links that have to be predicted fall into the unseen ↔ unseen category. Because the presence of unseen entities in the inference graph is fundamentally incompatible with the shallow embedding paradigm [2].\nILPC is the largest graph and is almost an order of magnitude larger than all the existing fully inductive benchmarks ( FB15k-237, WN18RR, and NELL-995) and its inference graph size is challenging for modern GNN methods with inductive capabilities [2].\n\nApart from being challenging to be solved by common approaches, the ILPC KGs are much more sparse than the transductive wikidata dataset mentioned in the paper. The mean and median neighborhood sizes of the test set are 9.5 and 4 in the small subset of ILPC, and 9.6 and 3.0 in the small large of ILPC respectively. On Wikidata the mean neighborhood size is 1403 and the median neighborhood size is 10. The sparseness of the data can lead to such modest performance gain. Still, there are no existing solutions that beat the baseline provided in the ILPC paper. We provided an approach that can overperform the existing baseline and which is elegantly simple and more interpretable than GNN-based solutions.\n\n**Answers to the questions:**\n\nThank you for your question. We will explain it in more detail in a future version of the paper.\nTeacher forcing is a common technique for training autoregressive models (e.g., language models). Teacher forcing uses gold outputs as inputs to the next step instead of the model's own predictions. You can find more details and mathematical formulation in this paper [3].\n\n**On the missing references:**\n\nThank you for the reference, we will mention it in our paper. Indeed, they also use neighbor information to get richer representations. Though their approach is promising, especially masking neighbors to learn better representations, it has the following drawbacks:\n- The neighborhood is fed to the model without prioritizing neighbors. As recent work [4] showed, it is important for language models to put the most important information at the start or the end of its context. Important neighbors can be simply lost in the middle and the proposed  step of sorting triplets by their relevance in our work aims to address this issue. Moreover, adding neighborhoods to a generative language model gives a start to a whole new space for experimentation - in addition to searching for better retrieval methods, we propose to investigate the suitability of integration of memory in LM for KG-related tasks to retain important structural information for querying in larger quantities than the limited context of standard LMs. Thank you for mentioning this work, we think it would be important to emphasize this aspect in further versions of the manuscript.\n- The main contribution of the paper is the first generative pipeline with explicit graph neighborhood information. The method in the paper you mentioned is based on classification, which implies it has the same drawbacks as KGE methods - it is challenging for such a model to be applied in the inductive setting, where it should predict previously unseen entities. \n- The number of parameters is larger than the model we used for the transductive datasets \n\n\n**Thank you for your invaluable feedback. Once again, we express our deep appreciation for your feedback and look forward to revising our manuscript accordingly.**\n\n\n**Links**\n\n[1] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" The Journal of Machine Learning Research 21.1 (2020): 5485-5551.\n\n[2] Galkin, Mikhail, Max Berrendorf, and Charles Tapley Hoyt. \"An open challenge for inductive link prediction on knowledge graphs.\" arXiv preprint arXiv:2203.01520 (2022).\n\n[3] Williams, Ronald J., and David Zipser. \"A learning algorithm for continually running fully recurrent neural networks.\" Neural computation 1.2 (1989): 270-280.\n\n[4] Liu N. F. et al. Lost in the middle: How language models use long contexts //arXiv preprint arXiv:2307.03172. – 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is good and very clear. \n2. The results are promising and show that adding neighborhood information clearly helps.", "reasons_to_reject": "1. Some background information is missing. This makes it difficult to follow if one isn't knowledgeable about some information. Specifically, language models, with an emphasis on T5 and KGT5, are referenced often. It would be beneficial to have a brief introduction of these models in the appendix.\n2. While the results are promising, they are quite small on the two ILPC datasets.", "questions_to_authors": "1. Could you give more information on what \"teacher forcing\" is (line 153)?", "paper_topic_and_main_contributions": "In this paper, the authors study the problem of generative knowledge graph completion. Specifically, they focus on enhancing current generative transformer-based LLM models with additional neighborhood information. The niehgborhood information is chosen from the head nodes' neighborhood into the query. They detail their results on both transductive and inductive benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Additional references on existing transformer-based KGC models should be included. One important one is [1]. While it does not include textual information, it is a transformer-based model that also considers the use of neighboring information.\n\n[1] Chen, Sanxing, et al. \"HittER: Hierarchical Transformers for Knowledge Graph Embeddings.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021."}}]

Author rebuttal:

Meta-review: For the task of knowledge graph prediction, this paper argues that a generative LM can be used to capture relevant graph neighborhood information. The paper presents a decent number of experimental results, with generally strong performance. Reviews were mixed, but overall general consensus on sufficient soundness. Reviews were contextualized wrt general PC guidelines about common pitfalls. While a number of typos and clarifications need to be addressed, the authors provided lengthy feedback, which suggests they will make these revisions. Even accounting for this being a short paper, I recommend the next revision, camera-ready or not, be restructured so that the main experimental take-aways are front-and-center.
Decision: Accept-Findings