SlowGuess/NAACL_2022_OCR huggingface dataset:
We test StATIK on three datasets. The first two are the inductive versions of WN18RR and FB15k-237 created in Daza et al. (2021). These datasets split the entities into a training set,  $\mathcal{E}_{train}$ , a validation set,  $\mathcal{E}_{val}$ , and a test set,  $\mathcal{E}_{test}$ . The training graph consists of the all the triples in which the head and tail are both in  $\mathcal{E}_{train}$ . The validation and test
2The code for our work can be found at https://github.com/Elanmarkowitz/StATIK and more details for reproduction can be found in the supplement.
Model	WN18RR	FB15k-237
MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
Commonsense-KBC**	0.01	0.0055	0.009	0.019	0.00028	0.00001	0.000028	0.000056
GloVe-BOW*	0.170	0.055	0.215	0.405	0.172	0.099	0.188	0.316
BE-BOW*	0.180	0.045	0.244	0.450	0.173	0.103	0.184	0.316
GloVe-DKRL*	0.115	0.031	0.141	0.282	0.112	0.062	0.111	0.211
BE-DKRL*	0.139	0.048	0.169	0.320	0.144	0.084	0.151	0.263
BLP-TransE†	0.285	0.135	0.361	0.580	0.195	0.113	0.213	0.363
BLP-DistMult†	0.248	0.135	0.288	0.481	0.146	0.076	0.156	0.286
BLP-ComplEx†	0.261	0.156	0.297	0.472	0.148	0.081	0.154	0.283
BLP-SimplE†	0.239	0.144	0.265	0.435	0.144	0.077	0.152	0.274
StAR‡	0.321	0.192	0.381	0.576	0.163	0.092	0.176	0.309
StATIK	0.516	0.425	0.558	0.690	0.224	0.143	0.248	0.381
Improvement	60.7%	121.3%	54.6%	19.8%	14.9%	26.5%	16.4%	5.0%
graphs given by the union  $\mathcal{E}_{val} \cup \mathcal{E}_{train}$  or  $\mathcal{E}_{test} \cup \mathcal{E}_{val} \cup \mathcal{E}_{train}$  respectively, and all triples contained within. The evaluation triples are those triples that were not present in the training data (nor in the validation set for the test triples). This is evaluation for the dynamic setting.
The third benchmark is Wikidata5M, curated and published by Wang et al. (2019b). Wikidata5M contains close to 5 million entities and 20 million triples. We use the inductive split intended by the authors of (Wang et al., 2019b), in which neither head nor tail in the validation or test triples have been seen during training. This is evaluation for the transfer setting. See Table 2 for an overview of the datasets.
Since we are focused on the task of link prediction in the inductive setting, our main baselines are BERT for Link Prediction (BLP) (Daza et al., 2021) and StAR (Wang et al., 2021). Both recently proposed models that also seek to leverage textual information. For completeness, we also include the other baselines from Daza et al. (2021). These are a DKRL (Xie et al., 2016) implementation and a bag-of-words (BOW) model that represents an entity as the average of all its word embeddings. For these models, entity and relation features are created using GloVe embeddings as well as context free BERT embeddings. On Wikidata5M, we also compare against KEPLER (Wang et al., 2019b), the work that curated the dataset. To the best of our knowledge, these are the only inductive, embed
Table 3: Inductive Link Prediction results on WN18RR and FB15k-237 in the dynamic setting. **Results of running (Malaviya et al., 2020) on our inductive data splits. *Baselines used in (Daza et al., 2021), † results of (Daza et al., 2021), ‡ Results of running (Wang et al., 2021) on our inductive data splits.
Model	Wikidata-5M
MRR	H@1	H@3	H@10
KEPLER*	0.402	0.222	0.514	0.730
BLP-ComplEx†	0.489	0.262	0.664	0.877
BLP-SimpleE†	0.493	0.289	0.639	0.866
StATIK	0.770	0.765	0.771	0.779
Improvement	56.1%	164.7%	16.1%	-12.5%
Table 4: Inductive Link Prediction results on Wikidata-5M in the transfer setting. *Results of (Wang et al., 2019b), †Model variants that perform the best on Wikidata-5M from (Daza et al., 2021).
ding free, link prediction models, that also seek to jointly leverage structural and textual information.
We present three sets of results: (1) The main results, which comprise the performance of StATIK on the three benchmark datasets; (2) Ablation studies to understand how different choices in our model affect the overall performance; (3) Inference speed results to demonstrate utility. Results on FB15k-237 and WN18RR are averaged over 5 runs, while results on Wikidata-5M are averages over 2 runs.
5.4.1 Main Results
Tables 3 and 4 summarize the performance of StATIK in comparison to other state of the art baselines in the dynamic and transfer settings. StATIK sets a new state-of-the-art on MRR,
Model Variant	WN18RR	FB15k-237
Use MPNN	Finetune LM	MRR	H@1	H@10	MRR	H@1	H@10
X	X	0.082	0.032	0.176	0.129	0.079	0.224
X	✓	0.307	0.213	0.500	0.211	0.132	0.362
✓	X	0.488	0.365	0.629	0.179	0.105	0.328
Table 5: Analyzing the importance of each component of the model to overall performance. As a reference, the model variant used to obtain results in table 3 uses the MPNN as well as finetunes the language model.
Model	Dataset
WN18RR	FB15k-237
KGBert	46000	87000
STAR	434	321
BLP	14	21
StATIK (ours)	9.5	4
Table 6: Inference time per query in milliseconds per Query (ms / q) comparison between inductive models. Lower is better.
Hits@1 and Hits@3 on all datasets, sometimes by a remarkably large margin and only underperforms on Hits@10 on Wikidata-5M. In general we notice diminishing benefits as we go from Hits@1 to Hits@10. As noted in Wang et al. (2021), Hits@1 is a weakness for textual encoding based paradigms. StAR and BLP include a structural objective to attempt to remedy this but are outperformed by our GNN based model. Hence it is not only important to incorporate structure but also do so effectively.
5.4.2 Ablation Studies
In addition to the main results we run two sets of ablation experiments. The first of these is to understand the influence each model component (MPNN or Language Model) has on the overall model performance. These results are summarized in Table 5. The second set of ablation experiments demonstrate the effect maximum number of words allowed in the entity description has on StatIK.
6.1 Inductivity
StATIK shows much greater ability to generalize to entirely new entities that are added to the knowledge graph as evidenced by the much superior performance compared to baselines in both dynamic (Table 3) and transfer settings (Table 4). We hypothesize that this is due to the power of MPNNs
to generalize to new nodes added to a graph as well as entirely new graphs (Hamilton et al., 2017a; Velickovic et al., 2018; Wang et al., 2020b). We also note the effectiveness of our model at Hits@1 in the transfer setting, with less drop from Hits@3 and Hits@10 than in the dynamic setting. This phenomenon remains to be more deeply explored but possible factors include the quantity of data, repeated structural patterns in the test set, or other peculiarities of this dataset, such as how the transfer graph was constructed.
6.2 Influence of Model Components
As shown in table 5, we find that while both the language model and MPNN are important to model performance, which matters more depends on the dataset being evaluated on due to differing graph structure. On Wordnet, the MPNN influences overall model performance more, possibly because the graph has more repeated structure. On Freebase, with richer text descriptions, the language model influences the overall model performance more.
6.3 Effects of Description Length
Unsurprisingly, more input data means better performance. However, there are diminishing returns to description length. This is consistent with both Wordnet and Freebase as demonstrated in Figure 3. Since increasing the max word length can increase the computational demand as a result of the language model, there is a need to trade off between model and computational performance.
6.4 Inference Scalability
Inference scalability is paramount in any potentially useful knowledge graph completion model, especially when the link prediction task involves a large number of target entities. Our comparison with other state of the art, inductive knowledge graph completion models in Table 6 demonstrates that StATIK is much quicker at inference time
Figure 3: Max word length effect on performance.
than other competing models. These results are consistent with the theoretical computational complexity of each model (Table 1).
Our work introduces and explores a new model for inductive knowledge graph completion that utilizes structural and textual information, and sets state-of-the-art on multiple benchmarks.
Ideas for future work include (i) testing this model with other types of graph neural networks that have already proved effective on KGs; (ii) testing alternative neighborhood sampling techniques beyond uniform random sampling; and (iii) extending this work to domains beyond text-based KGs.
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001117C0053, NSF MLWINS-2002874, USC-Meta Research Center, USC-ISI MINDS Group, and a gift from the Intel PrivateAI institute. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. We would also like to acknowledge the reviewers who helped improve this work.
Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2018. Hypernetwork knowledge graph embeddings. CoRR, abs/1808.07018.
Lisa Bauer. 2021. Identify, align, and integrate: Matching knowledge graphs to commonsense reasoning tasks. In EACL.
Rajarshi Bhowmik and Gerard de Melo. 2020. Explainable link prediction for emerging entities in knowledge graphs. In SEMWEB.
Wen bing Huang, Tong Zhang, Yu Rong, and J. Huang. 2018. Adaptive sampling towards fast graph representation learning. *ArXiv*, abs/1809.05343.
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, J. Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS.
Jian Jhen Chen, Tengfei Ma, and Cao Xiao. 2018. Fast-gcn: Fast learning with graph convolutional networks via importance sampling. *ArXiv*, abs/1801.10247.
Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, and Gerhard Weikum. 2019. Look before you hop: Conversational question answering over knowledge graphs using judicious context expansion. Proceedings of the 28th ACM International Conference on Information and Knowledge Management.
Damai Dai, Hua Zheng, Fuli Luo, Pengcheng Yang, Baobao Chang, and Zhifang Sui. 2021. Inductively representing out-of-knowledge-graph entities by optimal estimation under translational assumptions. ArXiv, abs/2009.12765.
Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2018. Building dynamic knowledge graphs from text using machine reading comprehension. CoRR, abs/1810.05682.
Daniel Daza, Michael Cochez, and Paul T. Groth. 2021. Inductive entity representations from text via link prediction. Proceedings of the Web Conference 2021.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2017. Convolutional 2d knowledge graph embeddings. CoRR, abs/1707.01476.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multi-hop relational reasoning for knowledge-aware question answering. In EMNLP.
Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann. 2020. Message passing for hyper-relational knowledge graphs. In EMNLP.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. CoRR, abs/1704.01212.
Liyu Gong and Qiang Cheng. 2019. Exploiting edge features for graph neural networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9203-9211.
Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong, and Qing He. 2020. A survey on knowledge graph-based recommender systems. ArXiv, abs/2003.00911.
William L. Hamilton, Rex Ying, and Jure Leskovec. 2017a. Inductive representation learning on large graphs. CoRR, abs/1706.02216.
William L. Hamilton, Zhitao Ying, and J. Leskovec. 2017b. Inductive representation learning on large graphs. In NIPS.
Ben Hixon, Peter Clark, and Hannaneh Hajishirzi. 2015. Learning knowledge graphs for question answering through conversational dialog. In NAACL.
A. Hogan, E. Blomqvist, Michael Cochez, C. d'Amato, Gerard de Melo, C. Gutierrez, J. E. L. Gayo, S. Kirrane, S. Neumaier, A. Polleres, R. Navigli, A. N. Ngomo, S. M. Rashid, A. Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and A. Zimmermann. 2021. Knowledge graphs. Communications of the ACM, 64:96 - 104.
Jin Huang, Wayne Xin Zhao, Hong-Jian Dou, Ji rong Wen, and Edward Y. Chang. 2018. Improving sequential recommendation with knowledge-enhanced memory networks. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.
Seyed Mehran Kazemi and D. Poole. 2018. Simple embedding for link prediction in knowledge graphs. In NeurIPS.
Thomas Kipf and M. Welling. 2017. Semi-supervised classification with graph convolutional networks. ArXiv, abs/1609.02907.
Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. 2021. Learning dynamic embeddings for temporal knowledge graphs. WSDM '21, page 535-543, New York, NY, USA. Association for Computing Machinery.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. _Kagnet: Knowledge-aware graph networks for commonsense reasoning_. ArXiv, abs/1909.02151.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020. Commonsense knowledge base completion with structural and semantic context. Proceedings of the 34th AAAI Conference on Artificial Intelligence.
Elan Markowitz, Keshav Balasubramanian, Mehrnoosh Mirtaheri, Sami Abu-El-Haija, Bryan Perozzi, G. V. Steeg, and A. Galstyan. 2021. Graph traversal with tensor functionals: A meta-algorithm for scalable learning. ArXiv, abs/2102.04350.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912.01703.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Apoory Saxena, Soumen Chakrabarti, and Partha P. Talukdar. 2021. Question answering over temporal knowledge graphs. In ACL.
M. Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and M. Welling. 2018. Modeling relational data with graph convolutional networks. *ArXiv*, abs/1703.06103.
Haseeb Shah, Johannes Villmow, Adrian Ulges, Ulrich Schwanecke, and Faisal Shafait. 2019. An open-world extension to knowledge graph completion models. ArXiv, abs/1906.08382.
Baoxu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In AAAI.
Zhiqing Sun, Zhihong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. ArXiv, abs/1902.10197.
Komal K. Teru, E. Denis, and William L. Hamilton. 2020. Inductive relation prediction by subgraph reasoning. In ICML.
Théo Trouillon, Johannes Welbl, S. Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML.
Shikhar Vashishth, Soumya Sanyal, V. Nitin, and P. Talukdar. 2020. Composition-based multi-relational graph convolutional networks. ArXiv, abs/1911.03082.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Petar Velickovic, Guillem Cucurull, A. Casanova, Adriana Romero, P. Lio', and Yoshua Bengio. 2018. Graph attention networks. ArXiv, abs/1710.10903.
Baocheng Wang and Wentao Cai. 2020. Knowledge-enhanced graph neural networks for sequential recommendation. Inf., 11:388.
Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. 2021. Structure-augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Conference 2021, pages 1737-1748.
Hongwei Wang, Hongyu Ren, and Jure Leskovec. 2020a. Entity context and relational paths for knowledge graph completion. CoRR, abs/2002.06757.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. Dkn: Deep knowledge-aware network for news recommendation. Proceedings of the 2018 World Wide Web Conference.
Peifeng Wang, Jialong Han, Chenliang Li, and Rong Pan. 2019a. Logic attention based neighborhood aggregation for inductive knowledge graph embedding. In AAAI.
Rui Wang, Bicheng Li, Shengwei Hu, Wenqian Du, and Min Zhang. 2020b. Knowledge graph embedding via graph attenuated attention networks. IEEE Access, 8:5212-5224.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019b. KEPLER: A unified model for knowledge embedding and pre-trained language representation. CoRR, abs/1911.06136.
Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and M. Sun. 2016. Representation learning of knowledge graphs with entity descriptions. In AAAI.
Jun Yan, Mrigank Raman, Tianyu Zhang, Ryan A. Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren. 2021. Learning contextualized knowledge structures for commonsense reasoning. ArXiv, abs/2010.12873.
B. Yang, Wen tau Yih, X. He, Jianfeng Gao, and L. Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. CoRR, abs/1412.6575.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kgbert: Bert for knowledge graph completion. ArXiv, abs/1909.03193.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. ArXiv, abs/2104.06378.
Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020. Grounded conversation generation as guided traverses in commonsense knowledge graphs. In ACL.
B Ethical Impact Statement
This work is designed for knowledge graph completion and thus may amplify the ethical ramifications of any work that utilizes such knowledge graphs. In addition, this work is designed to help automate the KG completion process. As a result, this may reduce the number of required employed maintainers of such knowledge bases. However, it also provides tools to increase the effectiveness of volunteer knowledge base maintainers. While impossible to predict the long term effects, we see many more clear positive near term effects than negative ones.
C Data Format
The datasets are structured as tab seperated files.
The triple splits (ind-train.tsv, ind-valid.tsv, ind-test.tsv) are formatted as head-entity<tab>relation<tab>tail-entity
The entity description files (entity2text.txt, entity2textlong.tsv) are formatted as head_entity<tab>description
The relation description files (relation2text.txt) are formatted as relation<tab>description
D.1 Infrastructure
Experiments were conducted on a single GPU server equipped with 8 Nvidia RTX 5000 GPUs and an AMD EPYC 7502 32-Core Processor. All models are written using PyTorch (Paszke et al., 2019), and are trained using the GPUs in a data parallel fashion.
The GPU budget per run is 16 GPU-hrs for WN18RR, 24 GPU-hrs for FB15k-237, 480 GPU-hrs for Wikidata5M.
Minimal hyperparameter optimization was performed, instead, hyperparameters were copied
from those used in BLP (Daza et al., 2021). However, to take full advantage of the GPU's available, we increased the batch size to 32 training triples (64 training queries) per GPU and increased the learning rate from 2e-5 to 2e-4 to roughly compensate. We keep the epochs the same as in BLP (40 epochs for WN18RR/FB15k-237 and 5 epochs for Wikidata5M). Optimizer was AdamW with betas (.9, .98) and linear schedule with warmup of  $20\%$  of training steps.
Sampled neighbors per entity was set to 10 for WN18RR and Wikidata5M, and tuned to 20 for FB15k-237 due to its higher average degree.
We use the bert-base-cased version of BERT from HuggingFace (Wolf et al., 2020) as the language model for both feature extraction and training. This model has 110 million parameters, bringing the total number of parameters to 113 million.
LayerNorm is also used for the initial entity and relation representations as well as for each edge and entity updates in the MPNN.
We use 24 words of text for each input and hardcap the number of tokens at 64. As per standard practice, special token [CLS] is used at the beginning of text, and [CLS] is used to separate entity and relation text as well as at the end of the tokenization.
The main limitation of the model is that it requires textual data for entities in order to run. It also does not work on graphs that constantly change the set of relation types.