SlowGuess/ACL_2020_OCR huggingface dataset:
pre-trained embeddings with 100-dimension used in (Nguyen et al., 2019). We use Adam (Kingma and Ba, 2015) to optimize the model. The parameters of our model are selected via grid search according to the MRR on the validation set. We select the dropout rate from  $\{0.1, 0.2, 0.4, 0.5\}$ , the learning rate from  $\{0.001, 0.0005, 0.0002, 0.0001\}$ , the  $L_{2}$  norm of parameters from  $\{1e^{-3}, 1e^{-5}, 1e^{-8}\}$ , the batch size from  $\{32, 64, 128, 256, 512\}$  and the smoothing parameter  $\lambda$  in Equation 13 from  $\{1, 5, 10\}$ . Finally, the learning rate is set to 0.0002 for WN18RR and 0.0001 for FB15k-237. The  $L_{2}$  norm of parameters is set to  $1e^{-5}$ . The batch size is set to 256. The dropout rate is set to 0.4 for WN18RR and 0.2 for FB15k-237. The smoothing parameter in Equation 13 is set to  $\lambda = 5$ . The number of filters for each convolution operation in the Inception module is set to 32. We
Models	WN18RR	FB15k-237
MR	MRR	Hits@10	MR	MRR	Hits@10
ConvE	4187	0.430	0.520	244	0.325	0.501
KBGAT	1921	0.412	0.554	270	0.157	0.331
InceptionE	2317	0.451	0.563	215	0.334	0.518
ReInceptionE w/o N	1942	0.449	0.573	185	0.348	0.525
ReInceptionE w/o E	1809	0.412	0.569	186	0.343	0.522
ReInceptionE	1894	0.483	0.582	173	0.349	0.528
Table 3: Impact of different modules contributes the KGE task.
observe that MRR performance increases slowly, starting to stagnate around 200 epochs. Finally, we train the model up to 200 epochs in the following experiments. The source codes are available at https://github.com/JuneTse/ReInceptionE.
We compare our results with various state-of-the-art methods. Experimental results are summarized in Table 2. For all KGE models, a key step is to create the invalid triples to construct the negative samples. Most recently, Sun et al. (2020) investigated the inappropriate evaluation problem happened in ConvKB (Nguyen et al., 2018), CapsE (Nguyen et al., 2019) and KBGAT (Nathani et al., 2019). In fact, this issue comes from the unusual score distribution, e.g., the score function for some invalid triples gets the same values as the valid triples. Sun et al. (2020) also found that KBGAT removed the invalid triples when they appeared in the test set during negative sampling, suffering from the leakage of test triples. Therefore, we take the results (marked with the superscript  $b$ ) from (Sun et al., 2020) for ConvKB, CapsE and KBGAT. Besides, we also list the results reported in the original papers (marked with the superscript  $a$ ).
From Table 2, we can see that our proposed ReInceptionE obtains competitive results compared with the state-of-the-art methods. On WN18RR dataset, the ReInceptionE achieves the best results using Hits@10 and MRR, and the second-best results using MR. On FB15k-237 dataset, the ReInceptionE obtains the second-best results using MR, and comparable results using MRR and Hits@10.
Our proposed ReInceptionE is closely related to ConvE (Dettmers et al., 2018) and KBGAT (Nathani et al., 2019). Compared with ConvE, ReInceptionE achieves large performance gains on both WN18RR and FB15k-237 (ConvE vs. ReInceptionE). The reason is that instead of simply con
catenating the head and relation embeddings, ReInceptionE takes head and relation as two channels of the input and applies the Inception network to capture the rich interactions, which is able to learn expressive features by using filters with various scales. Unlike KBGAT, the ReInceptionE takes the (entity, relation) pair as a query and utilizes the relation-aware attention mechanism to gather the most relevant local neighbors and global entity information for the given query. The results again verify the effectiveness of the relation-aware local and global information for KGE.
Some other methods have been proposed to address the KGE task, such as pLogicNet (Ou and Tang, 2019), RPJE (Niu et al., 2020), CoKE (Wang et al., 2019), TuckER (Balazevic et al., 2019a), D4-Gumbel (Xu and Li, 2019) and HAKE (Zhang et al., 2020). pLogicNet (Ou and Tang, 2019) and RPJE (Niu et al., 2020) leverage logic rules to improve the performance. CoKE (Wang et al., 2019) uses Transformer (Vaswani et al., 2017) to encode contextualized representations. HAKE (Zhang et al., 2020) embeds entities in the polar coordinate system to learn semantic hierarchies. D4-Gumbel (Xu and Li, 2019) uses the dihedral group to model relation composition. TuckER (Balazevic et al., 2019a) uses Tucker decomposition to learn tensor factorization for KGE. These methods take a series of different ways to model the KGE task. For example, logic rules play an important role to determine whether a triple is valid or not, we suspect that the performance of our proposed ReInceptionE can be further improved when taking the logic rules into account. We will leave the comparison and deep analysis in the future work.
3.3 Impact of Different Modules
We describe the experimental results in Table 3 to investigate the impact of different modules in ReInceptionE. In Table 3, "InceptionE" is the baseline
5935
Models	Predicting head	Predicting tail
1-1	1-N	N-1	N-N	1-1	1-N	N-1	N-N
WN18RR	ConvE	0.975	0.414	0.110	0.950	0.975	0.153	0.303	0.949
InceptionE	0.976	0.587	0.128	0.957	0.952	0.231	0.482	0.957
ReInceptionE	0.976	0.586	0.152	0.961	0.976	0.272	0.494	0.958
FB15k-237	ConvE	0.303	0.590	0.137	0.400	0.272	0.088	0.845	0.545
InceptionE	0.573	0.624	0.175	0.452	0.557	0.124	0.865	0.557
ReInceptionE	0.609	0.651	0.185	0.473	0.594	0.149	0.872	0.603
Table 4: Link prediction results for each relation category on the WN18RR and FB15k-237 test sets using Hits@10. Following (Bordes et al., 2013), we classify relations into four groups: one-to-one (1-1), one-to-many (1-N), many-to-one (N-1) and many-to-many (N-N).
Query and Target	Top Neighbors and Predictions
Query: (Jack London, nationality, ?)
Target: America	Top Neighbors:
(place_lived, Oakland) Prob: 0.415
(place_of_birth, San Francisco) Prob: 0.353
(Berkeley, student) Prob: 0.083
(influence_by, Friedrich Nietzsche) Prob: 0.042
(influence_by, Charles Darwin) Prob: 0.031
Top Predictions:
America, United Kingdom, Canada, Australia, Germany
Query: (Jerry Lewis, languages, ?)
Target: English Language	Top Neighbors:
(place_of_birth, Newark) Prob: 0.197
(place_lived, Newark) Prob: 0.173
(Nutty Professor II, story_by) Prob: 0.105
(award_nomination, Razzie Award for Worst Actor) Prob: 0.089
(nominated_for, Law & Order: Special Victims Unit) Prob: 0.082
Top Predictions:
English Language, Spanish Language, French Language,
Italian Language, Japanese Language
Table 5: Two examples of top 5 attention neighbors and predictions for the given queries.
model without using relation-aware local neighbors and global entities. "ReInception w/o N" is the model without using relation-aware local neighbor information while "ReInception w/o E" is the model without using relation-aware global entity information. Besides, we also take two closely related models ConvE and KBGAT for fair comparison.
From Table 3, we can see that our baseline InceptionE outperforms the closely related CNN-based model ConvE. Compared with ConvE, InceptionE is more powerful because it can capture the rich interaction features by using filters with various scales. And the ReInceptionE, which incorporates relation-aware local neighborhood and global entity information, outperforms the related graph-based model KBGAT. Table 3 also shows that the ReInceptionE outperforms InceptionE, "ReInception w/o N" and "ReInception w/o E" by a large margin
on both datasets, which reconfirms our observations that relation-aware local neighbors and global entities can play different contributions for KGE.
3.4 Evaluation on different Relation Types
In this subsection, we present the experimental results on different relation types on WN18RR and FB15k-237 using Hits@10. We choose the closely related model ConvE, as well as InceptionE as the baselines. Following (Bordes et al., 2013), we classify the relations into four groups: one-to-one (1-1), one-to-many (1-N), many-to-one (N-1) and many-to-many (N-N), based on the average number of tails per head and the average number of heads per tail. Table 4 shows the link prediction results for each relation category. From Table 4, we find that InceptionE achieves better performance than ConvE for all relation types, indicating that increasing the number of interactions between head and re
5936
lation embeddings is indeed beneficial to KGE task. Furthermore, our proposed ReInceptionE significantly outperforms ConvE and InceptionE for all relation types. In particular, ReInceptionE obtains larger improvements for complex relations, such as one-to-many, many-to-one and many-to-many. This again verifies our observations that increasing the interactions and taking the local-global structural information allows the model to capture more complex relations.
In order to further analyze how relation-aware neighbors contribute to KGE task, we give two examples in Table 5. For the query (Jack London, nationality, ?), ReInceptionE assigns the highest attention scores for neighbors (place_lived, Oakland), since Oakland and America are close to each other in embedding space because of other relations between them. And the top predictions for the query are a set of entities with the type of country. For the second example (Jerry Lewls, languages, ?), ReInceptionE assigns the very high score for neighbor (place_of_birth, Newark). This can allow us to project (place_of_birth, Newark) into the Jerry Lewis region of the embedding space, which can lead to a high score for predicting the target English Language. These examples give clear evidence of how our proposed ReInceptionE benefits the KGE task.
In this paper, we propose a novel relation-aware Inception network for knowledge graph embedding, called ReInceptionE. ReInceptionE takes the benefits of ConvE and KBGAT together. The proposed method first employs Inception network to learn the query embedding, with the aim of increasing the interaction between head and relation embeddings, while at the same time to keep the parameter efficient. Then, we gather the relation-aware local neighborhood and global entity information with an attention mechanism and enrich the query embedding with the joint local-global structural information. Empirical studies demonstrate that our proposed method obtains comparative performance compared with the state-of-the-art performance on two widely used benchmark datasets WN18RR and FB15k-237.
This work was supported by the National Natural Science Foundation of China under Grants 61972290 and 61972173, and also supported by the National Key R&D Program of China under Grant 2018YFC1604000. This research was also supported in part by the research grant from Natural Sciences and Engineering Research Council (NSERC) of Canada and York Research Chairs (YRC) program. We thank anonymous reviewers for their thorough review comments on this paper.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019a. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the EMNLP-IJCNLP.
Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019b. Multi-relational poincaré graph embeddings. In Proceedings of the NeurIPS.
Trapit Bansal, Da-Cheng Juan, Sujith Ravi, and Andrew McCallum. 2019. A2N: Attending to neighbors for knowledge graph inference. In Proceedings of the ACL.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the SIGMOD.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of the NIPS.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI.
Takuma Ebisu and Ryutaro Ichise. 2019. Graph pattern entity ranking model for knowledge graph completion. In Proceedings of the NAACL.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the AISTATS.
Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Proceedings of the ACL.
He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the ACL.
5937
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the ACL.
Xiaotian Jiang, Quan Wang, and Bin Wang. 2019. Adative convolution for multi-relational learning. In Proceedings of the NAACL-HLT.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the ICLR.
Timothee Lacroix, Nicolas Usunier, and Guillaume Obozinski. 2018. Canonical tensor decomposition for knowledge base completion. In Proceedings of the ICML.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Soren Auer, and Christian Bizer. 2015. Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI.
Maas, Andrew L, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of the ACL.
Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. 2019. Anytime bottom-up rule learning for knowledge graph completion. In Proceedings of the IJCAI.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of the ACL.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the NAACL.
Dai Quoc Nguyen, Thanh Vu, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2019. A capsule network-based embedding model for knowledge graph completion and search personalization. In Proceedings of the NAACL.
Guanglin Niu, Yongfei Zhang, Bo Li, Peng Cui, Si Liu, Jingyang Li, and Xiaowei Zhang. 2020. Rule-guided compositional representation learning on knowledge graphs. In Proceedings of the AAAI.
Byungkook Oh, Seungmin Seo, and Kyong-Ho Lee. 2018. Knowledge graph completion by context-aware convolutional learning with multi-hop neighborhoods. In Proceedings of the CIKM.
Meng Ou and Jian Tang. 2019. Probabilistic logic neural networks for reasoning. In Proceedings of the NeurIPS.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In Proceedings of the ESWC.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In Proceedings of the AAAI.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In Proceedings of the ICLR.
Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha P. Talukdar, and Yiming Yang. 2020. A re-evaluation of knowledge graph completion methods. In Proceedings of the ACL.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the CVPR.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the CVPR.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the EMNLP.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of ICML.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2020. Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In Proceedings of the AAAI.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the NIPS.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of the ICLR (Poster).
5938
Kai Wang, Yu Liu, Xiujuan Xu, and Dan Lin. 2018. Knowledge graph embedding with entity neighbors and deep memory network. CoRR, abs/1808.03752.
Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu. 2019. CoKE: Contextualized knowledge graph embedding. CoRR, abs/1911.02168.
Canran Xu and Ruijiang Li. 2019. Relation embedding with dihedral group in knowledge graph. In Proceedings of the ACL, pages 263-272.
Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in LSTMs for improving machine reading. In Proceedings of the ACL.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of ICLR.
Shihui Yang, Jidong Tian, Honglun Zhang, Junchi Yan, Hao He, and Yaohui Jin. 2019. Transms: Knowledge graph embedding for complex relations by multidirectional semantics. In Proceedings of the IJCAI.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. *Quaternion knowledge graph embedding*. In *Proceedings of the NeurIPS*.
Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. 2020. Learning hierarchy-aware knowledge graph embeddings for link prediction. In Proceedings of the AAAI.
5939