SlowGuess/ACL_2020_OCR huggingface dataset:
entity embeddings after learning them during KG embedding learning phase (Section 4.2) is necessary. Results on WebQuestionsSP (Table 3) highlight the fact that, even with a small number of training examples EmbedKGQA can learn good question embeddings that can infer the multi-hop path required to answer the questions.
Our method on WebQSP KG-50 outperforms all baselines including PullNet, which uses extra textual information and is the state-of-the-art model. Even though WebQuestionsSP has fewer questions, EmbedKGQA is able to learn good question embeddings that can infer mission links in KG. This can be attributed to the fact that relevant and necessary information is being captured through KG embeddings, implicitly.
5.4 QA on KG with missing links
State-of-the-art KGQA models like PullNet and GraftNet require a path between the head entity and the answer entity to be present in the Knowledge Graph to answer the question. For example, in PullNet, the answer is restricted to be one of the entities present in the extracted question subgraph. For the incomplete KG case where only  $50\%$  of the original triples are present, PullNet (Sun et al., 2019a) reports a recall of 0.544 on the MetaQA 1-hop dataset. This means that only for 54.4 percent of questions, all the answer entities are present in the extracted question subgraph, and this puts a hard limit on how many questions the model can answer in this setting.
EmbedKGQA, on the other hand, uses Knowledge Graph Embeddings rather than a localized sub-graph to answer the question. It uses the head embedding and question embedding, which implicitly captures the knowledge of all observed and unobserved links around the head node. This is possible because of the link prediction property of
Table 3: Performance on WebQuestionsSP dataset. All baseline results were taken from Sun et al. (2019a). The values reported are hits@1. Numbers in brackets correspond to a setting where text was used to augment the incomplete KG (WebQSP KG-50). For more details please refer Section 5.3.2.
Model	Accuracy
ComplEx	20.1
EmbedKGQA	29.9
Table 4: QA results on MetaQA 1-hop for the experiments in which there is no link between head entity and answer entity. We have compared the results with the KG completion methods in which gold relation of the question is known. The details are provided in Section 5.4.
Model	WebQSP
KG-Full	WebQSP
KG-50
EmbedKGQA	66.6	53.2
{+ 2-hop filtering}	72.5	51.8
{+ 2-hop filtering, - Relation matching}	58.7	48.5
{− Relation matching}	48.1	47.4
Table 5: This table shows the importance of relation matching module (Section 4.4.1) and effect of neighbourhood based filtering on EmbedKGQA in the WebQuestionsSP dataset. EmbedKGQA in itself contains the relation matching module. Here we try to see the effect of ablating the relation matching module and adding a 2-hop neighbourhood filtering during answer selection. Please refer Section 5.5 for more details.
Knowledge Graph Embeddings.
So unlike other QA systems, even if there is no path between the head and answer entity, our model should be able to answer the question if there is sufficient information in the KG to be able to predict that path (See Fig. 1).
We design an experiment to test this capability of our model. For all questions in the validation set of the MetaQA 1-hop dataset, we removed all the triples from the Knowledge Graph that can be directly used to answer the question. For example, given the question 'what language is [PK] in' in the validation set, we removed the triple  $(PK, in\_language, Hindi)$  from the KG. The dataset also contains paraphrases of the same question, for, e.g., 'what language is the movie [PK] in' and 'what is the language spoken in the movie [PK]'. We also removed all paraphrases of validation set questions from the training dataset since we only want to evaluate the KG completion property of our model and not a linguistic generalization.
In such a setting, we expect models that rely only on sub-graph retrieval to achieve 0 hits@1. However, our model delivers a significantly better 29.9 hits@1 in this setting. This shows that our
model can capture the KG completion property of ComplEx embeddings and apply it to answer questions which was otherwise impossible.
Further, if we know the relation corresponding to each question, then the problem of 1-hop KG QA is the same as KG completion in an incomplete Knowledge Graph. Using the same training KG as above and using the removed triples as the test set, we do tail prediction using KG embeddings. Here we obtain 20.1 hits@1. The lesser score can be attributed to the fact that ComplEx embedding uses only the KG while our model uses the QA data as well - which in itself represents knowledge. Our model is first trained on the KG and then uses these embeddings to train the QA model, and thus it can leverage the knowledge present in both the KG and QA data.
5.5 Effect of Answer Selection Module
We analyse the effect of the answer selection module (Section 4.4) on EmbedKGQA in the WebQuestionsSP dataset by ablating the relation matching module. Furthermore, in order to compare with other methods that restrict the answer to a neighbourhood in the KG (Sun et al. (2019a), Sun et al. (2018)), we experimented with restricting the candidate set of answer entities to only the 2-hop neighbourhood of the head entity. The results can be seen in Table 5. As we can see, relation matching has a significant impact on the performance of EmbedKGQA on both WebQSP KG-full and WebQSP KG-50 settings.
Also, as mentioned earlier, WebQSP KG (Free-base subset) has an order of magnitude more entities than MetaQA (1.8M versus 134k in MetaQA) and the number of possible answers is large. So reducing the set of answers to a 2-hop neighbourhood of the head entity showed improved performance in the case of WebQSP KG-Full. However, this caused a degradation in performance on WebQSP KG-50. This is because restricting the answer to a 2-hop neighbourhood on an incomplete KG may cause the answer to not be present in the candidates (Please refer figure 1).
In summary, we find that relation matching is an important part of EmbedKGQA. Moreover, we suggest that n-hop filtering during answer selection may be included on top of EmbedKGQA for KGs which are reasonably complete.
In this paper, we propose EmbedKGQA, a novel method for Multi-hop KGQA. KGs are often incomplete and sparse which poses additional challenges for multi-hop KGQA methods. Recent recent for this problem have tried to address the incompleteness problem by utilizing an additional text corpus. However, the availability of a relevant text corpus is often limited, thereby reducing broad-coverage applicability of such methods. In a separate line of research, KG embedding methods have been proposed to reduce KG sparsity by performing missing link prediction. EmbedKGQA utilizes the link prediction properties of KG embeddings to mitigate the KG incompleteness problem without using any additional data. It trains the KG entity embeddings and uses it to learn question embeddings, and during the evaluation, it scores (head entity, question) pair again all entities, and the highest-scoring entity is selected as an answer. EmbedKGQA also overcomes the shortcomings due to limited neighborhood size constraint imposed by existing multi-hop KGQA methods. EmbedKGQA achieves state-of-the-art performance in multiple KGQA settings, suggesting that the link prediction properties of KG embeddings can be utilized to mitigate the KG incompleteness problem in Multi-hop KGQA.
We would like to thank the anonymous reviewers for their constructive feedback, and Ashutosh Kumar, Aditya Rastogi and Chandrahas from the Indian Institute of Science for their insightful comments. This research is supported in part by a grant from Intel and the Ministry of Human Resource Development, Government of India.
Ivana Balažević, Carl Allen, and Timothy M. Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. arXiv preprint arXiv:1901.09590.
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2503-2514.
Antoine Bordes, Sumit Chopra, and Jason Weston.
2014a. Question answering with subgraph embeddings. arXiv preprint arXiv:1406.3676.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Antoine Bordes, Jason Weston, and Nicolas Usunier. 2014b. Open question answering with weakly supervised embedding models. In Joint European conference on machine learning and knowledge discovery in databases, pages 165-180. Springer.
Zihang Dai, Lei Li, and Wei Xu. 2016. Cfo: Conditional focused neural question answering with large-scale knowledge bases. arXiv preprint arXiv:1606.01994.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence.
Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multicolumn convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 260-269.
Google. 2013. Freebase data dumps. https:// developers.google.com/freebase/data.
Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 221-231.
Frank L Hitchcock. 1927. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics, 6(1-4):164-189.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems, pages 4284-4295.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef,
Soren Auer, et al. 2015. Dbpedia-a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195.
Dingcheng Li, Jingyuan Zhang, and Ping Li. 2018. Representation learning for question classification via topic sparse autoencoder and entity embedding. In 2018 IEEE International Conference on Big Data (Big Data), pages 126-133. IEEE.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Denis Lukovnikov, Asja Fischer, Jens Lehmann, and Soren Auer. 2017. Neural network-based question answering over knowledge graphs on word and character level. In Proceedings of the 26th international conference on World Wide Web, pages 1211-1220. International World Wide Web Conferences Steering Committee.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126.
Tom M. Mitchell, William W. Cohen, Estevam R. Hruschka Jr., Partha P. Talukdar, Bo Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi Mishra, Matt Gardner, Bryan Kisiel, Jayant Krishnamurthy, Ni Lao, Kathryn Mazaitis, Thahir Mohamed, Ndapandula Nakashole, Emmanouil A. Platanios, Alan Ritter, Mehdi Samadi, Burr Settles, Richard C. Wang, Derry Wijaya, Abhinav Gupta, Xinlei Chen, Abulhair Saparov, Malcolm Greaves, and Joel Welling. 2018. Never-ending learning. Commun. ACM, 61(5):103-115.
Salman Mohammed, Peng Shi, and Jimmy Lin. 2017. Strong baselines for simple question answering over knowledge graphs with and without neural networks. arXiv preprint arXiv:1712.01969.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In ICML, volume 11, pages 809-816.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697-706. ACM.
Haitian Sun, Tania Bedrax-Weiss, and William W Cohen. 2019a. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. arXiv preprint arXiv:1904.09537.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. arXiv preprint arXiv:1809.00782.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019b. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pages 2071-2080.
Ledyard R Tucker. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279-311.
Ferhan Ture and Oliver Jojic. 2016. No need to pay attention: Simple recurrent neural networks work!(for answering" simple" questions). arXiv preprint arXiv:1606.05029.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2019. Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. arXiv preprint arXiv:1911.00219.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014a. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.
Min-Chul Yang, Nan Duan, Ming Zhou, and Hae-Chang Rim. 2014b. Joint relational embeddings for knowledge-based question answering. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 645-650.
Min-Chul Yang, Do-Gil Lee, So-Young Park, and Hae-Chang Rim. 2015. Knowledge-based question answering using the semantic embedding space. Expert Systems with Applications, 42(23):9086-9104.
Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base.
Wen tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In ACL.
Wenpeng Yin, Mo Yu, Bing Xiang, Bowen Zhou, and Hinrich Schütze. 2016. Simple question answering by attentive convolutional neural network. arXiv preprint arXiv:1606.03391.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In Thirty-Second AAAI Conference on Artificial Intelligence.