SlowGuess/NAACL_2022_OCR huggingface dataset:
PATHRGCN: For PATHRGCN, we use RoBERTa-BASE to encode the question text, and Walklet (Perozzi et al., 2017) to generate the unsupervised node-representations for the KG corresponding to the question. We use Walklet instead of Node2Vec since it exhibits the highest performance over several node classification tasks (Rozemberczki and Sarkar, 2020). Moreover, it does not require any additional features to generate the embeddings and is computationally fast; Walklet was  $\approx 20$  times faster than Node2Vec.
Baselines: We defer the reader to Appendix B.2 for the exact hyper-parameter settings and experimental details of the baselines.
We evaluate the performance of the baselines and our proposed approaches across two metrics commonly used in KGQA, namely, Hits@1 and Accuracy. For a given question, Hits@1 has a value of 100 if the highest-scoring candidate is a correct
2https://spacy.io/usage/spacy-101# annotations-ner
CloudKGQA	Mod-WebQSP
Method	Hits@1	Hits@K	Accuracy	Hits@1	Hits@K	Accuracy
EmbedKGQA	31.6 ± 3.3	31.6 ± 3.3	31.6 ± 3.3	29.1 ± 1.9	32.6 ± 2.2	25.1 ± 1.8
Rel-GCN + TransE	44.9 ± 8.7	52.5 ± 6.1	41.4 ± 6.3	49.4 ± 2.3	59.6 ± 1.2	48.5 ± 1.8
GlobalGraph + TransE	46.6 ± 3.6	56.1 ± 1.9	43.6 ± 2.5	48.4 ± 0.6	59.1 ± 0.7	48.3 ± 0.9
PATHCBR (Ours)	95.4 ± 0.3	96.7 ± 0.3	95.8 ± 0.5	49.3 ± 0.1	56.0 ± 0.1	48.0 ± 0.1
PATHRGCN + Walklet (Ours)	90.4 ± 2.1	91.3 ± 1.5	90.7 ± 1.5	68.6 ± 0.2	75.2 ± 0.4	68.5 ± 0.3
Table 2: Performance of the baselines and our approaches on CloudKGQA, and Mod-WebQSP.  $K$  is the number of correct answers. We report the mean and standard deviation across 5 runs. The best performance is highlighted.
No Masking	Masking Entities	Masking Proper Nouns
CloudKGQA	Hits@1	Hits@K	Acc	Hits@1	Hits@K	Acc	Hits@1	Hits@K	Acc
Path Sequence	67.9	67.9	67.9	67.9	67.9	67.9	66.4	66.4	66.4
One-Hot Vector	88.8	89.4	88.8	95.4	96.7	95.8	82.4	84.9	83.6
Text Embedding	83.6	86.1	84.8	95.7	96.9	96.0	78.4	80.9	79.5
Mod-WebQSP	Hits@1	Hits@K	Acc	Hits@1	Hits@K	Acc	Hits@1	Hits@K	Acc
Path Sequence	33.0	37.9	32.8	41.6	46.5	41.1	47.4	52.2	46.2
One-Hot Vector	32.5	41.1	32.3	44.6	52.1	43.7	49.3	56.0	48.0
Text Embedding	13.7	21.1	16.1	22.4	28.7	23.5	25.2	32.1	26.7
Table 3: Mean performance of PATHCBR across different settings for entity masking and encoding path information, as a sequence of relations (Path Sequence), as a One-Hot Vector, or as a Text Embedding using a PTLM. The best performance is highlighted in bold and the second best is underlined.
answer; else, it is 0. Accuracy denotes the fraction of answers predicted correctly amongst the top K candidates (as a percentage). We also measure Hits@K for a question, for which the value is 100 if the answer is present amongst the top K candidates; else it is 0. For both Accuracy and Hits@K, K is the number of correct answers. We carry out experiment for five random seeds and report the mean and standard deviation. We perform statistical significance using the paired bootstrapped test of Berg-Kirkpatrick et al. (2012) in Dror et al. (2018).
In this section, we pose the following research questions (RQs) and attempt to answer the same. We present instances of preprocessed questions that serve as input to the model.
RQ1. How do our proposed approaches fare on PERKGQA compared to KGQA baselines?
We observe that both PATHCBR and PATHRGCN, yield the highest performance on CloudKGQA, outperforming the existing baselines by over  $100\%$  for Hits@1 and Accuracy in Table 2. We attribute the poor performance of prior KGQA techniques to their inability to (i) learn global node embeddings over the large base
KG or (ii) update the embeddings during training.
For Mod-WebQSP, PATHRGCN achieves the highest performance outperforming preexisting baselines significantly (p-value  $\leq 0.001$ ). However, PATHCBR achieves performance comparable to the baselines, and can answer questions corresponding to templates encountered during training, for instance, "who plays ken barlow in coronation". We attribute the low performance of PATHCBR to:
(i) The underlying global KG for Mod-WebQSP is more complex and dense. There are 572 possible relations as opposed to 8 for CloudKGQA. Moreover, there can be multiple relations between two entities, (e.g. 'location.cOUNTRY.capital' and 'location.contained_by' are both valid relations between Tokyo and Japan), a characteristic absent in CloudKGQA. The possible paths increase exponentially with hops, and additional supervision afforded by GNNs helps answer these questions with long-range dependencies (Wang et al., 2020b).
(ii) Not all possible relations encountered during inference were available during training. E.g., the most relevant question retrieved for "what was wayne gretzky's first team" was "what team does plaxico burress play for", because the relation corresponding to "first team" was absent during training. At times, the pretrained language model could
CloudKGQA	Mod-WebQSP
Method	Hits@1	Hits@K	Accuracy	Hits@1	Hits@K	Accuracy
Rel-GCN + TransE	44.9 ± 8.7	52.5 ± 6.1	41.4 ± 6.3	49.4 ± 2.3	59.6 ± 1.2	48.5 ± 1.8
GlobalGraph + TransE	46.6 ± 3.6	56.1 ± 1.9	43.6 ± 2.5	48.4 ± 0.6	59.1 ± 0.7	48.3 ± 0.9
PATHRGCN + TransE	51.4 ± 4.8	68.4 ± 2.6	57.0 ± 4.4	53.1 ± 0.9	62.6 ± 0.7	52.0 ± 0.8
Rel-GCN + Walklet	79.1 ± 3.9	79.8 ± 4.2	79.3 ± 4.0	63.0 ± 1.1	71.3 ± 0.8	63.0 ± 1.2
GlobalGraph + Walklet	86.3 ± 3.8	87.2 ± 4.0	86.5 ± 3.9	64.4 ± 0.9	72.6 ± 0.9	64.6 ± 0.8
PATHRGCN + Walklet	90.4 ± 2.1	91.3 ± 1.5	90.7 ± 1.5	68.6 ± 0.2	75.2 ± 0.4	68.5 ± 0.3
PATHRGCN + Walklet - NL	90.3 ± 7.1	91.1 ± 6.9	90.6 ± 6.8	65.7 ± 1.0	73.0 ± 1.1	65.8 ± 1.0
Table 4: Performance of the baselines and PATHRGCN when initialized with different node embeddings. We report the mean and standard deviation across 5 runs. The best performance is highlighted. NL stands for Node Loss.
not infer the query's semantic meaning. E.g, the most relevant question for "what town was martin luther king assassinated in" was "what town was abe lincoln born in", despite the occurrence of questions like "where was huey newton killed". Thus if the templates are widely different, it is not sufficient to encode the question using a PTLM; rather, we need to fine-tune the questions to learn meaningful representation.
We further inspect the capabilities of our techniques to address the individual characteristics of PERKGQA, namely multiple answers, variable hop distance, multiple constraints, and variable KG size. Our approaches outperform baselines consistently and significantly on all such fronts.
A thorough analysis of our proposed approaches to the different properties of these two datasets reveals their complementary strengths. We note PATHRGCN has a better performance on larger KG size, more answers, longer hops, and additional constraints, and vice-versa for PATHCBR. We defer the reader to Appendix C for these results.
RQ2. What is the impact of entity masking and encoding different path-information strategies on PATHCBR's performance?
We observe that masking entities using NER, or proper nouns using a POS Tagger improves performance in Table 3. The only exception is for CloudKGQA where due to arbitrary naming conventions (e.g. "abc123"), entities were not detected as proper nouns creating inconsistent templates. We observe that encoding relations as a one-hot vector yields better performance than a text embedding, especially when the relation-names exhibit high lexical overlap as in Mod-WebQSP. Moreover, representing the path information as a sequence of relations cannot deal with unseen templates as in CloudKGQA. We highlight instances that substan
tiate our claim in Appendix C.
RQ3. What role does graph structure and path-information play on PERKGQA?
We investigate the benefits of unsupervised graph representation learning techniques to initialize node embeddings. In particular, we compare the efficacy of Walklet and TransE embeddings, when applied to Rel-GCN, GlobalGraph, and PATHRGCN. We see significant improvements for all models when TransE embeddings are substituted with Walklet in Table 4.
Since we operate for individual KGs, TransE does not have sufficient information to generate meaningful node representations. Walklet leverages the neighbourhood information and thus can capture the structural representation for each KG. PATHRGCN significantly outperforms the baselines on both fronts, when all three models are initialized with Walklet or when all three models are initialized with TransE embeddings.
We also investigate the importance of incorporating node loss (NL in Table 2) for additional supervision. This aids Mod-WebQSP, where multiple relations between entities give rise to several possible paths between source and answer, most of which are spurious. Since multiple paths do not exist for CloudKGQA, removing the node loss does not deteriorate performance.
The task of KGQA has evolved from a simple-classification setting (Mohammed et al., 2018) to an information retrieval paradigm (Wang et al., 2020b; Saxena et al., 2020; Yasunaga et al., 2021; Sun et al., 2019; Xiong et al., 2019) that can tackle multi-hop relations or complex questions. Other approaches include semantic parsing (Lan and Jiang, 2020; Ding et al., 2019; Maheshwari et al., 2019;
Zhu et al., 2020; Ren et al., 2021) and reinforcement learning (Das et al., 2018; Lin et al., 2018; Saha et al., 2019; Ansari et al., 2019). We investigate graph-based information retrieval methods in this work since they achieve SOTA performance without any additional information like logical forms or semantic parses. This sets us apart from recent work on KGQA generalizability (Gu et al., 2021; Chen et al., 2021) which requires such logical forms during training; information often unavailable for real-world data settings. Our work also differs from Sidiropoulos et al. (2020) which is more focused on entity-linking and relation prediction for unseen domains and leverages existing web resources, which is not applicable to us.
Most KGQA approaches that operate in an information retrieval setting over predefined (or base) knowledge graphs follow a similar procedure to make the problem computationally feasible. (Sun et al., 2018, 2019; Wang et al., 2020b,a). They first construct a smaller sub-graph for each question from the base graph, using the Personalized PageRank algorithm (Haveliwala, 2003). then re-use the base graph's node representation to initialize the nodes in the sub-graph. Thus during inference, they already have prior representation of the nodes. However, in our setting, we encounter new KG during inference, and thus we need to learn the representations of those unseen nodes from scratch.
Our PATHCBR approach is closely related to Das et al. (2020), which performs relation linking such as (Delhi, capital_of, _?_.). They first retrieve entities similar to the query entity and the corresponding reasoning paths that lead to an answer for those retrieved entities. They then apply reasoning paths to the query entity. PATHCBR differs in two ways; (i) We operate upon complex or compositional questions and retrieve similar templates rather than entities, and (ii) We do not use a rule-based framework to generate reasoning paths. Rather, we encode the retrieved path information as an embedding and use it to score paths generated during inference to ensure generalization. In a similar vein, Das et al. (2021) uses a neuro-symbolic case-based reasoning approach for answering complex, multi-hop questions. However, their approach cannot be applied to our setting since it requires logical forms (SPARQL queries). We circumvent this requirement by designing PATHRGCN that leverages GNNs, KGs' structure, and path information between source and answers.
We propose PERKGQA, a realistic setting for performing question answering over knowledge graphs; for each user's question, we have their corresponding KG but no additional information. Such a setting addresses the challenges of unseen nodes during inference, and prevents access to information of other users while being computationally feasible. However, state-of-the-art KGQA techniques that require learning node representations a priori fare poorly. We propose two approaches, a simple non-parametric case-based-reasoning model and a supervised neural architecture, harness path information for QA. Our approaches improve upon the baselines by  $6.5\%$  on an academic dataset and  $10.5\%$  on an internal dataset.
Having demonstrated the applicability of PERKGQA in the cloud service provider domain, we aim to explore other scenarios involving personalized or sensitive information, like healthcare. Prior work in medical NLP has focused predominately on generic or ontological knowledge such as UMLS. A personalized KG, constructed over a patient's health records, will encode information specifically for the individual and not the general population, e.g. whether the patient is allergic to certain medications. We plan to collaborate with medical professionals and create personalized KG in the healthcare domain to assist patients.
Furthermore, we seek to address certain limitations of our current approach, namely their inability to tackle spurious paths. We plan to rectify it either by explicitly providing the correct path information or incorporating some learning paradigm to detect them (He et al., 2021). Moreover, for PATHCBR, the retrieval phase is a bottleneck since one needs to compare a given query with all possible training questions and requires better indexing schemes like FAISS (Johnson et al., 2019). Likewise, the inference time for both approaches increases as the number and the length of the paths increase. However, PATHRGCN can adapt to longer paths since the node embeddings provide some degree of additional supervision. Nevertheless, we plan to explore techniques beyond embedding-based approaches, namely semantic parsing or query-graph generation, to alleviate the path-based constraint and adapt them to our PERKGQA. In the absence of gold logical forms, we plan to learn semantic parses through a weakly-supervised or distantly-supervised setting similar to Cheng et al. (2019).
Ghulam Ahmed Ansari, Amrita Saha, Vishwajeet Kumar, Mohan Bhambhani, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2019. Neural program induction for kbqa without gold programs or query annotations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 4890-4896. International Joint Conferences on Artificial Intelligence Organization.
Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995-1005, Jeju Island, Korea. Association for Computational Linguistics.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26.
Samuel Broscheit, Daniel Ruffinelli, Adrian Kochsiek, Patrick Betz, and Rainer Gemulla. 2020. Libkke-a knowledge graph embedding library for reproducible research. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 165-174.
Shuang Chen, Qian Liu, Zhiwei Yu, Chin-Yew Lin, JianGuang Lou, and Feng Jiang. 2021. Retrack: A flexible and efficient framework for knowledge base question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 325-336.
Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata. 2019. Learning an executable neural semantic parser. Computational Linguistics, 45(1):59-94.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy, Alex Smola, and Andrew McCallum. 2018. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In International Conference on Learning Representations.
Rajarshi Das, Ameya Godbole, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2020. Nonparametric reasoning in knowledge bases. In *Automated Knowledge Base Construction*.
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay-Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases.
Jiwei Ding, Wei Hu, Qixin Xu, and Yuzhong Qu. 2019. Leveraging frequent query substructures to generate formal queries for complex question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2614-2622, Hong Kong, China. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Matthias Fey and Jan E. Lenssen. 2019. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477-3488.
Taher H Haveliwala. 2003. Topic-sensitive pagerank: A context-sensitive ranking algorithm for web search. IEEE transactions on knowledge and data engineering, 15(4):784-796.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In WSDM.
Xiaofeng Huang, Jixin Zhang, Zisang Xu, Lu Ou, and Jianbin Tong. 2021. A knowledge graph based question answering method for medical domain. PeerJ Computer Science, 7:e667.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
Endri Kacupaj, Joan Plepi, Kuldeep Singh, Harsh Thakkar, Jens Lehmann, and Maria Maleshkova. 2021. Conversational question answering over knowledge graphs with transformer and graph attention networks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 850-862, Online. Association for Computational Linguistics.
Yunshi Lan and Jing Jiang. 2020. Query graph generation for answering multi-hop complex questions from knowledge bases. In Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, pages 969-974, Online. Association for Computational Linguistics.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3243-3253, Brussels, Belgium. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Niklas Lüdemann, Ageda Shiba, Nikolaos Thymianis, Nicolas Heist, Christopher Ludwig, and Heiko Paulheim. 2020. A knowledge graph for assessing agressive tax planning strategies. In International Semantic Web Conference, pages 395-410. Springer.
Gaurav Maheshwari, Priyansh Trivedi, Denis Lukovnikov, Nilesh Chakraborty, Asja Fischer, and Jens Lehmann. 2019. Learning to rank query graphs for complex question answering over knowledge graphs. In International semantic web conference, pages 487-504. Springer.
Salman Mohammed, Peng Shi, and Jimmy Lin. 2018. Strong baselines for simple question answering over knowledge graphs with and without neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 291-296, New Orleans, Louisiana. Association for Computational Linguistics.
Junwoo Park, Youngwoo Cho, Haneol Lee, Jaegul Choo, and Edward Choi. 2020. Knowledge graph-based question answering with electronic health records. arXiv preprint arXiv:2010.09394.
Bryan Perozzi, Vivek Kulkarni, Haochen Chen, and Steven Skiena. 2017. Don't walk, skip! online learning of multi-scale network embeddings. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, pages 258-265.
Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michihiro Yasunaga, Haitian Sun, Dale Schuurmans, Jure Leskovec, and Denny Zhou. 2021. Lego: Latent execution-guided reasoning for multi-hop question answering on knowledge graphs. In International Conference on Machine Learning, pages 8959-8970. PMLR.
Benedek Rozemberczki and Rik Sarkar. 2020. Characteristic functions on graphs: Birds of a feather, from statistical descriptors to parametric models. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1325-1334.
Amrita Saha, Ghulam Ahmed Ansari, Abhishek Laddha, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2019. Complex program induction for querying knowledge bases in the absence of gold programs. Transactions of the Association for Computational Linguistics, 7:185-200.
Apoory Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498-4507, Online. Association for Computational Linguistics.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer.
Georgios Sidiropoulos, Nikos Voskarides, and Evangelos Kanoulas. 2020. Knowledge graph simple question answering for unseen domains. In *Automated Knowledge Base Construction*.
Saurabh Srivastava, Mayur Patidar, Sudip Chowdhury, Puneet Agarwal, Indrajit Bhattacharya, and Gautam Shroff. 2021. Complex question answering on knowledge graphs using machine translation and multi-task learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3428-3439, Online. Association for Computational Linguistics.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380-2390, Hong Kong, China. Association for Computational Linguistics.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, Brussels, Belgium. Association for Computational Linguistics.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071-2080. PMLR.
Xu Wang, Shuai Zhao, Bo Cheng, Jiale Han, Yingting Li, Hao Yang, and Guoshun Nan. 2020a. Hgman: multi-hop and multi-answer question answering based on heterogeneous knowledge graph (student abstract). In Proceedings of the AAAI Conference