SlowGuess/NAACL_2022_OCR huggingface dataset:
on Artificial Intelligence, volume 34, pages 13953-13954.
Xu Wang, Shuai Zhao, Jiale Han, Bo Cheng, Hao Yang, Jianchang Ao, and Zhenzi Li. 2020b. Modelling long-distance node relations for KBQA with global dynamic graph. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2572-2582, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Improving question answering over incomplete KBs with knowledge-aware reader. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4258-4264, Florence, Italy. Association for Computational Linguistics.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546, Online. Association for Computational Linguistics.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, Berlin, Germany. Association for Computational Linguistics.
Yucheng Zhou, Xiubo Geng, Tao Shen, Wenqiang Zhang, and Daxin Jiang. 2021. Improving zero-shot cross-lingual transfer for multilingual question answering over knowledge graph. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5822-5834, Online. Association for Computational Linguistics.
Shuguang Zhu, X. Cheng, and Sen Su. 2020. Knowledge-based question answering by tree-to-sequence learning. Neurocomputing, 372:64-72.
A Constructing Mod-WebQSP
We also mimic our setting on the publicly-available WebQSP dataset (Yih et al., 2016), which operates on the Freebase KG,  $\mathcal{F}^1$ . We use the pruned version of the dataset provided by Saxena et al. (2020). To completely mimic our setting, we construct a graph  $\mathcal{F}_q$  associated with each question  $q$  with the caveat that a significant fraction of nodes we encounter during inference are not observed during training.
Each question is associated with a source entity, as noted in the dataset of Saxena et al. (2020). The question's corresponding KG, comprises all nodes, a distance of k-hop from the source entity, where k is the shortest distance between the source and the answer. We limit ourselves to  $k = 2$ , similar to Saxena et al. (2020). Furthermore, to constrain the size of  $\mathcal{F}_q$ , we randomly sample 1000 paths at a k-hop distance from the source entity; these are inclusive of all paths that lead from the source to an answer.
We observe a small fraction of questions  $(\approx 5\%)$  which have  $\geq 100$  answers; these correspond to simple 1-hop questions like "What did Roald Dahl write?", or "Who are famous people from Spain?" We remove such questions to constrain the size of the KG. Since our objective was to retrieve all possible answers in the KG for a given question, there are no missing answers in the KG corresponding to the question in the Mod-WebQSP.
To achieve low overlap between nodes we encountered during training and inference, we modify them by assigning new identifiers. For example, a node, "m.0gzh" corresponding to "Abraham Lincoln", was modified to "KG_i_m.0gzh" and "KG_j_m.0gzh" for questions  $q_{i}$  and  $q_{j}$  in their respective KG  $\mathcal{F}_{q_i}$  and  $\mathcal{F}_{q_j}$ . Although these nodes have the same underlying entity name in the original KG,  $\mathcal{F}$ , their node representations are different in these two questions. We rank all relations in  $\mathcal{F}$ , based on the decreasing order of frequency, and chose the top 39 relations that occur in 95% of all triplets in  $\mathcal{F}$ . We modify only those nodes which are associated with these 39 relations.
We added the graph-identifiers to the most frequent relations to ensure a small degree of overlap between the training and the test sets, similar to
the CloudKGQA dataset, where certain entities were universal, like names of regions (India, USA). This would facilitate prior KGQA techniques, like EmbedKGQA, that perform KGC on the individual KGs to share embeddings and perform better. However, our proposed approaches, PATHCBR, and PATHRGCN remain agnostic to the degree of overlap. They do not keep track of any prior entities. Specifically PATHCBR masks these entities in the question, whereas PATHRGCN learns these entity representations from scratch for each KG.
In this section, we present the baselines in detail and our experimental settings.
EmbedKGQA: The EmbedKGQA model (Saxena et al., 2020) performs Knowledge Graph Completion (KGC) on an existing knowledge graph, to learn node representations. They use ComplEx (Trouillon et al., 2016) to generate node embeddings, to account for the anti-symmetric nature of the relations between nodes. Furthermore, they use RoBERTa (Liu et al., 2019) as the Pre-Trained Language Model (PTLM) to encode the question. They learn an objective function to select answers based on the similarity between question and node embeddings and further perform pruning based on the relation type to prevent over-generation of candidates. EmbedKGQA can perform arbitrary multihop reasoning, is not restricted to a specific neighbourhood, and can effectively handle incomplete links/edges. To ensure EmbedKGQA can be applied in our setting, we carried out KGC on the KG associated with the question instead of the entire Freebase KG. This ensures that the entity representations are distinct for each individual KG.
Rel-GCN: The Rel-GCN approach of Wang et al. (2020a) first constructs a smaller sub-graph  $\kappa_{q}$  for a given question, using PPR (Haveliwala, 2003) from the large base knowledge-graph,  $\kappa$ . They encode the question  $q$  using PTLM as  $v_{q}$ , and use TransE (Bordes et al., 2013) on  $\kappa$  to obtain the node representations  $v_{e_i}$  for node  $e_i$  in  $\kappa$ . They concatenate the node embedding with the question-embedding  $e_q$ , and then perform RGCN on  $\kappa_{q}$  to obtain their updated representations. These updated representations are used to score whether a given node is an answer or not. For PERKGQA setting we perform TransE not on the
1https://developers.google.com/ freebase/data
original graph,  $\mathcal{K}$ , but on each sub-graph  $\mathcal{K}_q$ .
GlobalGraph: The GlobalGraph technique of Wang et al. (2020b) is similar in conception to Rel-GCN, having the same steps, (i) sub-graph construction, (ii) encoding representations of question and nodes, (iii) running RGCN to update the node representations. Moreover, to capture long dependencies between nodes, the model leverages the set of incoming and outgoing relations to assign a global type for each node. They also identify nodes that are correlated with the question and construct a dynamic graph connecting such similar nodes. GCN over this dynamic graph yields updated representations for such nodes. Once again, for PERKGQA, we perform TransE on the individual KG associated with the question  $\mathcal{K}_q$ .
B.2 Experimental Details
We describe the hyper-parameters we employ for the parametric models, namely PATHRGCN and the baselines.
PATHRGCN: For PATHRGCN, we use RoBERTa-BASE to encode the question text, and Walklet (Perozzi et al., 2017) during initialization to generate the unsupervised node-representations for each KG. The embedding sizes for the question, nodes, and GNN layers was set to 768, 128, and 200, respectively. We fix L, the number of GNN layers to 1. For Path-RGCN, the length of an answer-path is chosen based on the maximum distance between a source entity and an answer entity encountered during training. This corresponds to a distance of 3 for CloudKGQA and a distance of 2 for Mod-WebQSP. We used Adam optimizer with a low learning rate of 2e-5, a decay of 5e-4, and patience of 30, and trained for 100 epochs. Each model took around 3 hours to complete on a p3.8x large EC2 instance.
Baselines: For Rel-GCN (Wang et al., 2020a), and GlobalGraph (Wang et al., 2020b), we use RoBERTa-BASE (Liu et al., 2019) to encode the question, and TransE embeddings to initialize the nodes (Bordes et al., 2013). We use the publicly available PyTorch-Geometric library (Fey and Lenssen, 2019) to implement RGCN (Schlichtkrull et al., 2018) for these two baselines. The embedding dimensions for our question, node, and GNN layers are 768, 128, and 200 respectively. The number of GNN layers, was set to 2 and 1 for Rel-GCN and GlobalGraph respectively, as specified in their papers. For EmbedKGQA, we use the publicly
available code of Saxena et al. (2020) along with the default hyper-parameters for training. We use the publicly-available, LibKGE (Broscheit et al., 2020) library to generate Complex embeddings for each KG.
RQ1. What is the impact of entity masking and encoding different path-information strategies on PATHCBR's performance?
We investigate the impact of different strategies for masking entities and encoding path information on the performance of the PERKGQAtask for the two datasets and report them in Table 3.
(i) Entity-masking: For Mod-WebQSP, entity masking using either a publicly-available NER or a POS Tagger, shows a huge boost in performance as seen in Table 3. Masking entities facilitates retrieving relevant questions which share similar answer types rather than similar entity names in the query. For example, for "What county is greeley colorado in ?", the most relevant question retrieved after masking is "What county is novato california in?", as opposed to "What college is in greeley colorado?". We observe a similar trend for CloudKGQA when we mask entities linked to nodes in the KG. However, the performance drops substantially when we use a POS-Tagger. Since the naming convention for nodes is arbitrary, like "abc123", they are not detected as proper nouns; this creates inconsistent templates, and irrelevant questions appear higher in the ranked list.
(ii) Encoding path information: We observe that encoding relations as one-hot vectors fare just as well, if not better than encoding the relation-text using a PTLM. This is especially true for Mod-WebQSP where relation-names have high lexical overlap and thus exhibit high similarity. For example, for "where is jamarcus russell from", the correct relation is "people.person.place_of_birth", but the relation predicted, was "people.person.date_of_birth". Encoding relations as one-hot-vectors circumvents this issue. Encoding the path-information, as a sequence of relations works well for Mod-WebQSP but not for our CloudKGQA, since the questions encountered during inference have different templates.
RQ2. How does our proposed approaches fare against the baselines for different KGQA prop
https://github.com/mallabiisc/EmbedKGQA
(a) CloudKGQA: Accuracy vs # Answers
(c) CloudKGQA: Accuracy vs Subgraph size
(e) CloudKGQA: Accuracy vs # Hops
(g) CloudKGQA: Accuracy vs # source entities
(b) Mod-WebQSP: Accuracy vs # Answers
(d) Mod-WebQSP: Accuracy vs # Subgraph Size
(f) Mod-WebQSP: Accuracy vs # Hops
(h) CloudKGQA: Accuracy vs # Constraints
Figure 4: Performance of the different techniques on the CloudKGQA dataset and the Mod-WebQSP dataset across different properties of the dataset. For CloudKGQA, we investigate the difference in performance based on the number of hops, head-nodes, logical constraints, and KG size. For Mod-WebQSP, we observe the difference in performance based on the number of hops and the size of the subgraph.
erties?
We investigate the performance of the different methods (accuracy) on the PERKGQA task for different properties of the dataset. The methods we investigated were (i) PATHRGCN (ii) PATHCBR (iii) GlobalGraph initialized with Walklet (iv) PATHRGCN initialized with TransE, and (v) GlobalGraph initialized with TransE, the best baseline without any modifications. We investigate the following dataset properties.
(i) Variable number of answers: We observe the performance for variable number of answers, for CloudKGQA in Figure 4a and for Mod-WebQSP in Figure 4b.
(ii) Variable size of the graph: We note the effect of for varying graph size on different methods for
CloudKGQA in Figure 4c and for Mod-WebQSP in Figure 4d.
(iii) Variable Hop Distance: We investigate the performance for varying number of hops for the CloudKGQA in Figure 4e and for Mod-WebQSP in Figure 4f.
(iv) Complex Questions: We observe specifically for CloudKGQA how the accuracy across methods varies for complex questions, based on the varying number of head-nodes in Figure 4g and the number of logical constraints in Figure 4h. This information was available to us for our internal dataset but not for Mod-WebQSP.
For CloudKGQA, we observe that our nonparametric PATHCBR approach achieves the highest performance when the number of answers is
few  $(\leq 3)$ , the subgraph is comparatively smaller (# edges  $\leq 50$ ), the number of hops is few  $(\leq 2)$ , and when there are fewer constraints, (number of logical constraints  $\leq 2$ , and number of source entities  $\leq 3$ ). PATHRGCN boasts a comparative higher performance for the converse scenarios, i.e., greater answers, a larger size of the KG, more hops, and additional constraints. This observation highlights the trade-off between model complexity and the complexity of the question itself. The only exception lies for the 2-hop cases wherein PATHCBR achieves a score of 1.0 because the questions seen during training had a similar template, and answers were found within two hops. Nevertheless, across all sub-cases, we see that our proposed architectures, PATHRGCN or PATHCBR, boasts the highest performance, while the GlobalGraph + TransE, the best performing baseline, achieve the lowest performance. The baseline fares are consistently poorer than the PATHRGCN + TransE, which shows that incorporating the path information was beneficial across all stages.
For Mod-WebQSP, we see that our PATHRGCN model consistently boasts the highest accuracy across all sub-cases. The trend is similar to Cloud-KGQA, where the PATHRGCN model can handle a larger KG size and more considerable hop distance. The only difference is the higher performance of PATHCBR when there are more answers, which is justifiable since the mean number of answers for Mod-WebQSP is five instead of two.
D Ethical Risks
In this paper, we propose PERKGQA, a realistic setting for performing question answering over knowledge graphs; for each user's question, we have their corresponding KG, but no additional information, and we have to perform QA using that limited information. We acknowledge that our setting could be applicable in scenarios involving personalized or sensitive information, like healthcare or insurance providers. Our settings is explicitly designed to deter access to another user's information when a given user poses a query. However, the model might provide different answers to different users despite the same query because the underlying KG is different. We acknowledge that our proposed idea is still in its infancy and requires more research to deem it suitable for real-world applications.