SlowGuess/NAACL_2022_OCR huggingface dataset:
PERKGQA: Question Answering over Personalized Knowledge Graphs
Ritam Dutt*
rdutt@andrew.cmu.edu
Kasturi Bhattacharjee
kastb@amazon.com
drot@amazon.com
Carolyn Penstein Rosé
cp3@andrew.cmu.edu
Previous studies on question answering over knowledge graphs have typically operated over a single knowledge graph (KG). This KG is assumed to be known a priori and is leveraged similarly for all users' queries during inference. However, such an assumption is not applicable to real-world settings, such as healthcare, where one needs to handle queries of new users over unseen KGs during inference. Furthermore, privacy concerns and high computational costs render it infeasible to query the single KG that has information about all users while answering a specific user's query. The above concerns motivate our question answering setting over personalized knowledge graphs (PERKGQA) where each user has restricted access to their KG. We observe that current state-of-the-art KGQA methods that require learning prior node representations fare poorly. We propose two complementary approaches, PATHCBR and PATHRGCN for PERKGQA. The former is a simple non-parametric technique that employs case-based reasoning, while the latter is a parametric approach using graph neural networks. Our proposed methods circumvent learning prior representations, can generalize to unseen KGs, and outperform strong baselines on an academic and an internal dataset by  $6.5\%$  and  $10.5\%$ .
The task of Question Answering over Knowledge Graphs (KGQA), involves answering a natural language question by querying a predefined knowledge graph (KG), such as WikiData or Freebase. Progress in KGQA research has addressed several challenges, such as answering complex questions, multi-hop reasoning, (Lan and Jiang, 2020; Ren et al., 2021), conversational KGQA (Kacupaj et al., 2021), and multi-lingual KGQA (Zhou et al., 2021), and has also found applications in tax, in
surance, and healthcare (Ludemann et al., 2020; Huang et al., 2021; Park et al., 2020).
Most KGQA research has focused on generalizable or generic knowledge, which assumes there is a predefined global KG for all queries. This assumes that nodes used during inference were already defined in the KG during training and holds for cases that focus on generalizable knowledge. This work proposes approaches that circumvent the need to make such an assumption.
Furthermore, using a single global KG to handle queries of different users raises additional concerns, especially when a user's query requires situated knowledge such as personal information.
- Scalability: The massive size of the global KG makes it computationally expensive to apply sophisticated neural architectures over it.
- Privacy: The unfettered access to information of all individuals raises ethical or legal concerns.
In this paper, we formulate PERKGQA or question answering over personalized knowledge graphs. Here the user has access to their specific KG, a subset of the global KG that contains only the information relevant to the user. We are restricted to the user's KG to answer their queries during training and inference. Such a setting addresses the challenges above of scalability, privacy, and generalizing over unseen KGs.
PERKGQA appears deceivingly simple in conception since we afford access to a subset of the larger global KG. One can claim that our setting is similar to the KGQA subtask where subgraphs and questions are predefined, and thus, traditional KGQA methods are applicable. However, information retrieval based KGQA methods employ knowledge graph completion techniques like TransE (Bordes et al., 2013) to learn node representations over the global KG and reuse them during inference. Alternately, other approaches leverage additional information such as semantic parses, logical forms, and query graphs to answer queries.
* Work done during internship at Amazon AWS AI Labs.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 253 - 268
This sets PERKGQA apart because we lack access to any prior information, be it text, semantic parses, or prior representations of KG nodes. Our setting requires learning node representations from scratch for each KG to handle unknown entities during inference. Moreover, other challenges prevalent in KGQA settings, namely multi-hop reasoning or answering complex/constraint-based questions, are also applicable to PERKGQA. To the best of our knowledge, we are the first to address the challenges of KGQA over unseen KGs in the absence of any additional information.
We propose two approaches, PATHCBR and PATHRGCN, that are well-suited to these settings. PATHCBR is a simple non-parametric case-based reasoning approach that encodes path information of past queries to answer a new query. PATHRGCN is a parametric approach that employs graph neural networks, path information, and the KG's structure to extract answers. These approaches circumvent the need for learning prior node representations and can be readily applied to unseen KGs.
Contributions of the paper:
- We formulate PERKGQA, a new setting for KGQA where we operate over unseen KGs in the absence of any additional information. We observe that SOTA methods that need to learn underlying node embeddings fare poorly.
- To encourage research, we modify an existing academic dataset (Yih et al., 2016) and make it available for research (as Mod-WebQSP).
- We propose PATHCBR and PATHRGCN, which outperform baselines on Mod-WebQSP and an internal dataset by  $6.5\%$  and  $10.5\%$  respectively.
A Knowledge Graph (KG) is represented as  $\mathcal{K} = (\mathcal{V},\mathcal{E},\mathcal{R})$ , where  $\mathcal{V}$  is the set of entities,  $\mathcal{R}$  is the set of relations, and  $\mathcal{E}$  is the set of triplets.  $(e_1,r,e_2)$ ,  $e_1,e_2\in \mathcal{V}$ , and  $r\in \mathcal{R}$ . Thus  $\mathcal{E}\subset (\mathcal{V}\times \mathcal{R}\times \mathcal{V})$ . Given a natural language question  $q$ , the objective of KGQA is to retrieve answer entities from  $\mathcal{V}$ .
For PERKGQA, we treat each question as posed by a separate user, and each question is associated with its corresponding knowledge graph,  $\mathcal{K}_q$ . A given  $\mathcal{K}_q$  has a subset of nodes,  $\nu_q$  and relations,  $\mathcal{R}_q$ . Two knowledge graphs,  $\mathcal{K}_q$  and  $\mathcal{K}_{q^*}$  associated with questions  $q$  and  $q^{*}$  can have a varying degree of overlap, even being distinctly different.
2.2 Running Example
We now demonstrate the applicability of PERKGQA for a cloud service provider (e.g. Microsoft Azure) in Figure 1. Here, users (blue and red) can create cloud resources (yellow), and index them using a unique system identifier. These resources have a corresponding user-specific tag (green), are located in a specific region (orange), and have predefined services deployed on them (purple). The entire system can be envisioned as a knowledge graph (CloudKG) where nodes represent concepts (users and services), and edges define the relations between concepts. Due to confidentiality, user names are replaced with anonymous identifiers, while concept and relation names in CloudKG are modified. The underlying schema is unchanged.
Deploying a chatbot-based assistant that performs QA over CloudKG would facilitate use, especially by novice users. It would enable users to navigate the system and glean information by posing natural language questions. In Figure 1, when User 101 asks "Which resources have nlp-serv and demo_1 tags?", the system is expected to answer "res_1, res_3". We refer to Figure 1 as a running example in subsequent sections. As new users become a part of CloudKG, the QA system should accommodate their requests over the corresponding KG without any training. KGQA approaches that operate upon the entire CloudKG would be computationally infeasible due to the massive size of the user-base<sup>1</sup>. Moreover, the approach should be privacy-preserving wherein a given user's information is not revealed to another.
We operate on two datasets: an internal dataset, CloudKGQA, built on top of CloudKG, and an academic dataset called Mod-WebQSP designed to mimic our setting. An instance in either dataset follows the same task formulation in Section 2, namely, for each question  $q$ , there exists a corresponding KG,  $K_{q}$ , which contains all the necessary information. Also, each question  $q$  is associated with one or more source entities; these correspond to nodes in the  $K_{q}$  linked through salient mentions of entities in  $q$ . E.g., the source entity for, "Who was responsible for Lincoln's assassination?" is the node corresponding to Abraham Lincoln.
<sup>1</sup>https://www.statista.com/statistics/321215/global-consumer-cloud-computing-users/
Figure 1: PERKGQA for a cloud service provider setting. The two users (in blue and red) create cloud resources (in yellow) in specific regions (in orange), and deploy services e.g. Chatbot service, or Analytics (in purple) on them. The users assign customized tags (in green) to the resources. Each user has their unique KG. The system should scale to support queries of new users over unseen KGs without any retraining or additional knowledge.
3.1 CloudKGQA
The internal dataset, which we refer to as CloudKGQA, entails question-answering of a customer's queries on their respective cloud resources. We refer the readers to Figure 1 as we present examples that outline the key characteristics of CloudKGQA.
- Multiple Answers: A question can have one or more correct answers.
- Varying Complexity: A question can either be simple or complex.
(i) Simple: The question can be answered by a single-hop relation, e.g. "Which resource has the tag nlpServe?"
(ii) Complex : The question involves logical operations like union or intersection, e.g. "Show me resources in US and India" or contains multiple constraints, e.g. "Which resource has the TTS and MongoDB service and is located in US?" has three constraints, TTS, MongoDB, and US.
- Multi-Hop distance: The distance between the source entities and the answers is variable (e.g., the number of hops for "Show me tags for resources in US" is 2 in Figure 1).
- Variable graph size: The size of the KG varies in terms of the number of nodes, edges, and relations for each question.
- Unseen nodes: Nodes that appear in the KG during inference might not be seen while training.
3.2 Modified WebQSP (Mod-WebQSP)
We also operate on the publicly-available WebQSP dataset (Yih et al., 2016), built over Freebase  $(\mathcal{F})$  We chose WebQSP since it shares similar char
acteristics of CloudKGQA, namely the presence of multi-answer, multi-hop, simple and complex questions. To completely mimic our setting, we construct a KG,  $\mathcal{F}_q$  for each question  $q$ , with the caveat that a significant fraction of nodes remains unseen during inference. We describe our process for creating individual KG in the Appendix A. Our modification achieves a low overlap of  $4.0\%$  between entities across training and test splits, implying that  $96\%$  of entities remain unseen.
3.3 Differences between the datasets
We present the descriptive statistics of the two datasets in Table 1 corresponding to the mean number of nodes, edges, relations, answers, and hops for a KG. We also depict the degree of overlap between nodes in training and test splits. The number of instances in CloudKGQA and Mod-WebQSP are 800 and 4468, respectively. Moreover, we split the data into train, development, and test for both datasets in the ratio of 8:1:1.
We observe that CloudKGQA is comparatively smaller in size, had significantly fewer relations, but had longer reasoning chains. Moreover, CloudKGQA had more complex questions in terms of logical operations and multiple-constraints. Specifically, CloudKGQA had one or more source entities for each question,  $q$ , whereas Mod-WebQSP had only one source entity. The KGs in CloudKGQA had a similar underlying schema; different KGs had the same set of relations but different entities. However, the questions in the test data had distinct question templates from those during training, as
1) Retrieve Similar Questions
Figure 2: PATHCBR Overview: (1) Retrieve questions similar to a given query template from set of questions; (2) Encode path information as a path embedding; (3) Score generated paths using the retrieved path embedding.
seen in Figure 2. The Mod-WebQSP dataset, on the other hand, had KGs with different relations, but questions in the test data were similar to those asked during training. We chose these two datasets because they capture two different scenarios.
Dataset	CloudKGQA	Mod-WebQSP
Nodes	23.39	518.21
Edges	35.59	1334.10
Relations	8.00	36.20
Answers	1.99	4.94
Hops	1.75	1.36
Overlap	3.21%	4.01%
Table 1: An overview of the statistics of the two datasets, CloudKGQA and Mod-WebQSP. We present the mean number of nodes, edges, relations, answers, and hops, and the overlap between nodes during test and train.
4.1 PATHCBR
PATHCBR is a non-parametric approach that employs case-based reasoning to retrieve queries without any training. Given a question  $q$ , the corresponding knowledge graph  $\mathcal{K}_q$  and the source entities,  $s_1, s_2, \dots, s_k$ , PATHCBR (Figure 2) performs the following steps:
(i) Query Retrieval: For a query,  $q$ , we first retrieve similar questions from the available training set. We consider a question to be similar if they share similar answer types with the query rather than the entities (Das et al., 2020). We perform Named Entity Recognition (NER) to identify text-spans that correspond to source entities  $s_1, s_2, \dots, s_k$  in  $K_q$  (Sun et al., 2019; Wang et al., 2020b). We substitute the extracted text spans with a special [MASK] token, yielding the masked query template  $q_{\text{MASK}}$ . We hypothesize that masking entities can help us learn the association of the entity
with the template and could generalize to unseen entities. We employ a pretrained language model, such as RoBERTa, to create a contextualized embedding of  $q_{\mathrm{MASK}}$  and call it  $v_{q}$ . We then retrieve the top  $n$  questions  $(q_{1}, \dots, q_{n})$  and their respective KGs,  $(\mathcal{K}_{q_1}, \dots, \mathcal{K}_{q_n})$  ranked by decreasing cosine similarity between  $v_{q}$  and  $v_{q_i}$ . The  $v_{q_i}$  are created in the same manner as  $v_{q}$ . We represent the steps of masking and retrieving below:
$$
q _ {\text {M A S K}} \leftarrow \operatorname {M A S K} (q)
$$
$$
v _ {q} \leftarrow \operatorname {R O B E R T A} \left(q _ {\text {M A S K}}\right)
$$
$$
\left(q _ {1}, \mathcal {K} _ {q _ {1}}\right), \dots , \left(q _ {n}, \mathcal {K} _ {q _ {n}}\right) \leftarrow \operatorname {R E T R I E V} \left(v _ {q}\right)
$$
(ii) Encoding path information: We now construct the answer paths for the retrieved KGs  $\mathcal{K}_{q_i}$ . An answer path  $p_{s_{ij},a_{ik}}$  comprises a sequence of relations, starting from a source  $s_{ij}$  entity to the answer entity  $a_{ik}$  in  $\mathcal{K}_{q_i}$ . There can be multiple answer paths between the source and the answer, but for simplicity we consider only the shortest paths, similar to Srivastava et al. (2021). We represent an answer path, either explicitly as a sequence of relations  $(r_{i1},r_{i2},..r_{im})$  leading from  $s_{ij}$  to  $a_{ik}$ , or by pooling over its constituent relation embeddings  $(v_{r_{i1}},v_{r_{i2}},..v_{r_{im}})$ . We describe different approaches to obtain the relation embedding  $v_{r_i}$  in Section 5. Once we have embeddings for individual paths, we pool across all possible answer paths over the retrieved KGs,  $\mathcal{K}_q$  to obtain the retrieved path embedding,  $v_P^q$  for  $q$ . We describe the steps to encode the path information below:
$$
p _ {s _ {i j}, a _ {i k}} \leftarrow \left[ r _ {i 1}, r _ {i 2}, \dots , r _ {i m} \right]
$$
$$
v _ {p _ {s _ {i j}, a _ {i k}}} \leftarrow \operatorname {M A X - P O O L} \left(\left[ v _ {r _ {i 1}}, v _ {r _ {i 2}}, \dots , v _ {r _ {i m}} \right]\right)
$$
$$
v _ {P} ^ {q} \leftarrow \operatorname {M A X - P O O L} \left(\left[ \forall v _ {p _ {s _ {i j}, a _ {i k}}} \right]\right)
$$
(iii) Scoring generated paths: For the given query  $q$ , we generate all possible paths of a certain length,
Figure 3: PATHRGCN Overview: (1) Initialize the question using a pretrained language model (PTLM) and the nodes in the corresponding KG; (2) Perform information propagation using RGCN to update node embeddings; (3) Encode path information from the source entities (shown in green) to all possible target nodes by pooling over the constituent node embeddings; (4) Perform answer prediction at both the path and node level.
arising from  $s_1, s_2, \dots, s_k$ . The length of the path is determined by the maximum length of the answer path encountered during retrieval. These generated paths (say  $p_j$ ) constitute a sequence of relations arising from the source node (say  $r_1, r_2, \dots, r_m$ ), similar to the retrieved paths. We encode them by pooling over the constituent relation embeddings to obtain  $v_{p_j}$ , the generated path embedding. We finally score the generated path embedding against the retrieved path embedding  $v_P^q$ ; a higher similarity implies that the generated path is more likely to lead to an answer. However, if we store the path information explicitly as a sequence of relations, then the nodes we reach by traversing the retrieved sequences are answers for  $q$ . The equations follow:
$$
p _ {j} \gets [ r _ {1}, \dots , r _ {i m} ]
$$
$$
v _ {p _ {j}} \leftarrow \operatorname {M A X - P O O L} \left(\left[ v _ {r _ {1}}, \dots , v _ {r _ {m}} \right]\right)
$$
$$
\operatorname {s c o r e} \left(v _ {p _ {j}}\right) \leftarrow \operatorname {S I M} \left(v _ {p _ {j}}, v _ {P} ^ {q}\right).
$$
4.2 PATHRGCN
We now propose our parametric PATHRGCN model that can encode and fine-tune path embeddings for KGQA. Given a question  $q$ , the corresponding knowledge graph  $\mathcal{K}_q$  and the source entities,  $s_1, s_2, \dots, s_k$ , PATHRGCN (Figure 3), encompass the following steps during training:
(i) Initialization: We encode  $q$  using a pretrained language model (PTLM) such as RoBERTa (Liu et al., 2019), to obtain the corresponding representation,  $v_{q}$ . We use unsupervised graph representation learning techniques like Node2Vec (Grover and Leskovec, 2016) and Walklet (Perozzi et al., 2017), that leverage the neighbourhood information of nodes in  $\mathcal{K}_q$  to obtain the corresponding
embeddings:  $v_{e_1}, v_{e_2}, \dots, v_{e_N}$  for the  $N$  nodes  $e_1, e_2, \dots, e_N$  in  $\mathcal{K}_q$ . Unlike Wang et al. (2020a,b), we do not use pretrained word embeddings since user-provided names can be arbitrary.
$$
v _ {q} \leftarrow \operatorname {R O B E R T A} (q)
$$
$$
v _ {e _ {1}}, v _ {e _ {2}}, \dots , v _ {e _ {N}} \leftarrow \operatorname {W A L K L E T} \left(\mathcal {K} _ {q}\right).
$$
(ii) Information propagation using GNN: We employ graph neural networks (GNN) to update the node representations of  $\kappa_{q}$ . We modify  $\kappa_{q}$  by adding the inverse-relations between nodes and self-loops to facilitate information propagation across both directions similar to Wang et al. (2020a,b). We concatenate  $v_{e_i}$  with  $v_{q}$  and a binary value of  $b_{i}$ .  $b_{i}$  has a value of 1 or 0, corresponding to whether  $e_i$  is a source entity. The resultant representation,  $h_{e_i}^0 = [v_q,v_{e_i},b_i]$ , is then passed as input to the first GNN layer, and the representations of all nodes are updated. We perform such updates L times, where L denotes the number of GNN layers, resulting in the final representation of  $h_{e_i}^L$ . We use softmax as the non-linear activation and add dropout for regularization between updates. We use the RGCN model (Schlichtkrull et al., 2018) to account for different relationships between nodes.
$$
h _ {e _ {i}} ^ {0} \gets v _ {q} \oplus v _ {e _ {i}} \oplus b _ {i}
$$
$$
h _ {e _ {i}} ^ {j + 1} \gets \mathrm {R G C N} (h _ {e _ {i}} ^ {j})
$$
(iii) Path embedding generation: We construct all possible paths  $p_1, p_2, \ldots, p_m$  up to a fixed distance from the source entities, and generate their corresponding path embeddings. The embeddings for path  $p_j$  or  $v_{p_j}$  is obtained by pooling over the updated representations of the nodes that constitutes  $p_j$ . We hypothesize that learning the path structure
can provide intermediate supervision (Srivastava et al., 2021) and can help prune-out nodes that are unlikely to be reached from the source.
$$
v _ {p _ {j}} \gets \mathrm {M A X - P O O L} (h _ {e _ {i}} ^ {L}) \forall e _ {i} \in p _ {j}
$$
(iv) Answer prediction: We perform answer prediction both at the node and path level. We concatenate the updated representation for node  $e_i$  as  $h_{e_i}^L$ , with the question-embedding  $v_q$ , and pass it through a linear layer with sigmoid activation. to obtain  $\hat{y}_{e_i}$ . This represents the probability of  $e_i$  being an answer and is trained against the ground truth value of  $y_{e_i}$ . We perform the same procedure at the path level to obtain the probability of path  $p_j$  that leads to  $e_i$  as  $(y_{p_j, e_i})$ . We use binary cross-entropy loss for answer prediction at the node level (NL) and path level (PL) and minimize these losses jointly during training. Specifically:
$$
\hat {y} _ {e _ {i}} \leftarrow \sigma \left(\operatorname {F F N} \left(h _ {e _ {i}} ^ {L} \oplus v _ {q}\right)\right)
$$
$$
\hat {y} _ {p _ {j}, e _ {i}} \leftarrow \sigma \left(\operatorname {F F N} \left(v _ {p _ {j}} \oplus v _ {q}\right)\right)
$$
$$
\mathrm {N L} = - \sum_ {e _ {i} \in \mathcal {K} _ {q}} y _ {e _ {i}} \cdot l o g (\hat {y} _ {e _ {i}})
$$
$$
\mathrm {P L} = - \sum_ {e _ {i}} \sum_ {\forall p _ {j} \sim e _ {i}} y _ {e _ {i}} \cdot l o g (\hat {y} _ {p _ {j}, e _ {i}})
$$
Inference: During inference, given a question  $q^{*}$  and its corresponding sub-graph  $\kappa_{q^*}$ , the learnt PATHRGCN models outputs (i) probability that the node  $e_1, e_2, \dots, e_N$  is an answer and (ii) probability that the paths  $p_1, p_2, \dots, p_m$  leads to an answer. Thus for a given entity,  $e_i$ , we compute the maximum probability amongst all paths that end in  $e_i$ . We compute the mean of this probability alongside the probability of  $e_i$  being an answer.
We choose GNN-based retrieval models as our baselines since they have achieved high performance across different KGQA datasets without additional information (query-answer paths or semantic parses). We experiment with three relevant KGQA retrieval techniques, namely, Embed-KGQA (Saxena et al., 2020), Rel-GCN (Wang et al., 2020a), and GlobalGraph (Wang et al., 2020b). We do not use baselines that require additional textual information to generate the heterogeneous graph, such as GraftNet (Sun et al., 2018)
or PullNet (Sun et al., 2019) since this information is not available to us. We present a detailed description of the baselines in the Appendix B.1.
PATHCBR: We experiment with how masking entities impact QA performance. For CloudKGQA, we identify entities by performing string-match over text spans in the question to their corresponding nodes in the KG. For Mod-WebQSP, we use the publicly available SpaCy  $\mathrm{NER}^2$ . We also experiment with SpaCy's POS-Tagger to mask proper nouns. The masked query is encoded using the [CLS] token of RoBERTa-BASE (Liu et al., 2019). We experiment with different ways to encode relations, either as a one-hot vector or using RoBERTa-BASE to encode the text. We perform max-pooling over the constituent relation embedding to obtain the resultant path-embedding. Likewise, max-pooling over the resultant path-embeddings yields the retrieved path-embedding. We also experimented with mean-pooling, but max-pooling fared consistently better. The generated paths are similarly encoded during inference. We compute cosine-similarity between a generated and retrieved path embedding. We retrieve the top 5 questions in descending order of their similarity for a given query.