SlowGuess/ACL_2020_OCR huggingface dataset:
Apoorv Saxena* Aditay Tripathi* Partha Talukdar
{apoorvsaxena,aditayt,ppt}@iisc.ac.in
Knowledge Graphs (KG) are multi-relational graphs consisting of entities as nodes and relations among them as typed edges. Goal of the Question Answering over KG (KGQA) task is to answer natural language queries posed over the KG. Multi-hop KGQA requires reasoning over multiple edges of the KG to arrive at the right answer. KGs are often incomplete with many missing links, posing additional challenges for KGQA, especially for multi-hop KGQA. Recent research on multi-hop KGQA has attempted to handle KG sparsity using relevant external text, which isn't always readily available. In a separate line of research, KG embedding methods have been proposed to reduce KG sparsity by performing missing link prediction. Such KG embedding methods, even though highly relevant, have not been explored for multi-hop KGQA so far. We fill this gap in this paper and propose EmbedKGQA. EmbedKGQA is particularly effective in performing multi-hop KGQA over sparse KGs. EmbedKGQA also relaxes the requirement of answer selection from a prespecified neighborhood, a sub-optimal constraint enforced by previous multi-hop KGQA methods. Through extensive experiments on multiple benchmark datasets, we demonstrate EmbedKGQA's effectiveness over other state-of-the-art baselines.
Knowledge Graphs (KG) are multi-relational graphs consisting of millions of entities (e.g., San Jose, California, etc.) and relationships among them (e.g., San Jose-cityInState-California). Examples of a few large KGs include Wikidata (Google, 2013), DBPedia (Lehmann et al., 2015), Yago (Suchanek et al., 2007), and NELL (Mitchell
Question: What are the genres of movies written by Louis Mellis? Answer : Crime
Figure 1: Challenges with Multi-hop QA over Knowledge Graphs (KGQA) in sparse and incomplete KGs: Absence of the edge has genre(Gangster No. 1, Crime) in the incomplete KG makes it much harder to answer the input NL question, as the KGQA model potentially needs to reason over a longer path over the KG (marked by bold edges). Existing multi-hop KGQA methods also impose heuristic neighborhood limits (shaded region in the figure), which often makes the true answer (Crime in this example) out of reach. EmbedKGQA, our proposed method, overcomes these limitations by utilizing embeddings of the input KG during multi-hop KGQA. For more details, please refer Figure 2 and Section 4.
et al., 2018). Question Answering over Knowledge Graphs (KGQA) has emerged as an important research area over the last few years (Zhang et al., 2018; Sun et al., 2019a). In KGQA systems, given a natural language (NL) question and a KG, the right answer is derived based on analysis of the question in the context of the KG.
In multi-hop KGQA, the system needs to perform reasoning over multiple edges of the KG to infer the right answer. KGs are often incomplete, which creates additional challenges for KGQA systems, especially in case of multi-hop KGQA. Recent methods have used an external text corpus to handle KG sparsity (Sun et al., 2019a, 2018). For example, the method proposed in (Sun et al., 2019a) constructs a question-specific sub-graph from the KG, which is then augmented with supporting text documents.
*Equal contribution
EmbedKGQA's source code is available at https://github.com/mallabiisc/EmbedKGQA
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498-4507 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Overview of EmbedKGQA, our proposed method for Multi-hop QA over Knowledge Graphs (KGQA). EmbedKGQA has three modules: (1) KG Embedding Module (Section 4.2) learns embeddings for all entities in the input KG, (2) Question Embedding Module (Section 4.3) learns an embedding for the question, and (3) the Answer Selection Module (Section 4.4) selects the final answer by incorporating the question and relation similarity scores. EmbedKGQA's use of embeddings makes it more effective in handling KG sparsity. Moreover, since EmbedKGQA considers all entities as candidate answers, it doesn't suffer from the limited neighborhood out-of-reach issues of existing Multi-hop KGQA methods. Please refer Section 4 for detailed description of EmbedKGQA.
Graph CNN (Kipf and Welling, 2016) is then applied over this augmented sub-graph to arrive at the final answer. Unfortunately, availability and identification of relevant text corpora is a challenge on its own which limits broad-coverage applicability of such methods. Moreover, such methods also impose pre-specified heuristic neighborhood size limitation from which the true answer needs to be selected. This often makes the true answer out of reach of the model to select from.
In order to illustrate these points, please consider the example shown in Figure 1. In this example, Louis Mellis is the head entity in the input NL question, and Crime is the true answer we expect the model to select. If the edge has genre(Gangster No. 1, Crime) were present in the KG, then the question could have been answered rather easily. However, since this edge is missing from the KG, as is often the case with similar incomplete and sparse KGs, the KGQA model has to potentially reason over a longer path over the KG (marked by bolded edges in the graph). Moreover, the KGQA model imposed a neighborhood size of 3-hops, which made the true answer Crime out of reach.
In a separate line of research, there has been a large body of work that utilizes KG embeddings to predict missing links in the KG, thereby reducing KG sparsity (Bordes et al., 2013; Trouillon et al.,
2016; Yang et al., 2014a; Nickel et al., 2011). KG embedding methods learn high-dimensional embeddings for entities and relations in the KG, which are then used for link prediction. In spite of its high relevance, KG embedding methods have not been used for multi-hop KGQA – we fill this gap in this paper. In particular, we propose **EmbedKGQA**, a novel system which leverages KG embeddings to perform multi-hop KGQA. We make the following contributions in this paper:
1. We propose EmbedKGQA, a novel method for the multi-hop KGQA task. To the best of our knowledge, EmbedKGQA is the first method to use KG embeddings for this task. EmbedKGQA is particularly effective in performing multi-hop KGQA over sparse KGs.
2. EmbedKGQA relaxes the requirement of answer selection from a pre-specified local neighborhood, an undesirable constraint imposed by previous methods for this task.
3. Through extensive experiments on multiple real-world datasets, we demonstrate Embed-KGQA's effectiveness over state-of-the-art baselines.
We have made EmbedKGQA's source code available to encourage reproducibility.
KGQA: In prior work (Li et al., 2018) TransE, (Bordes et al., 2013) embeddings have been used to answer factoid based questions. However, this requires ground truth relation labeling for each question and it does not work for multi-hop question answering. In another line of work (Yih et al., 2015) and (Bao et al., 2016) proposed extracting a particular sub-graph to answer the question. The method presented in (Bordes et al., 2014a), the sub-graph generated for a head entity is projected in a high dimensional space for question answering. Memory Networks have also been used to learn high dimensional embeddings of the facts present in the KG to perform QA (Bordes et al., 2015). Methods like (Bordes et al., 2014b) learn a similarity function between the question and the corresponding triple during training, and score the question with all the candidate triples at the test time. (Yang et al., 2014b) and (Yang et al., 2015) utilize embedding based methods to map natural language questions to logical forms. Methods like (Dai et al., 2016; Dong et al., 2015; Hao et al., 2017; Lukovnikov et al., 2017; Yin et al., 2016) utilize neural networks to learn a scoring functions to rank the candidate answers. Some works like (Mohammed et al., 2017; Ture and Jojic, 2016) consider each relation as a label and model QA task as a classification problem. Extending these kinds of approaches for multi-hop question answering is non-trivial.
Recently, there has been some work in which text corpus is incorporated as a knowledge source in addition to KG to answer complex questions on KGs (Sun et al., 2018, 2019a). Such approaches are useful in case the KG is incomplete. However, this leads to another level of complexity in the QA system, and text corpora might not always be available.
KG completion methods: Link prediction in Knowledge Graphs using KG embeddings has become a popular area of research in recent years. The general framework is to define a score function for a set of triples  $(h, r, t)$  in a KG and constraining them in such a way that the score for a correct triple is higher than the score for an incorrect triple.
RESCAL (Nickel et al., 2011) and DistMult (Yang et al., 2015) learn a score function containing a bi-linear product between head entity and tail entity vectors and a relation matrix. ComplEx (Trouillon et al., 2016) represents entity vectors and relation matrices in the complex space. SimplE
(Kazemi and Poole, 2018) and TuckER (Balažević et al., 2019) are based on Canonical Polyadic (CP) decomposition (Hitchcock, 1927) and Tucker decomposition (Tucker, 1966) respectively.
TransE (Bordes et al., 2013) embeds entities in high dimensional real space and relation as translation between the head and the tail entities. RotatE (Sun et al., 2019b) on the other hand projects entities in complex space and relations are represented as rotations in the complex plane.
ConvE (Dettmers et al., 2018) utilizes Convolutional Neural Networks to learn a scoring function between the head entity, tail entity and relation. InteractE (Vashisth et al., 2019) improves upon ConvE by increasing feature interaction.
In this section, we formally define a Knowledge Graph(KG) and then describe link prediction task on incomplete KGs. We then describe KG embeddings and explain the ComplEx embedding model.
3.1 Knowledge Graph
Given a set of entities  $\mathcal{E}$  and relations  $\mathcal{R}$ , a Knowledge Graph  $\mathcal{G}$  is a set of triples  $\mathcal{K}$  such that  $\mathcal{K} \subseteq \mathcal{E} \times \mathcal{R} \times \mathcal{E}$ . A triple is represented as  $(h, r, t)$ , with  $h, t \in \mathcal{E}$  denoting subject and object entities respectively and  $r \in \mathcal{R}$  the relation between them.
3.2 Link Prediction
In link prediction, given an incomplete Knowledge Graph, the task is to predict which unknown links are valid. KG Embedding models achieve this through a scoring function  $\phi$  that assigns a score  $s = \phi(h, r, t) \in \mathbb{R}$ , which indicates whether a triple is true, with the goal of being able to score all missing triples correctly.
3.3 Knowledge Graph Embeddings
For each  $e \in \mathcal{E}$  and  $r \in \mathcal{R}$ , Knowledge Graph Embedding (KGE) models generate  $e_e \in \mathbb{R}^{d_e}$  and  $e_r \in \mathbb{R}^{d_r}$ , where  $e_e$  and  $e_r$  are  $d_e$  and  $d_r$  dimensional vectors respectively. Each of the embedding methods also has a scoring function  $\phi: \mathcal{E} \times \mathcal{R} \times \mathcal{E} \to \mathbb{R}$  to assign some score  $\phi(h, r, t)$  to a possible triple  $(h, r, t)$ ,  $h, t \in \mathcal{E}$  and  $r \in \mathcal{R}$ . Models are trained in a way such that for every correct triple  $(h, r, t) \in \mathcal{K}$  and incorrect triple  $(h', r', t') \notin \mathcal{K}$  the model assigns scores such that  $\phi(h, r, t) > 0$  and  $\phi(h', r', t') < 0$ . A scoring function is generally a function of  $(e_h, e_r, e_t)$ .
3.3.1 ComplEx Embeddings
ComplEx (Trouillon et al., 2016) is a tensor factorization approach that embeds relations and entities in complex space. Given  $h, t \in \mathcal{E}$  and  $r \in \mathcal{R}$ , ComplEx generates  $e_h, e_r, e_t \in \mathbb{C}^d$  and defines a scoring function:
$$
\begin{array}{l} \phi (h, r, t) = \operatorname {R e} (\langle e _ {h}, e _ {r}, \bar {e} _ {t} \rangle) \\ = \operatorname {R e} \left(\sum_ {k = 1} ^ {d} e _ {h} ^ {(k)} e _ {r} ^ {(k)} \bar {e} _ {t} ^ {(k)}\right) \tag {1} \\ \end{array}
$$
such that  $\phi (h,r,t) > 0$  for all true triples, and  $\phi (h,r,t) <   0$  for false triples. Re denotes the real part of a complex number.
4 EmbedKGQA: Proposed Method
In this section, we first define the problem of KGQA and then describe our model.
Let  $\mathcal{E}$  and  $\mathcal{R}$  be the set of all entities and relations respectively in a KG  $\mathcal{G}$ , and  $\mathcal{K} \subseteq \mathcal{E} \times \mathcal{R} \times \mathcal{E}$  is the set of all available KG facts. The problem in KGQA involves, given a natural language question  $q$  and a topic entity  $e_h \in \mathcal{E}$  present in the question, the task is to extract an entity  $e_t \in \mathcal{E}$  that correctly answers the question  $q$ .
4.1.1 EmbedKGQA Overview
We work in a setting where there is no fine-grained annotation present in the dataset, such as the question type or the exact logic reasoning steps. For example, co-actor is a combination of starred Actor $^{-1}$  and starred actor relations, but our model does not require this annotation.
EmbedKGQA uses Knowledge Graph embeddings to answer multi-hop natural language questions. First it learns a representation of the KG in an embedding space. Then given a question it learns a question embedding. Finally it combines these embedding to predict the answer.
In the following sections, we introduce the EmbedKGQA model. It consists of 3 modules:
1. KG Embedding Module creates embeddings for all entities in the KG.
2. Question Embedding Module finds the embedding of a question
3. Answer Selection Module reduces the set of candidate answer entities and selects the final answer
4.2 KG Embedding Module
ComplEx embeddings are trained for all  $h, t \in \mathcal{E}$  and all  $r \in \mathcal{R}$  in the KG such that  $e_h, e_r, e_t \in \mathbb{C}^d$ . The entity embeddings are used for learning a triple scoring function between the head entity, question, and answer entity. Based on the coverage of the KG entities in the QA training set, the entity embeddings learned here are either kept frozen or allowed to be fine-tuned in the subsequent steps.
4.3 Question Embedding Module
This module embeds the natural language question  $q$  to a fixed dimension vector  $e_q \in \mathbb{C}^d$ . This is done using a feed-forward neural network that first embeds the question  $q$  using RoBERTa (Liu et al., 2019) into a 768-dimensional vector. This is then passed through 4 fully connected linear layers with ReLU activation and finally projected onto the complex space  $\mathbb{C}^d$ .
Given a question  $q$ , topic entity  $h \in \mathcal{E}$  and set of answer entities  $\mathcal{A} \subseteq \mathcal{E}$ , it learns the question embedding in a way such that
$$
\phi \left(e _ {h}, e _ {q}, e _ {a}\right) > 0 \quad \forall a \in \mathcal {A}
$$
$$
\phi \left(e _ {h}, e _ {q}, e _ {\bar {a}}\right) <   0 \quad \forall \bar {a} \notin \mathcal {A}
$$
where  $\phi$  is the ComplEx scoring function (1) and  $e_a, e_{\bar{a}}$  are entity embeddings learnt in the previous step.
For each question, the score  $\phi(.)$  is calculated with all the candidate answer entities  $a' \in \mathcal{E}$ . The model is learned by minimizing the binary cross-entropy loss between the sigmoid of the scores and the target labels, where the target label is 1 for the correct answers and 0 otherwise. Label smoothing is done when the total number of entities is large.
4.4 Answer Selection Module
At inference, the model scores the (head, question) pair against all possible answers  $a' \in \mathcal{E}$ . For relatively smaller KGs like MetaQA, we simply select the entity with the highest score.
$$
e_{ans} = \operatorname *{arg  max}_{a^{\prime}\in \mathcal{E}}\phi (e_{h},e_{q},e_{a^{\prime}})
$$
However if the knowledge graph is large, pruning the candidate entities can significantly improve the performance of EmbedKGQA. The pruning strategy is described in the following section.
Train	Dev	Test
MetaQA 1-hop	96,106	9,992	9.947
MetaQA 2-hop	118,948	14,872	14,872
MetaQA 3-hop	114,196	14,274	14,274
WebQSP	2,998	100	1,639
Table 1: Statistics for MetaQA and WebQuestionsSP datasets. Please refer section 5.1 for more details.
4.4.1 Relation matching
Similar to PullNet (Sun et al., 2019a) we learn a scoring function  $S(r, q)$  which ranks each relation  $r \in \mathcal{R}$  for a given question  $q$ . Let  $h_r$  be the embedding of a relation  $r$  and  $q' = (
, w_1,.., w_{|q|}, 
)$  be the sequence of words in question  $q$  which are input to RoBERTa. The scoring function is defined as the sigmoid of the dot product of the final output of the last hidden layer of RoBERTa  $(h_q)$  and the embedding of relation  $r(h_r)$ .
$$
h _ {q} = \operatorname {R o B E R T a} \left(q ^ {\prime}\right)
$$
$$
S (r, q) = \operatorname {s i g m o i d} \left(h _ {q} ^ {T} h _ {r}\right)
$$
Among all the relations, we select those relations which have score greater than 0.5. It is denoted as the set  $\mathcal{R}_a$ . For each candidate entity  $a'$  that we have obtained so far (Section 4.4), we find the relations in the shortest path between head entity  $h$  and  $a'$ . Let this set of relations be  $\mathcal{R}_{a'}$ . Now the relation score for each candidate answer entity is defined as the size of their intersection.
$$
\operatorname {R e l S c o r e} _ {a ^ {\prime}} = \left| \mathcal {R} _ {a} \cap \mathcal {R} _ {a ^ {\prime}} \right|
$$
We use a linear combination of the relation score and ComplEx score to find the answer entity.
$$
e _ {a n s} = \underset {a ^ {\prime} \in \mathcal {N} _ {h}} {\arg \max} \phi (e _ {h}, e _ {q}, e _ {a ^ {\prime}}) + \gamma * \mathrm {R e l S c o r e} _ {a ^ {\prime}}
$$
where  $\gamma$  is a tunable hyperparameter.
5 Experimental Details
In this section, we first describe the datasets that we evaluated our method on, and then explain the experimental setup and the results.
1. MetaQA (Zhang et al., 2018) dataset is a large scale multi-hop KGQA dataset with
more than 400k questions in the movie domain. It has 1-hop, 2-hop, and 3-hop questions. In our experiments, we used the "vanilla" version of the questions. Along with the QA data, MetaQA also provides a KG with 135k triples, 43k entities, and nine relations.
2. WebQuestionsSP (tau Yih et al., 2016) is a smaller QA dataset with 4,737 questions. The questions in this dataset are 1-hop and 2-hop questions and are answerable through Freebase KG. For ease of experimentation, we restrict the KB to be a subset of Freebase which contains all facts that are within 2-hops of any entity mentioned in the questions of WebQuestionsSP. We further prune it to contain only those relations that are mentioned in the dataset. This smaller KB has 1.8 million entities and 5.7 million triples.
We compare our model with the Key-Value Memory Network (Miller et al., 2016), the GraftNet (Sun et al., 2018) and the Pullnet (Sun et al., 2019a) for WebQuestionsSP dataset. For MetaQA dataset we also compare with the VRN (Zhang et al., 2018). These methods implement multi-hop KGQA, and except VRN, use additional text corpus to mitigate the KG sparsity problem.
- VRN (Zhang et al., 2018) uses variational learning algorithm to perform Multi-Hop QA over KG.
- Key-Value Memory Network (KVMem) (Miller et al., 2016) is one of the first models that attempts to do QA over incomplete KBs by augmenting it with text. It maintains a memory table which stores KB facts and text encoded into key-value pairs and uses this for retrieval.
- GraftNet (Sun et al., 2018) uses heuristics to create a question-specific subgraph containing KG facts, entities and sentences from the text corpora and then uses a variant of graph CNN (Kipf and Welling, 2016) to perform reasoning over it.
- PullNet (Sun et al., 2019a) also creates a question-specific sub-graph but instead of using heuristics, it learns to "pull" facts and sentences from the data to create a more relevant
Model	MetaQA KG-Full	MetaQA KG-50
1-hop	2-hop	3-hop	1-hop	2-hop	3-hop
VRN	97.5	89.9	62.5	-	-	-
GraftNet	97.0	94.8	77.7	64.0 (91.5)	52.6 (69.5)	59.2 (66.4)
PullNet	97.0	99.9	91.4	65.1 (92.4)	52.1 (90.4)	59.7 (85.2)
KV-Mem	96.2	82.7	48.9	63.6 (75.7)	41.8 (48.4)	37.6 (35.2)
EmbedKGQA (Ours)	97.5	98.8	94.8	83.9	91.8	70.3
Table 2: Results on MetaQA dataset. All baseline results were taken from Sun et al. (2019a). We have considered both full KG (MetaQA KG-Full) and  $50\%$  KG (MetaQA KG-50) settings. The numbers reported in this table are hits@1. Numbers in brackets correspond to a setting where text was used to augment the incomplete KG (MetaQA KG-50). For more details please refer section 5.3.1.
sub-graph. It also uses a graph CNN approach to perform reasoning.
The complete KG setting is the easiest setting for QA because the datasets are created in such a way that the answer always exists in the KG, and there is no missing link in the path. However, it is not a realistic setting, and the QA model should also be able to work on an incomplete KG. So we simulate an incomplete KB by randomly removing half of the triples in the KB (we randomly drop a fact with probability  $= 0.5$ ). We call this setting KG-50 and we call full KG setting KG-Full in the text.
In the next section we will answer the following questions:
Q1. Can Knowledge Graph embeddings be used to perform multi-hop KGQA? (Section 5.3)
Q2. Can EmbedKGQA be used to answer questions when there is no direct path between the head entity and the answer entity? (Section 5.4)
Q3. How much does the answer selection module help in the final performance of our model? (Section 5.5)
5.3 KGQA results
In this section, we have compared our model with baseline models on MetaQA and WebQuestionsSP datasets.
5.3.1 Analysis on MetaQA
MetaQA has different partitions of the dataset for 1-hop, 2-hop, and 3-hop questions. In the full KG setting (MetaQA KG-Full) our model is comparable to the state-of-the-art for 2-hop questions and establishes the state-of-the-art for 3-hop questions. EmbedKGQA performs similar to the state-of-the in case of 1-hop question which is expected because the answer node is directly connected to the
head node and it is able to learn the corresponding relation embedding from the question. On the other hand performance on 2-hop and 3-hop questions suggest that EmbedKGQA is able to infer the correct relation from the neighboring edges because the KG embeddings can model composition of relations. Pullnet and GraftNet also perform similarly well because the answer entity lies in the question sub-graph most of the times.
We have also tested our method on the incomplete KG setting, as explained in the previous section. Here we find that the accuracy of all baselines decreases significantly compared to the full KG setting, while EmbedKGQA achieves state-of-the-art performance. This is because MetaQA KG is fairly sparse, with only 135k triples for 43k entities. So when  $50\%$  of the triples are removed (as is done in MetaQA KG-50), the graph becomes very sparse with an average of only 1.66 links per entity node. This causes many head entity nodes of questions to have much longer paths  $(>3)$  to their answer node. Hence models that require question-specific sub-graph construction (GraftNet, PullNet) are unable to recall the answer entity in their generated sub-graph and therefore performs poorly. However, their performance improves only after including additional text corpora. On the other hand, EmbedKGQA does not limit itself to a sub-graph and utilizing the link prediction properties the KG embeddings, EmbedKGQA is able to infer the relation on missing links.
5.3.2 Analysis on WebQuestionsSP
WebQuestionsSP has a relatively small number of training examples but uses a large KG (Freebase) as background knowledge. This makes multi-hop KGQA much harder. Since all the entities of the KG are not covered in the training set, freezing the
Model	WebQSP KG-Full	WebQSP KG-50
KV-Mem	46.7	32.7 (31.6)
GraftNet	66.4	48.2 (49.7)
PullNet	68.1	50.1 (51.9)
EmbedKGQA	66.6	53.2