DEIM Forum 2019 G7-5

A Scheme for Factoid Question Answering over Knowledge Base
Buzaaba HAPPY† and Toshiyuki AMAGASA††† Systems and Information Engineering University of Tsukuba†† Center for Computational Sciences University of Tsukuba
1 Chome-１−１ Tennodai, Tsukuba, Ibaraki Prefecture 305-8577, Japan
E-mail: †happy-b@kde.cs.tsukuba.ac.jp, ††amagasa@cs.tsukuba.ac.jp
Abstract Large scale knowledge bases like Freebase or DBpedia, consist of a large pool of information that isused to answer questions expressed in natural language. The question answering over knowledge graphs aims at
providing the answers to natural language questions by looking up facts that are already stored in the knowledge
base. In this work, we focus on simple questions which require the extraction of a single fact from the knowledge
base to be answered. Although this might seem less challenging, recent attempts that have applied end-to-end approaches with complex neural networks have shown that it is still a challenging task especially when asked against
a large knowledge base. We propose a combination of standard neural network and non neural network techniques
for answering factoid questions over knowledge base. Our approach diﬀers from previous end-to-end approaches by
splitting the question answering problem into independent sub-problems namely; detecting entities, linking entities
to the knowledge base and classifying the question into one of the relation types in the knowledge base. Our approach yields competitive results with state-of-art approaches that apply complex neural networks on a similar dataset.
Key words Question Answering, Knowledge Base

1 Introduction

cessing tasks like opinion extraction[3], sentence classification[4] and dependency parsing[5].

Question answering (QA) often refers to building systems

This study focuses on simple factoid question answer-

that automatically answers questions expressed in naturaling (SimpleQA) based on simpleQuestions benchmark[6] inlanguage. It is one of the oldest research areas with a variety

which answering a question requires the extraction of a single

of applications in Natural Language Processing (NLP) tasks

fact from the knowledge base.

such as information retrieval and entity extraction. Recently,

Information in many knowledge bases is stored in the form

Question Answering has also been applied in developing chat-of RDF triples (subject, predicate, object) [7], for exam-

bots[1] and dialogue systems designed to simulate humanple a freebase triple (/m/02mjmr, /People/Person.place of

conversation. There are two major paradigms of question

birth, /m/025vc42) where /m/02mjmr represent the free-answering: Question Answering over free-text which aims

base MID for Barack Obama, and /m/025vc42 is the theat providing the answers to questions expressed in natural

MID for Honolulu. Therefore asking a question like ”wherelanguage without any domain restriction and On the otherwas Barack Obama born?”, getting the correct answer fromhand, Question Answering over knowledge base (KB) which

the knowledge base would require retrieving the subject en-provides the answer to the question by looking up the facts

tity /m/02mjmr in freebase that represents Barack Obama

stored in the KB. Traditionally, research in the question an-

and the predicate /People/Person.place of birth. The

swering domain used a pipeline of conventional linguistic-

retrieved entity and predicate would form a structured query

based NLP techniques, such as part-of-speech tagging, pars-represented as a pair of the subject entity and the relation

ing and conference resolution. Many of the state-of-art QA

that would be used to obtain the right answer /m/025vc42

systems like IBM Watson[2] have applied these methods.

which represents the object entity Honolulu in freebase.

However, the recent developments in deep learning has

Although several studies on Simple question answering

triggered a line of work tempting many researchers to investi-

task have increasingly applied complex neural network ar-gate more language agnostic approaches together with com-chitectures, our argument inspired by an emerging researchplex neural network architectures that have outperformed

area that aims at improving empirical rigor in the machine

traditional approaches on a variety of natural language pro-learning field by focusing on knowledge and insights as op-

posed to simply winning sculley et al[8], is of understanding
what exactly contributes to the eﬀectiveness of a particu-

Table 1 A table showing a few samples from the back-projected
entity detection train dataset

lar neural network, we avoid the complex NN architectures
and apply standard/baseline NN architectures on SimpleQuestions data set which generates results competitive tothe state of the art. We take a step further to show thatcombining NN with non neural network techniques like conditional random field (crf) achieves reasonable results. Basing on our results, we seem to agree with the argument thatstandard neural network architectures when properly tuned,
Outperform some recent complex models[9].

2 Related work
Question answering over knowledge base research has grad-Ture and Jojic[18] which argues that baseline methods whenually developed from earlier domain-specific question answer-

fully explored can equally produce competitive results. His

ing (Tang and Mooney[10]), to open-domain question an-

work formulates the question answer problem into two ma-swering based on large scale knowledge bases. For several

chine learning tasks: entity detection and relation classifica-

years research has been conducted to tackle this problem bytion which then applies simple recurrent neural network anddirectly parsing the natural language question into a struc-urges that taking advantage of the problem structure yields

tured query using semantic parsing Kwiatkowski et al[11],accurate and eﬃcient results compared to complex neuraland more recent work includes designing knowledge specificnetwork methods. We extend on this work to explore thelogical representation and grammar parsing Berant et al[12].

performance of non neural network techniques on a similar

Another line of research tackles the problem by deep learning powered similarity matching. In his work, Bordes etal[6] proposed the single-relation factoid question answering.

problem.

3 ApproachThis work introduces a new data-set SimpleQuestions with

The task of question answering over knowledge base can

108,000 questions built on Freebase and proposes a memory

be represented as follows; Let G be the knowledge base rep-

network to solve the simple question answering task. This

resenting a set of triples;data set prompted a new line of work and in the past years
several researchers have applied even more complex neural

G = {(Si, P i, Oi)}

(1)

network architectures to address this problem: Golub andwhere: Si: Subject entity, Pi: Predicate or relation, and Oi:

He[13] proposed a character-level attention-based encoder-an Object entity.

decoder model, Lukovinikov et al[14] applies a hierarchicalword-level and character-level question encoder to train a

Therefore given a simple natural language question q represented as a sequence of words,

neural network in an end-to-end manner. Dai et al[15] proposes a conditional probabilistic framework using BIGRUs
to infer the target relation first and then the target subject

q = {w1, w2, ..., wt}(2)

The task is to find a triple;

associated with the candidate relations. Yin et al[16] usedcharacter-level convolutional Neural Network for entity link-(ŝ,p̂,ô) ∈ G(3)

ing and a separate word-level convolutional Neural Networkwith attentive max-pooling that models the relationship between the predicate and question pattern more eﬀectively.
Yu et al[17] applied a residual hierarchical BILSTM that performs hierarchical matching between questions and knowledge base relations for relation prediction, the results werethen combined with the entity linking output. The above
deep learning approaches, exploit increasingly complex techniques.
In this work we build on a more close related work bysuch that ô is the intended answer to the question. We
therefore formulate this task to finding the right subject ŝ
and predicate or relation p̂ referred to in the question q thatcharacterizes a set of triples in the knowledge base G that
contain the answer ô to the question.
As mentioned above, we split the question answering task
into the following sub-tasks;
Entity detection To identify the entity in the question
we formulate this as a sequence labeling problem where eachword or token is tagged as entity or non-entity.

Each question word/token is represented with a word embedding, the input word representation is then combinedLabeld Resultwith the hidden layer representation from the previous time

0

0

1

1

0

where

was

Baracka

Obama

born

Softmax Layer

step using either BiLSTM[19] or BiGRU[20] standard RNNs
which then applies a non-linear transformation to compute

RNN (BiLSTM/GRU)
Hidden layer

the hidden layer representation at the current time step. The
final hidden representation at the current time step is then

Word
EmbeddingInput words

projected to the output dimensional space and normalized
into a probability distribution via a softmax layer. The standard RNNs used in our model, apply both LSTMs and GRUs

Figure 1 Entity detection architecture.

for calculating the hidden states of the network. Below webriefly describe the gated recurrent units (GRU) shown in
Figure 2 because they are commonly used due to their ability to process longer sequences brought about by their ad-

softmax

ditive manipulation of the state vector and explicit filtering
C<t-1>

using gates.
Given a sentence, as we read the sentence from left to right,=a<t-1>the GRU is going to have a new memory variable called thememory cell/hidden state C

<t>

C<t>
=a<t>
Γu

Ĉ<t>
tanh

σ

which provides a bit of mem-ory to remember so that when the network gets further intoX<t>the sentence it can still remember the subject of the sentence
and so at time step t the GRU will output an activation function equivalent to the memory cell at the time step.
Therefore the current memory cell/hidden state C <t> attime step t is computed by interpolating between the previous hidden state C <t−1> at previous time step and thecandidate state Ĉ <t> at the current time step.
C<t>= Γu ⊙ Ĉ

<t>+ (1 − Γu ) ⊙ C<t−1>

Figure 2 Schematic representation of GRU

Γr = σ(Wr [C <t−1> , X <t> + br ])(7)

Wr and br are parameters.
(4)

with Γu the update vector and ⊙ the element wise vector

We also apply conditional random fields (crf) to sequencelabeling[21] to compare the entity detection performancewith recurrent neural networks. The crf is a conditional se-

product. For interpolation, the update gate which deter-

quence model which represents the probability of a hidden

mines how much of the previous state is leaked into the cur-state sequence given some observations. We train the crf us-

rent state Γu is computed using the current input X <t> anding Stanford Named Entity Recognizer (NER) a tool which

the previous state C <t−1> . The update gate can decide to

can label word sequences into four classes; person, organiza-

forget the previous state altogether or copy the previous statetion, location and non-entity. We tagged the question into

and ignore the current input.

four classes, where three of the classes were tagged as entityΓu = σ(Wu [C <t−1> , X <t> ] + bu )(5)(person, organization, location) and so there were two classes
entity and non-entity.

Where Wu and bu are parameter metrics to be learned during

Entity linking The generated candidate entities are thentraining and σ the Sigmoid activation function σ(x) = 1+e1−x

linked to the actual knowledge base node. We use Freebase

applied element wise to the vector entries.

as our knowledge base where each node is represented with

The current candidate memory cell/hidden state Ĉ <t> isa machine identifier/MID. We treat this as a fuzzy string

computed based on the current input X <t> and the previous

matching problem and follow a similar method proposed in

hidden state C <t−1> .

[18]. For linking the extracted entity to the actual knowledge

Ĉ <t> = tanh(Wc [Γr ⊙ C <t−1> , X <t> ] + bc )(6)base node, we build three diﬀerent indexes using dictionaris
in python; a names index which maps all entity MID’s in thewhere Wc and bc are parameter metrics, tanh the hyperbolic

Freebase subset to their names in the Freebase names file.

tangent activation function and Γr is the reset gate which

The second index is the inverted entity index which maps all

determines the parts of the previous state ignored in compu-

n-grams of an entity for n ∈ {1, 2, 3} to the entity MID. The

tation of the candidate state and it is computed as;Table 2 A table showing a few Freebase samples

Output relation type

People/person/place_of_birthSoftmax Layer

RNN (BiLSTM/GRU)
Hidden layer

Word
EmbeddingInput words

where

was

Baracka

Obama

born

Figure 3 Relation Classification architecture.
Table 3 A table showing a few simplequestions samples in training set

final index is the reachability index that maps each entity
node in the Freebase subset to all nodes reachable by a single relation and we append the retrieved entity MID’s to thecandidate list. An early termination proposed by Ture andJojic[18] that stops searching for entities of smaller n value
after candidate entities have been found is applied, and we
calculate levenshtein distance to score entities in the candidate list and rank them in a descending order.
Relation classification The question is classified as oneof the freebase knowledge base relations.

We examine amodel similar to that of entity detection, both BiLSTMs and
BiGRUs are applied to model dependencies among words inTable 4 A table showing statistics of the freebase subsets

the question. The diﬀerence is that relation classification is
not a tagging task, we therefore base the classification decision on the output of the last hidden layer for prediction asshown in figure 2.
We also use CNNs for relation classification and following
Kim et al[22], we modify the multi-channel model describedin his paper to a single static channel instead, and apply thesame model to our task of relation classification. CNNs are

1, and the SimpleQuestions data-set[6] table 2.

not recommended for sequence modeling but since they arebase knowledge base, entities are connected by prede-likely to extract local features by sliding filters over the word

fined predicates connecting from the Subject to the ob-

embeddings we adopt them.

ject.

In free-A triple (Subject, Predicate, Object) denoted as

After generating the candidate entities, and relations in(S, P, O) describe a fact for example (Barack Obama, Peo-the previous steps, we come up with all possible (entity, re-ple/Person.place.of.birth, Honolulu) refers to the fact thatlation) pairs, and using the reachability index we check the

Barack Obama was born in Honolulu.

existence of the given combination in the knowledge base.

The SimpleQuestions data-set consists of 108,442 natural

Since entity linking and relation prediction are carried out

language simple questions with their corresponding Freebase

separately, many combinations don’t exist in the knowledge

triples (subject, predicate, object) that provides an answer

base. Those combinations that don’t exist in the knowledge

the question. We use the training, validation and test splits

base are pruned. The final prediction is achieved by com-of 75,910, 10,845 and 21,687 questions respectively as pro-bining the scores of both entity linking and their relation

vided by the data-set.

prediction.

Following the previous work we use FB2M Freebase subset
as the knowledge base which has 2 million entities and 6,701

4 Experiments and Results
4. 1 Experiments
We experiment on Freebase knowledge base[23] table

relations table 3.
We compute precision, recall and F1 measure for evaluation in entity detection and evaluate recall for top results atTable 5 Entity detection results for a given model

Table 6 Entity linking results for a given model

Table 7 Relation prediction results for a given modelk (R@k) for both entity linking and relation prediction. The
prediction is marked as correct if both entity and relation
match the ground truth in end-to-end evaluation.
We initialize the model word embeddings with a 300dimensional pre-trained vectors provided by Glove[24]. The
pre-trained word embeddings implicitly integrate word semantics inferred from large text corpus based on the
distributional semantic hypothesis[25] which states that”words with similar meanings occur in similar context”. The
pre-trained word embeddings allows to find better matches
between words in the question and subject labels or relation
URI’s. It also allows to handle unseen words during training
when it comes to training.
We implement the model in PyTorch v0.2.1 with a single CPU 3.3 GHz Intel core i5 macOS Sierra. and we use
negative log likelihood loss function and Adam[26] for optimization, with the learning rate of 0.0001 in a mini-batch
setting with batch size 32. To implement the conditional random field (crf), we use the Stanford Named Entity Tagger(NER)[27].
4. 2 Results
In this section we present our results on the SimpleQuestions task and we begin with the results on individual components:
1. Entity Detection: Table 1 shows the models’ resultson the task of entity recognition. We evaluate the precision,
recall and F1-score on the the token span level. This meansthat (a true positive span) the predicted entity token span
exactly matches the ground truth from the back-projected
dataset. The results reveal that RNN (LSTM GRU) perform better with F1-score of 92.5% for the GRU. It can also
be noticed that the crf result of 90.2% is comparable.
2. Entity linking: Table 2 shows the model comparisonof entity linking results. The CRF entity linking results accuracy is comparable to both LSTM and GRU. Although
the crf may have performed slightly lower than the LSTM
and GRU on entity detection, the bottleneck is entity linking since we see more entities in the knowledge graph withthe same label that makes it diﬃcult to identify the correct
entity (MID).
3. Relation Prediction: For the task of predicting the relation type of the question, the relation or predicate is givenin the dataset. There are 1,837 unique relation types in thedataset. We conduct a large scale classification with 1,837
possible labels to assign a relation type to the question. from
Table 7 we see that on precision, CNN out performs both
RNN’s (BiLSTM and BiGRUs). We however see that both
RNN and CNN retrieval results (R@3) are essentially similar
but RNN better at (R@3).
5.

End-to-end:

Table 4 shows end-to-end results forvarious combinations of entity detection and relation prediction on test set.

The best model combination which

achieves 74.64 accuracy is the BiLSTM for entity detection and BiGRU for relation prediction. We also compare
our results with other state-of-the-art models on the SimpleQuestions test set . Our results outperform the complex
neural network models like Bodes et al’s memory network,
Golub and He’s attention-enhanced encoder-decoder framework and Lukovinikov et al’s complex character and wordlevel encoding. Our model is however outperformed by Dai etal and Yin et al which apply a separately trained segmentation. Ture and Jojic [18] reported a much higher accuracy, we
are not able to replicate their results since they do not release
their source code due to company restrictions as mentioned

Table 8 Diﬀerent models end-to-end test accuracy compared pre-owning the top entry in a leader board. Finally even thoughvious results mentioned in the related work

deep learning has opened potential for more generic solutions, taking advantage of the problem structure yields not
only accurate but also eﬃcient results.
Acknowledgments
This research was supported by 平成 30 年度共同研究 (SKY株式会社)（CPE30034)）「データエンジニアリングの知見の応用による SKYSEA Client View のログ及び資産情報の処理の高速化・軽量化・高度化」の助成を受けたものです。

Referencesby the author when I contacted via email. Besides that,our best accuracy is less than 2 points away from the next
highest reported result in the literature. When we replace
BiLSTM with CRF for entity detection/linking the accuracy
decrease by only 1.25 this shows that non neural network
baselines can still perform well. Despite the immense contribution of neural networks to the meaningful improvementsin the state of the art on the simple questions dataset, ourresults suggest that the improvements directly attributed toneural networks are modest than previous researchers may
have led the readers to believe.
It is important to pay attention when interpreting the results in table 4 due to non-determinism associated with training neural networks that can yield diﬀerences in accuracy
Reimers and Gurevych [28]. It was also demonstrated thatfor answer selection in question answering, issues ranging
from software versions, can significantly impact the accuracy
Crane [29].

5 Conclusion
In this work we explore simple yet eﬀective approach forsimple question answering. Our baseline NNs are less complex than the previous models described in the related work.
Moving forward we are more interested in formally decomposing the simple question answering task in sub-problemsof entity detection/linking and relation prediction and solve
each separately.
In conclusion, some points to consider especially for thesimple question answering over knowledge graphs, there is
still need to adequately examine simple baselines rigorously
before rushing to sophisticated deep learning techniques. secondary our deep learning results are exciting but it is also
necessary to consider none neural network baselines sincenatural language processing existed before deep learning.
Also as pointed out by Sculley [8] it is important to remember that the ultimate goal of science is knowledge and not[1] S. Quarteroni, ”A chatbot-based Interactive Question Answering System,” Proceedings of the 11th workshop on the
Semantics and Pragmatics of Dialogue, pp. 83-90, 2007.
[2] D. A. Furrucci, ”Introduction to this is watson” IBM Journal of Research and Development, vol. 56, no. 3.4, pp. 1-1,
2012.
[3] O. Irsoy and C. Cardie, ” Opinion Mining with Deep Recurrent Nueral Networks,” Proceedings of the conferenceon Empirical Methods in Natural Language Processing(EMNLP), pp. 1746-1751, 2014.
[4] Y.Kim, ”Convolutional Neural Networks for sentence Classification,” Proceedings of the conference on Empirical
Methods in Natural Language Processing (EMNLP), pp.
720-728, 2014.
[5] D.Chen and C.Manning, ”A Fast and Accurate Dependency
Parser using Neural Networks,” Proceedings of the conference on Empirical Methods in Natural Language Processing(EMNLP), pp. 740-750, 2014.
[6] Antoine Bordes, Sumit Chopra and Jason Weston, ”Largescale simple question answering with memory networks,”(CoRR), 2015.
[7] H.Bast, F.Burle, B.Buchhold, E.Haumann, ”Easy Accessto the Freebase Dataset,” Proceedings of the 23rd International Conference on World Wide Web (WWW), pp. 95-98,
2014.
[8] D. Sculley, J. Snoek, A. Rahimi, A. Wiltschko, ”WINNER’S CURSE?, ON PACE, PROGRESS, AND EMPIRICAL RIGOR,” Proceedings of the 6th International Conference on Learning Representations, Workshop Track (ICLR)
2018.
[9] G. Melis, C. Dyer, P. Blunsom, ”On the State of the Art
Evaluation in Neural Language Models,” ICLR, 2018.
[10] Lappoon R. Tang and Raymond J. Mooney, ”Using Multiple Clause Constructors in Inductive Logic Programming for
Semantic Parsing,” Proceedings of the 12th European Conference on Machine Learning (ECML), pp. 466-477, 2001.
[11] T. Kwiatkowski, E. Choi, Y. Artzi, L. Zettlemoyer, ”Scaling
Semantic Parsers with On-the-fly Ontology Matching,” Proceedings of the Conference on Emperical Methods in Natural Language Processing (EMNLP), pp. 1545-1556, 2013.
[12] J. Berant, A. Chou, R. Frostig and P. Liang, ”Semantic
Parsing on Freebase from Question-Answer Pairs,” Proceedings of the Conference on Empirical Methodes in Natural
Language Processing (EMNLP), pp. 1533-1544, 2013.
[13] D. Golub and X. He, ”Character-Level Question Answeringwith Attention,” Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
1598-1607, 2016.
[14] D. Lukovinikov, A. Fisher, J. Lehmann, and S. Auer,”Neural Network-based Question Answering over Knowledge Graphs on Word and Character Level,” Proceedingsof the 26th International Conference on World Wide Web

(WWW), pp. 1211-1220, 2017.
[15] Z. Dai, L. Li, and W. Xu, ”CFO: Conditional Focused Neural Question Answering with Large-scale Knowledge Bases,”
Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (ACL), pp. 800-810, 2016.
[16] W. Yin, M. Yu, B. Xiang, B. Zhou, and H. Schutze, ”Simple Question Answering by Attentive Convolutional Neural
Network,” Proceedings of the 26th International Conference
on Computational Linguistics (COLING), pp. 1746-1756,
2016.
[17] M. Yu, W. Yin, K.S. Hasan, C. dos Santos, B. Xiang, andB. Zhou, ”Improved Neural Relation Detection for Knowledge Base Question Answering,” Proceedings of the Annual
Meeting of the Association for Computational Linguistics(ACL), pp. 571-581, 2017.
[18] F. Ture and O. Jojic, ”No Need to Pay Attention: Simple
Recurrent Neural Networks Work! (for Answering ”Simple” Questions)” Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
2866-2872, 2017.
[19] A. Graves and J. Schmidhuber, ”Framewise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures,” Neural Network, pp. 602-610, 2005.
[20] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F.
Bougares, H. Schwenk, Y. Bengio, ”Learning Phrase Representation using RNN Encoder–Decorder for statistical Machine Translation,” Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP),
pp.1724-1734, 2014.
[21] J. Rose Finkel, T. Grenager, and C. Manning, ”Incoporating non-local information extraction systems by Gibbs sampling,” Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 363-370,
2005.
[22] Y. Kim, ”Convolutional Neural Networks for Sentence
Classification,” Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP),
pp.1746-1751, 2014.
[23] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Taylor,”Freebase: a collectively created graph database for structuring human knowledge,” Proceedings of the 2008 ACM
SIGMOD international conference on Management of Data,
pp. 1247-1250, 2008.
[24] J. Pennington, R. Socher, and C. Manning, ”Glove: Global
Vectors for Word Representation,” Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP), pp.1532-1543, 2014.
[25] M. Sahlgren. ”The distributional hypothesis.” Italian Journal of Linguistics, pp.33-54, 2008.
[26] Diederik P. Kingma, J. Lei Ba, ”ADAM: A METHOD FOR
STOCHASTIC OPTIMIZATION,” International Conference On learning Representation (ICLR), 2015.
[27] Jenny R. Finkel, Christopher D. Manning, ”Nested namedentity recognition,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP),
pp.141-150, 2009.
[28] N. Reimers and I. Gurevych, ”Reporting Score Distributions Makes a Diﬀerence: Performance Study of LSTMnetworks for Sequence Tagging,” Proceedings of the Conference on Empirical Methodes in Natural Language Processing (EMNLP), pp. 338-348, 2017.
[29] M. Crane, ”Questionable Answers in Question Answering
Research: Reproducibility and Variability of Published Results,” Transactions of the Association for Computational
Linguistics (ACL), pp. 241-252, 2018.