Venue: NAACL 2019

# Bidirectional Attentive Memory Networks for Question Answering over Knowledge Bases

 Yu Chen

Rensselaer Polytechnic Institute

cheny39@rpi.edu &Lingfei Wu

IBM Research

wuli@us.ibm.com &Mohammed J. Zaki

Rensselaer Polytechnic Institute

zaki@cs.rpi.edu

###### Abstract

When answering natural language questions over knowledge bases (KBs), different question components and KB aspects play different roles. However, most existing embedding-based methods for knowledge base question answering (KBQA) ignore the subtle interrelationships between the question and the KB (e.g., entity types, relation paths and context). In this work, we propose to directly model the two-way flow of interactions between the questions and the KB via a novel Bidirectional Attentive Memory Network, called BAMnet. Requiring no external resources and only very few hand-crafted features, on the WebQuestions benchmark, our method significantly outperforms existing information-retrieval based methods, and remains competitive with (hand-crafted) semantic parsing based methods. Also, since we use attention mechanisms, our method offers better interpretability compared to other baselines.

## 1 Introduction

With the rapid growth in large-scale knowledge bases (KBs) such as DBPedia Auer et al. (2007) and FreeBase Google (2018), knowledge base question answering (KBQA) has drawn increasing attention over the past few years. _Given questions in natural language (NL), the goal of KBQA is to automatically find answers from the underlying KB_, which provides a more natural and intuitive way to access the vast underlying knowledge resources.

One of the most prominent challenges of KBQA is the lexical gap. For instance, the same question can be expressed in various ways in NL while a KB usually has a canonical lexicon. It is therefore non-trivial to map an NL question to a structured KB. The approaches proposed to tackle the KBQA task can be roughly categorized into two groups: semantic parsing (SP) and information retrieval (IR) approaches. SP-based approaches address the problem by constructing a semantic parser that converts NL questions into intermediate logic forms, which can be executed against a KB. Traditional semantic parsers Wong and Mooney (2007) require annotated logical forms as supervision, and are limited to narrow domains with a small number of logical predicates. Recent efforts overcome these limitations via the construction of hand-crafted rules or features Abujabal et al. (2017); Hu et al. (2018) schema matching Cai and Yates (2013), and using weak supervision from external resources Krishnamurthy and Mitchell (2012).

Unlike SP-based approaches that usually assume a pre-defined set of lexical triggers or rules, which limit their domains and scalability, IR-based approaches directly retrieve answers from the KB in light of the information conveyed in the questions. These IR-based approaches usually do not require hand-made rules and can therefore scale better to large and complex KBs. Recently, deep neural networks have been shown to produce strong results on many NLP tasks. In the field of KBQA, under the umbrella of IR-based approaches, many embedding-based methods Bordes et al. (2014); Hao et al. (2017) have been proposed and have shown promising results. These methods adopt various ways to encode questions and KB subgraphs into a common embedding space and directly match them in that space, and can be typically trained in an end-to-end manner.

Compared to existing embedding-based methods that encode questions and KB subgraphs independently, we introduce a novel Bidirectional **A**tentive **M**emory **n**etwork, called **BAMnet** that captures the mutual interactions between questions and the underlying KB, which is stored in a content-addressable memory. We assume that the world knowledge (i.e., the KB) is helpful for better understanding the questions. Similarly, the questions

themselves can help us focus on important KB aspects. To this end, we design a _two-layered bidirectional attention network_. The _primary attention network_ is intended to focus on important parts of a question in light of the KB and important KB aspects in light of the question. Built on top of that, the _secondary attention network_ is intended to enhance the question and KB representations by further exploiting the two-way attention. Through this idea of _hierarchical two-way attention_, we are able to distill the information that is the most relevant to answering the questions on both sides of the question and KB.

We highlight the contributions of this paper as follows: 1) we propose a novel bidirectional attentive memory network for the task of KBQA which is intended to directly model the two-way interactions between questions and the KB; 2) by design, our method offers good interpretability thanks to the attention mechanisms; 3) on the WebQuestions benchmark, our method significantly outperforms previous information-retrieval based methods while remaining competitive with (handcrafted) semantic parsing based methods.

## 2 Related work

Two broad classes of SP-based and IR-based approaches have been proposed for KBQA. The former attempts to convert NL questions to logic forms. Recent work focused on approaches based on weak supervision from either external resources Krishnamurthy and Mitchell (2012); Berant et al. (2013); Yao and Van Durme (2014); Hu et al. (2018); Yih et al. (2015); Yavuz et al. (2016), schema matching Cai and Yates (2013), or using hand-crafted rules and features Unger et al. (2012); Berant et al. (2013); Berant and Liang (2015); Reddy et al. (2016); Bao et al. (2016); Abujabal et al. (2017); Hu et al. (2018); Bast and Haussmann (2015); Yih et al. (2015). A thread of research has been explored to generate semantic query graphs from NL questions such as using coarse alignment between phrases and predicates Berant et al. (2013), searching partial logical forms via an agenda-based strategy Berant and Liang (2015), pushing down the disambiguation step into the query evaluation stage Hu et al. (2018), or exploiting rich syntactic information in NL questions Xu et al. (2018a, b). Notably, another thread of SP-based approaches try to exploit IR-based techniques Yao and Van Durme (2014); Bast and Haussmann (2015); Yang et al. (2014); Yih et al. (2015); Bao et al. (2016); Yavuz et al. (2016); Liang et al. (2016) by computing the similarity of two sequences as features, leveraging a neural network-based answer type prediction model, or training end-to-end neural symbolic machine via REINFORCE Williams (1992). However, most SP-based approaches more or less rely on hand-crafted rules or features, which limits their scalability and transferability.

The other line of work (the IR-based) has focused on mapping answers and questions into the same embedding space, where one could query any KB independent of its schema without requiring any grammar or lexicon. Bordes et al. (2014b) were the first to apply an embedding-based approach for KBQA. Later, Bordes et al. (2014a) proposed the idea of subgraph embedding, which encodes more information (e.g., answer path and context) about the candidate answer. In follow-up work Bordes et al. (2015); Jain (2016), memory networks Weston et al. (2014) were used to store candidates, and could be accessed iteratively to mimic multi-hop reasoning. Unlike the above methods that mainly use a bag-of-words (BOW) representation to encode questions and KB resources, Dong et al. (2015); Hao et al. (2017) apply more advanced network modules (e.g., CNNs and LSTMs) to encode questions. Hybrid methods have also been proposed Feng et al. (2016); Xu et al. (2016); Das et al. (2017), which achieve improved results by leveraging additional knowledge sources such as free text. While most embedding-based approaches encode questions and answers independently, Hao et al. (2017) proposed a cross-attention mechanism to encode questions according to various candidate answer aspects. Differently, in this work, our method goes one step further by modeling the bidirectional interactions between questions and a KB.

The idea of bidirectional attention proposed in this work is similar to those applied in machine reading comprehension Wang and Jiang (2016); Seo et al. (2016); Xiong et al. (2016). However, these previous works focus on capturing the interactions between two bodies of text, in this work, we focus on modeling the interactions between one body of text and a KB.

## 3 A modular bidirectional attentive memory network for KBQA

Given an NL question, the goal is to fetch answers from the underlying KB. Our proposed BAMnet 

model consists of four components which are the _input module_, _memory module_, _reasoning module_ and _answer module_, as shown in Fig. 1.

### Input module

An input NL question is represented as a sequence of word embeddings () by applying a word embedding layer. We then use a bidirectional LSTM Hochreiter and Schmidhuber [1997] to encode the question as (in ) which is the sequence of hidden states (i.e., the concatenation of forward and backward hidden states) generated by the BiLSTM.

### Memory module

**Candidate generation** Even though all the entities from the KB could in principle be candidate answers, this is computationally expensive and unnecessary in practice. We only consider those entities which are "close" to the main topic entity of a question. An answer is the text description (e.g., a name) of an entity node. For example, _Ohio_ is the topic entity of the question "Who was the secretary of state of Ohio in 2011?" (see Fig. 2). After getting the topic entity, we collect all the entities connected to it within hops as candidate answers, which we denote as .

**KB representation** For each candidate answer from the KB, we encode three types of information: answer type, path and context.

Answer typeEntity type information is an important clue in ranking answers. For example, if a question uses the interrogative word _where_, then candidate answers with types relevant to the concept of location are more likely to be correct. We use a BiLSTM to encode its text description to get a -dimensional vector (i.e., the concatenation of last forward and backward hidden states).

Answer pathWe define an answer path as a sequence of relations from a candidate answer to a topic entity. For example, for the _Ohio_ question (see Fig. 2), the answer path of _Jon A. Husted_ can be either represented as a sequence of relation ids [_office_holder_, _governing_officials_] or the text description [_office_, _holder_, _governing_, _officials_]. We thus encode an answer path as via a BiLSTM, and as by computing the average of its relation embeddings via a relation embedding layer.

Answer contextThe answer context is defined as the surrounding entities (e.g., sibling nodes) of a candidate which can help answer questions with constraints. For example, in Fig. 2, the answer context of _Jon A. Husted_ includes the government position title _secretary of state_ and starting date _2011-01-09_. However, for simple questions without constraints, the answer context is unenc

Figure 1: Overall architecture of the BAMnet model.

Figure 2: A working example from Freebase. Relations in Freebase have hierarchies where high-level ones provide too broad or even noisy information about the relation. Thus, we choose to use the lowest level one.

 

essary and can potentially incorporate noise. We tackle this issue with two strategies: 1) we use a novel _importance module_ (explained later) to focus on important answer aspects, and 2) we only consider those context nodes that have overlap with the question. Specifically, for each context node (i.e., a sequence of words) of a candidate, we first compute the longest common subsequence between it and the question, we then encode it via a BiLSTM only if we get a non-stopwords substring. Finally, the answer context of a candidate answer will be encoded as the average of all context node representations, which we denote as \({\bf H}^{c}_{i}\).

**Key-value memory module** In our model, we use a key-value memory network Miller et al. (2016) to store candidate answers. Unlike a basic memory network Weston et al. (2014), its addressing stage is based on the key memory while the reading stage uses the value memory, which gives greater flexibility to encode prior knowledge via functionality separation. Thus, after encoding the answer type, path and context, we apply linear projections on them as follows:

\[\begin{split}&{\bf M}^{k_{t}}_{i}=f^{k}_{t}({\bf H}^{t_{1}}_{ i})\\ &{\bf M}^{v_{p}}_{i}=f^{v}_{b}(|{\bf H}^{v_{1}}_{i};{\bf H}^{v_{2}}_{i}|)\quad{\bf M }^{v_{p}}_{i}=f^{v}_{b}(|{\bf H}^{v_{1}}_{i};{\bf H}^{v_{2}}_{i}|)\\ &{\bf M}^{k_{c}}_{i}=f^{k}_{c}({\bf H}^{c}_{i})\quad{\bf M}^{v_{ c}}_{i}=f^{c}_{c}({\bf H}^{c}_{i})\end{split} \tag{1}\]

where \({\bf M}^{k_{t}}_{i}\) and \({\bf M}^{v_{t}}_{i}\) are \(d\)-dimensional key and value representations of answer type \(A^{t}_{i}\), respectively. Similarly, we have key and value representations for answer path and answer context. We denote \({\bf M}\) as a key-value memory whose row \({\bf M}_{i}=\{{\bf M}^{k}_{i},{\bf M}^{v}_{i}\}\) (both in \(\mathbb{R}^{d\times 3}\)), where \({\bf M}^{k}_{i}=[{\bf M}^{k_{t}}_{i};{\bf M}^{k_{p}}_{i};{\bf M}^{k_{c}}_{i}]\) comprises the keys, and \({\bf M}^{v}_{i}=[{\bf M}^{v_{i}}_{i};{\bf M}^{v_{i}}_{i};{\bf M}^{v_{i}}_{i}]\) comprises the values. Here \([,]\) and \([;]\) denote row-wise and column-wise concatenations, respectively.

### Reasoning module

The reasoning module consists of a generalization module, and our novel _two-layered bidirectional attention network_ which aims at capturing the two-way interactions between questions and the KB. The _primary attention network_ contains the KB-aware attention module which focuses on the important parts of a question in light of the KB, and the importance module which focuses on the important KB aspects in light of the question. The _secondary attention network_ (_enhancing module_ in Fig. 1) is intended to enhance the question and KB vectors by further exploiting the two-way attention.

#### KB-aware attention module

Not all words in a question are created equal. We use a KB-aware attention mechanism to focus on important components of a question, as shown in Fig. 3. Specifically, we first apply self-attention (SelfAtt) over all question word vectors \({\bf H}^{Q}\) to get a \(d\)-dimensional question vector \({\bf q}\) as follows

\[\begin{split}&{\bf q}=\text{BiLSTM}([{\bf H}^{Q}{\bf A}^{QQ^{T}},{ \bf H}^{Q}])\\ &{\bf A}^{QQ}=\text{softmax}(({\bf H}^{Q})^{T}{\bf H}^{Q})\end{split} \tag{2}\]

where softmax is applied over the last dimension of an input tensor by default. Using question summary \({\bf q}\), we apply another attention (AddAtt) over the memory to obtain answer type \({\bf m}_{t}\), path \({\bf m}_{p}\) and context summary \({\bf m}_{c}\):

\[\begin{split}&{\bf m}_{x}=\sum_{i=1}^{|A|}a_{i}^{x}\cdot{\bf M}^{ v_{x}}_{i}\\ &{\bf a}^{x}=\text{Att}_{\text{add}}({\bf q},{\bf M}^{k_{x}}) \end{split} \tag{3}\]

where \(x\in\{t,p,c\}\), and \(\text{Att}_{\text{add}}({\bf x},{\bf y})=\text{softmax}(\text{tanh}([{\bf x}^{ T},{\bf y}]{\bf W_{1}}){\bf W_{2}})\), with \({\bf W_{1}}\in\mathbb{R}^{2d\times d}\) and \({\bf W_{2}}\in\mathbb{R}^{d\times 1}\) being trainable weights.

So far, we have obtained the KB summary \({\bf m}=[{\bf m}_{t};{\bf m}_{p};{\bf m}_{c}]\) in light of the question. We proceed to compute the question-to-KB attention between question word \(q_{i}\) and KB aspects as formulated by \({\bf A}^{Qm}={\bf H}^{Q^{T}}{\bf m}\). By applying max pooling over the last dimension (i.e., the KB aspect dimension) of \({\bf A}^{Qm}\), that is, \({\bf a}^{Q}_{i}=\max_{j}{\bf A}^{Qm}_{ij}\), we select the strongest connection between \(q_{i}\) and the KB. The idea behind it is that each word in a question serves a specific purpose (i.e., indicating answer type, path or context), and max pooling can help find out that purpose. We then apply a soft-max over the resulting vector to obtain \(\tilde{\bf a}^{Q}\) which is a _KB-aware question attention vector_ since it indicates the importance of \(q_{i}\) in light of the KB.

Figure 3: KB-aware attention module. CAT: concatenation, SelfAtt: self-attention, AddAtt: additive attention.

 



**Importance module** The importance module focuses on important KB aspects as measured by their relevance to the questions. We start by computing a attention tensor which indicates the strength of connection between each pair . Then, we take the max of the question word dimension of and normalize it to get an attention matrix , which indicates the importance of each answer aspect for each candidate answer. After that, we proceed to compute question-aware memory representations . Thus, we have:

(4)

**Enhancing module** We further enhance the question and KB representations by exploiting two-way attention. We compute the KB-enhanced question representation which incorporates the relevant KB information by applying max pooling over the last dimension (i.e., the answer aspect dimension) of , that is, and then normalizing it to get a question-to-KB attention matrix from which we compute the question-aware KB summary and incorporate it into the question representation Finally, we obtain a dimensional KB-enhanced question representation .

Similarly, we compute a question-enhanced KB representation which incorporates the relevant question information:

(5)

**Generalization module** We add a one-hop attention process before answering. We use the question representation to query over the key memory via an attention mechanism, and fetch the most relevant information from the value memory, which is then used to update the question vector using a GRU Cho et al. (2014). Finally, we apply a residual layer He et al. (2016) (i.e., and batch normalization (BN) Ioffe and Szegedy (2015), which help the model performance in practice. Thus, we have

(6)

### Answer module

Given the representation of question which is and the representation of candidate answers which is , we compute the matching score between every pair as . The candidate answers are then ranked by their scores.

### Training and testing

**Training** Intermediate modules such as the _enhancing module_ generate "premature" representations of questions (e.g., and candidate answers (e.g., ). Even though these intermediate representations are not optimal for answer prediction, we can still use them along with the final representations to jointly train the model, which we find helps the training probably by providing more supervision since we are directly forcing intermediate representations to be helpful for prediction. Moreover, we directly match interrogative words to KB answer types. A question is represented by a 16-dimensional interrogative word (we use "which", "what", "who", "whow", "where", "when", "how", "why" and "whether") embedding with a candidate answer is represented by entity type embedding with the same size. We then compute the matching score between them. Although we only have weak labels (e.g., incorrect answers do not necessarily imply incorrect types) for the type matching task, and there are no shared representations between two tasks, we find in practice this strategy helps the training process as shown in Section 4.4.

_Loss Function:_ In the training phase, we force positive candidates to have higher scores than negative candidates by using a triplet-based loss function:

(7)

where and is a hinge loss

function, and \(A^{+}\) and \(A^{-}\) denote the positive (i.e., correct) and negative (i.e., incorrect) answer sets, respectively. Note that at training time, the candidate answers are extracted from the KB subgraph of the gold-standard topic entity, with the memory size set to \(N_{max}\). We adopt the following sampling strategy which works well in practice: if \(N_{max}\) is larger than the number of positive answers \(|A^{+}|\), we keep all the positive answers and randomly select negative answers to fill up the memory; otherwise, we randomly select \(\min(N_{max}/2,|A^{-}|)\) negative answers and fill up the remaining memory with random positive answers.

**Testing** At testing time, we need to first find the topic entity. We do this by using the top result returned by a separately trained topic entity predictor (we also compare with the result returned by the Freebase Search API). Then, the _answer module_ returns the candidate answer with the highest scores as predicted answers. Since there can be multiple answers to a given question, the candidates whose scores are close to the highest score within a certain margin, \(\theta\), are regarded as good answers as well. Therefore, we formulate the inference process as follows:

\[\hat{A}=\{\hat{a}~{}~{}~{}|~{}\hat{a}\in A~{}\&~{}\underset{a\neq\hat{c}}{ \mathrm{max}}\{S(\hat{\mathbf{q}},\overline{\mathbf{M}}_{a\prime}^{k})\}-S( \hat{\mathbf{q}},\overline{\mathbf{M}}_{a}^{k})<\theta\} \tag{8}\]

where \(\underset{a\neq\hat{c}}{\mathrm{max}}\{S(\hat{\mathbf{q}},\overline{\mathbf{ M}}_{a\prime}^{k})\}\) is the score of the best matched answer and \(A\) is the predicted answer set. Note that \(\theta\) is a hyper-parameter which controls the degree of tolerance. Decreasing the value of \(\theta\) makes the model become stricter when predicting answers.

### Topic entity prediction

Given a question \(Q\), the goal of a topic entity predictor is to find the best topic entity \(\hat{c}\) from the candidate set \(\{C_{i}\}_{i=1}^{|C|}\) returned by external topic entity linking tools (we use the Freebase Search API and S-MART Yang and Chang (2016) in our experiments). We use a convolutional network (CNN) to encode \(Q\) into a \(d\)-dimensional vector \(\mathbf{e}\). For candidate topic entity \(C_{i}\), we encode three types of KB aspects, namely, the entity name, entity type and surrounding relations where both entity name and type are represented as a sequence of words while surrounding relations are represented as a bag of sequences of words. Specifically, we use three CNNs to encode them into three \(d\)-dimensional vectors, namely, \(\mathbf{C}_{i}^{n}\), \(\mathbf{C}_{i}^{t}\) and \(\mathbf{C}_{i}^{r_{1}}\). Note that for surrounding relations, we first encode each of the relations and then compute their average. Additionally, we compute an average of the relation embeddings via a relation embedding layer which we denote as \(\mathbf{C}_{i}^{r_{2}}\). We then apply linear projections on the above vectors as follows:

\[\begin{split}\mathbf{P}_{i}^{k}&=f^{k}([\mathbf{ C}_{i}^{n};\mathbf{C}_{i}^{t};\mathbf{C}_{i}^{r_{1}};\mathbf{C}_{i}^{r_{2}}])\\ \mathbf{P}_{i}^{v}&=f^{v}([\mathbf{C}_{i}^{n};\mathbf{ C}_{i}^{t};\mathbf{C}_{i}^{r_{1}};\mathbf{C}_{i}^{r_{2}}])\end{split} \tag{9}\]

where \(\mathbf{P}_{i}^{k}\) and \(\mathbf{P}_{i}^{v}\) are \(d\)-dimensional key and value representations of candidate \(C_{i}\), respectively. Furthermore, we compute the updated question vector \(\hat{\mathbf{e}}\) using the generalization module mentioned earlier. Next, we use a dot product to compute the similarity score between \(Q\) and \(C_{i}\). A triplet-based loss function is used as formulated by \(o=g(\mathbf{e},\mathbf{P}_{i}^{k})+g(\hat{\mathbf{e}},\mathbf{P}_{i}^{k})\) where \(g(.)\) is the aforementioned hinge loss function. When training the predictor, along with the candidates returned from external entity linking tools, we do negative sampling (using string matching) to get more supervision. In the testing phase, the candidate with the highest score is returned as the best topic entity and no negative sampling is applied.

## 4 Experiments

This section provides an extensive evaluation of our proposed BAMnet model against state-of-the-art KBQA methods. The implementation of BAMnet is available at [https://github.com/hugochan/BAMnet](https://github.com/hugochan/BAMnet).

### Data and metrics

We use the Freebase KB and the WebQuestions dataset, described below:

_Freebase_ This is a large-scale KB Google (2018) that consists of general facts organized as subject-property-object triples. It has 41M non-numeric entities, 19K properties, and 596M assertions.

_WebQuestions_ This dataset Berant et al. (2013) (nlp.stanford.edu/software/sempre) contains 3,778 training examples and 2,032 test examples. We further split the training instances into a training set and development set via a 80%/20% split. Approximately 85% of questions can be directly answered via a single FreeBase predicate. Also, each question can have multiple answers. In our experiments, we use a development version of the dataset Baudis and Pichl (2016), which additionally provides (potentially noisy) entity mentions for each question 

Following (Berant et al., 2013), macro F1 scores (i.e., the average of F1 scores over all questions) are reported on the WebQuestions test set.

### Model settings

When constructing the vocabularies of words, entity types or relation types, we only consider those questions and their corresponding KB subgraphs appearing in the training and validation sets. The vocabulary size of words is \(V=100,797\). There are 1,712 entity types and 4,996 relation types in the KB subgraphs. Notably, in FreeBase, one entity might have multiple entity types. We only use the first one available, which is typically the most concrete one. For those non-entity nodes which are boolean values or numbers, we use "bool" or "num" as their types, respectively.

We also adopt a **query delexicalization** strategy where for each question, the topic entity mention as well as constraint entity mentions (i.e., those belonging to "date", "ordinal" or "number") are replaced with their types. When encoding KB context, if the overlap belongs to the above types, we also do this delexicalization, which will guarantee it matches up with the delexicalized question well in the embedding space.

Given a topic entity, we extract its 2-hop subgraph (i.e., \(h=2\)) to collect candidate answers, which is sufficient for _WebQuestions_. At training time, the memory size is limited to \(N_{max}=96\) candidate answers (for the sake of efficiency). If there are more potential candidates, we do random sampling as mentioned earlier. We initialize word embeddings with pre-trained GloVe vectors (Pennington et al., 2014) with word embedding size \(d_{v}=300\). The relation embedding size \(d_{p}\), entity type embedding size \(d_{t}\) and hidden size \(d\) are set as 128, 16 and 128, respectively. The dropout rates on the word embedding layer, question encoder side and the answer encoder side are 0.3, 0.3 and 0.2, respectively. The batch size is set as 32, and answer module threshold \(\theta=0.7\). As for the topic entity prediction, we use the same hyperparameters. For each question, there are 15 candidates after negative sampling in the training time. When encoding a question, we use a CNN with filter sizes 2 and 3. A linear projection is applied to merge features extracted with different filters. When encoding a candidate aspect, we use a CNN with filter size 3. Linear activation and max-pooling are used together with CNNs. In the training process, we use the Adam optimizer (Kingma and Ba, 2014) to train the model. The initial learning rate is set as 0.001 which is reduced by a factor of 10 if no improvement is observed on the validation set in 3 consecutive epochs. The training procedure stops if no improvement is observed on the validation set in 10 consecutive epochs. The hyper-parameters are tuned on the development set.

### Performance comparison

As shown in Table 1, our method can achieve an F1 score of 0.557 when the gold topic entity is known, which gives an upper bound of our model performance. When the gold topic entity is unknown, we report the results using: 1) the Freebase Search API, which achieves a recall@1 score of 0.857 on the test set for topic entity linking, and 2) the topic entity predictor, which achieves a recall@1 score of 0.898 for entity retrieval.

As for the performance of BAMnet on WebQuestions, it achieves an F1 score of 0.518 using the topic entity predictor, which is significantly better than the F1 score of 0.497 using the Freebase Search API. We can observe that BAMnet significantly outperforms previous state-of-the-art IR-based methods, which conclusively demonstrates the effectiveness of modeling bidirectional interactions between questions and the KB.

It is important to note that unlike the state-of-the-art SP-based methods, BAMnet relies on no external resources and very few hand-crafted features, but still remains competitive with those approaches. Based on careful hand-drafted rules, some SP-based methods (Bao et al., 2016; Yih et al., 2015) can better model questions with constraints and aggregations. For example, (Yih et al., 2015) applies many manually designed rules and features to improve performance on questions with constraints and aggregations, and (Bao et al., 2016) directly models temporal (e.g., "after 2000"), ordinal (e.g., "first") and aggregation constraints (e.g., "how many") by adding detected constraint nodes to query graphs. In contrast, our method is end-to-end, with very few hand-crafted rules.

Additionally, (Yavuz et al., 2016; Bao et al., 2016) train their models on external Q&A datasets to get extra supervision. For a fairer comparison, we only show their results without training on external Q&A datasets. Similarly, for hybrid systems (Feng et al., 2016; Xu et al., 2016), we only report results without using Wikipedia free text. It is in 

teresting to note that both Yih et al. (2015) and Bao et al. (2016) also use the ClueWeb dataset for learning more accurate semantics. The F1 score of Yih et al. (2015) drops from 0.525 to 0.509 if ClueWeb information is removed. To summarize, BAMnet achieves state-of-the-art performance of 0.518 without recourse to any external resources and relies only on very few hand-crafted features. If we assume gold-topic entities are given then BAMnet achieves an F1 of 0.557.

### Ablation study

We now discuss the performance impact of the different modules and strategies in BAMnet. Note that gold topic entity is assumed to be known when we do this ablation study, because the error introduced by topic entity prediction might reduce the real performance impact of a module or strategy. As shown in Table 2, significant performance drops were observed after turning off some key attention modules, which confirms that the real power of our method comes from the idea of hierarchical two-way attention. As we can see, when turning off the _two-layered bidirectional attention network_, the model performance drops from 0.557 to 0.534. Among all submodules in the attention network, the _importance module_ is the most significant since the F1 score drops to 0.540 without it, thereby confirming the effectiveness of modeling the query-to-KB attention flow. On the flip side, the importance of modeling the KB-to-query attention flow is confirmed by the fact that replacing the _KB-aware attention module_ with self-attention significantly degrades the performance. Besides, the secondary attention layer, the _enhancing module_, also contributes to the overall model performance. Finally, we find that the topic entity delexicalization strategy has a big influence on the model performance while the constraint delexicalization strategy only marginally boosts the performance.

 

and KB aspects, by visualizing the attention matrix \({\bf A}^{QM}\) produced by the _reasoning module_. Fig. 4 shows the attention heatmap generated for a test question "who did location surrender to in __number___" (where "location" and "____" are entity types which replace the topic entity mention "France" and the constraint entity mention "ww2", respectively in the original question). As we can see, the attention network successfully detects the interactions between "who" and answer type, "surrender to" and answer path, and focuses more on those words when encoding the question.

To further examine the importance of the two-way flow of interactions, in Table 3, we show the predicted answers of BAMnet with and without the _two-layered bidirectional attention network_ on samples questions from the WebQuestions test set. We divide the questions into three categories based on which kind of KB aspect is the most crucial for answering them. As we can see, compared to the simplified version which is not equipped with bidirectional attention, our model is more capable of answering all the three types of questions.

### Error analysis

To better examine the limitations of our approach, we randomly sampled 100 questions on which our method performed poorly (i.e., with per-question F1 score less than 0.6), and categorized the errors. We found that around 33% of errors are due to label issues of gold answers and are not real mistakes. This includes incomplete and erroneous labels, and also alternative correct answers. Constraints are another source of errors (11%), with temporal constraints accounting for most. Some questions have implicit temporal (e.g., tense) constraints which our method does not model. A third source of error is what we term type errors (13%), for which our method generates more answers than needed because of poorly utilizing answer type information. Lexical gap is another source of errors (5%). Finally, other sources of errors (38%) include topic entity prediction error, question ambiguity, incomplete answers and other miscellaneous errors.

## 5 Conclusions and future work

We introduced a novel and effective bidirectional attentive memory network for the purpose of KBQA. To our best knowledge, we are the first to model the mutual interactions between questions and a KB, which allows us to distill the information that is the most relevant to answering the questions on both sides of the question and KB. Experimental results show that our method significantly outperforms previous IR-based methods while remaining competitive with hand-crafted SP-based methods. Both ablation study and interpretability analysis verify the effectiveness of the idea of modeling mutual interactions. In addition, our error analysis shows that our method actually performs better than what the evaluation metrics indicate.

In the future, we would like to explore effective ways of modeling more complex types of constraints (e.g., ordinal, comparison and aggregation).

#### Acknowledgments

This work is supported by IBM Research AI through the IBM AI Horizons Network. We thank the anonymous reviewers for their constructive suggestions.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline KB & Questions & BAMnet w/o BiAttn. & BAMnet & Gold Answers \\ \hline  & What degrees did Obama & Harvard Law School, Columbia University, & Bachelor of Arts, & Juris Doctor, \\ Answer & get in college? & Occidental College & Political Science & Bachelor of Arts \\ Type & What music period did & Austrian Empire, Germany, Bonn & Classical music, & Opera, \\  & Beethoven live in? & \multirow{2}{*}{Australia} & \multirow{2}{*}{Queen Victoria} & \multirow{2}{*}{Question Victoria} \\ \cline{1-1} \cline{7-6}  & Where did Queensland get its name from? & & & \\ \cline{1-1} \cline{7-6}  & Where does Delaware & \multirow{2}{*}{Delaware Bay} & West Branch & West Branch \\ \cline{1-1} \cline{7-6}  & river start? & & & Delaware River, \\ \hline  & What are the major cities in Ukraine? & Kiev, Olyka, & \multirow{2}{*}{Kiev} & \multirow{2}{*}{Kiev} \\ \cline{1-1} \cline{7-6}  & Who is running for vice president & \multirow{2}{*}{David Petraeus} & \multirow{2}{*}{Joe Biden} & \multirow{2}{*}{Joe Biden} \\ \cline{1-1} \cline{7-6}  & with Barack Obama 2012? & & & \\ \hline \end{tabular}
\end{table}
Table 3: Predicted answers of BAMnet w/ and w/o bidirectional attention on the WebQuestions test set.

 



## References

* Abujabal et al. (2017) Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017. Automated template generation for question answering over knowledge graphs. In _WWW_, pages 1191-1200.
* Auer et al. (2007) Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In _The semantic web_, pages 722-735. Springer.
* Bao et al. (2016) Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In _COLING_, pages 2503-2514.
* Bast and Haussmann (2015) Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In _CIKM_, pages 1431-1440. ACM.
* Baudis and Pichl (2016) Petr Baudis and Jan Pichl. 2016. Dataset factoid webquestions. [https://github.com/brmson/dataset-factoid-webquestions](https://github.com/brmson/dataset-factoid-webquestions).
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _EMNLP_, pages 1533-1544.
* Berant and Liang (2015) Jonathan Berant and Percy Liang. 2015. Imitation learning of agenda-based semantic parsers. _TACL_, 3:545-558.
* Bordes et al. (2014a) Antoine Bordes, Sumit Chopra, and Jason Weston. 2014a. Question answering with subgraph embeddings. In _EMNLP_, pages 615-620.
* Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. _arXiv preprint arXiv:1506.02075_.
* Bordes et al. (2014b) Antoine Bordes, Jason Weston, and Nicolas Usunier. 2014b. Open question answering with weakly supervised embedding models. In _ECML-PKDD_, pages 165-180. Springer.
* Cai and Yates (2013) Qingqing Cai and Alexander Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In _ACL_, pages 423-433.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In _EMNLP_, pages 1724-1734.
* Das et al. (2017) Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answering on knowledge bases and text using universal schema and memory networks. _arXiv preprint arXiv:1704.08384_.
* Dong et al. (2015) Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multicolumn convolutional neural networks. In _ACL-IJCNLP_, volume 1, pages 260-269.
* Feng et al. (2016) Yansong Feng, Songfang Huang, Dongyan Zhao, et al. 2016. Hybrid question answering over knowledge base and free text. In _COLING_, pages 2397-2407.
* Google (2018) Google. 2018. Freebase data dumps. [https://developers.google.com/freebase](https://developers.google.com/freebase).
* Hao et al. (2017) Yanchao Hao, Yuanze Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In _ACL_, volume 1, pages 221-231.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _CVPR_, pages 770-778.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hu et al. (2018) Sen Hu, Lei Zou, Jeffrey Xu Yu, Haixun Wang, and Dongyan Zhao. 2018. Answering natural language questions by subgraph matching over knowledge graphs. _TKDE_, 30(5):824-837.
* Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. _arXiv preprint arXiv:1502.03167_.
* Jain (2016) Sarthak Jain. 2016. Question answering over knowledge base using factual memory networks. In _NAACL-HLT_, pages 109-115.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Krishnamurthy and Mitchell (2012) Jayant Krishnamurthy and Tom M Mitchell. 2012. Weakly supervised training of semantic parsers. In _EMNLP-CoNLL_, pages 754-765.
* Liang et al. (2016) Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. _arXiv preprint arXiv:1611.00020_.
* Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. _arXiv preprint arXiv:1606.03126_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, pages 1532-1543.
* Reddy et al. (2016) Siva Reddy, Oscar Tackstrom, Michael Collins, Tomas Kwiatkowski, Dipanjan Das, Mark Steedman, and Mirella Lapata. 2016. Transforming dependency structures to logical forms for semantic parsing. _TACL_, 4:127-140

Siva Reddy, Oscar Tackstrom, Slav Petrov, Mark Steedman, and Mirella Lapata. 2017. Universal semantic parsing. In _EMNLP_, pages 89-101.
* Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. _arXiv preprint arXiv:1611.01603_.
* Unger et al. (2012) Christina Unger, Lorenz Buhmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-based question answering over rdf data. In _WWW_, pages 639-648. ACM.
* Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. _arXiv preprint arXiv:1608.07905_.
* Wang et al. (2014) Zhenghao Wang, Shengquan Yan, Huaming Wang, and Xuedong Huang. 2014. An overview of microsoft deep qa system on stanford webquestions benchmark. Technical Report MSR-TR-2014-121, Microsoft Research.
* Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. _arXiv preprint arXiv:1410.3916_.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8(3-4):229-256.
* Wong and Mooney (2007) Yuk Wah Wong and Raymond Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In _ACL_, pages 960-967.
* Xiong et al. (2016) Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. _arXiv preprint arXiv:1611.01604_.
* Xu et al. (2016) Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence. _arXiv preprint arXiv:1603.00957_.
* Xu et al. (2018a) Kun Xu, Lingfei Wu, Zhiguo Wang, and Vadim Sheinin. 2018a. Graph2seq: Graph to sequence learning with attention-based neural networks. _arXiv preprint arXiv:1804.00823_.
* Xu et al. (2018b) Kun Xu, Lingfei Wu, Zhiguo Wang, Mo Yu, Liwei Chen, and Vadim Sheinin. 2018b. Exploiting rich syntactic information for semantic parsing with graph-to-sequence model. _arXiv preprint arXiv:1808.07624_.
* Yang et al. (2014) Min-Chul Yang, Nan Duan, Ming Zhou, and HaeChang Rim. 2014. Joint relational embeddings for knowledge-based question answering. In _EMNLP_, pages 645-650.
* Yang and Chang (2016) Yi Yang and Ming-Wei Chang. 2016. S-mart: Novel tree-based structured learning algorithms applied to tweet entity linking. _arXiv preprint arXiv:1609.08075_.
* Yao and Van Durme (2014) Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In _ACL_, volume 1, pages 956-966.
* Yavuz et al. (2016) Semih Yavuz, Izzeddin Gur, Yu Su, Mudhakar Srivarsa, and Xifeng Yan. 2016. Improving semantic parsing via answer type inference. In _EMNLP_, pages 149-159.
* Yih et al. (2015) Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _ACL_, pages 1321-1331.