Venue: ACL 2017

# Improved Neural Relation Detection for Knowledge Base Question Answering

 Mo Yu\({}^{\dagger}\) Wenpeng Yin\({}^{*}\) Kazi Saidul Hasan\({}^{\ddagger}\) Cicero dos Santos\({}^{\dagger}\)

Bing Xiang\({}^{\ddagger}\) Bowen Zhou\({}^{\dagger}\)

\({}^{\dagger}\)AI Foundations, IBM Research, USA \({}^{*}\)Center for Information and Language Processing, LMU Munich

\({}^{\ddagger}\)IBM Watson, USA {yum,kshasan,cicerons,bingxia,zhou}@us.ibm.com, wenpeng@cis.lmu.de

###### Abstract

Relation detection is a core component of many NLP applications including Knowledge Base Question Answering (KBQA). In this paper, we propose a hierarchical recurrent neural network enhanced by residual learning which detects KB relations given an input question. Our method uses deep residual bidirectional LSTMs to compare questions and relation names via different levels of abstraction. Additionally, we propose a simple KBQA system that integrates entity linking and our proposed relation detector to make the two components enhance each other. Our experimental results show that our approach not only achieves outstanding relation detection performance, but more importantly, it helps our KBQA system achieve state-of-the-art accuracy for both single-relation (SimpleQuestions) and multi-relation (WebQSP) QA benchmarks.

## 1 Introduction

Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples [3, 20, 4, 5, 6]. For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB. Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single _<head-entity, relation, tail-entity>_ KB tuple [7, 8, 9]; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question. The KBQA system in the figure performs two key tasks: (1) _entity linking_, which links \(n\)-grams in questions to KB entities, and (2) _relation detection_, which identifies the KB relation(s) a question refers to.

The main focus of this work is to improve the _relation detection_ subtask and further explore how it can contribute to the KBQA system. Although general relation detection1 methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration. As a result, there is a significant gap between general relation detection studies and KB-specific relation detection. First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100. In contrast, in KBQA even a small KB, like Freebase2M [1], contains more than 6,000 relation types. Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data. For example, the SimpleQuestions [1] data set has 14% of the golden test relations not observed in golden training tuples. Third, as shown in Figure 1(b), for some KBQA tasks like WebQuestions [3], we need to predict a chain of relations instead of a single relation. This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection. Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks.

Footnote 1: In the information extraction field such tasks are usually called _relation extraction_ or _relation classification_.

This paper improves KB relation detection to cope with the problems mentioned above. First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching. Second, noticing

that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations. Third, we use deep bidirectional LSTMs (_BiLSTM_s) to learn different levels of question representations in order to match the different levels of relation information. Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching.

In order to assess how the proposed _improved relation detection_ could benefit the KBQA end task, we also propose a simple KBQA implementation composed of _two-step relation detection_. Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the _raw question text_ by the relation detection model. This step is important to deal with the ambiguities normally present in entity linking results. (2) Finding the core relation (chains) for each _topic entity_2 selection from a much smaller candidate entity set after re-ranking. The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e.g., multiple entities in the question). Finally the highest scored query from the above steps is used to query the KB for answers.

Footnote 2: Following Yih et al. (2015), here _topic entity_ refers to the root of the (directed) query tree; and _core-chain_ is the directed path of relation from root to the answer node.

Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks.

## 2 Related Work

Relation ExtractionRelation extraction (RE) is an important sub-field of information extraction. General research in this field usually works on a (small) pre-defined relation set, where given a text paragraph and two target entities, the goal is to determine whether the text indicates any types of relations between the entities or not. As a result RE is usually formulated as a **classification task**. Traditional RE methods rely on large amount of hand-crafted features Zhou et al. (2005); Rink and Harabagiu (2010); Sun et al. (2011). Recent research benefits a lot from the advancement of deep learning: from word embeddings Nguyen and Grishman (2014); Gormley et al. (2015) to deep networks like CNNs and LSTMs Zeng et al. (2014); dos Santos et al. (2015); Vu et al. (2016) and attention models Zhou et al. (2016); Wang et al. (2016).

The above research assumes there is a fixed (closed) set of relation types, thus no zero-shot learning capability is required. The number of relations is usually not large: The widely used ACE2005 has 11/32 coarse/fine-grained relations; SemEval2010 Task8 has 19 relations; TAC

Figure 1: KBQA examples and its three key components. (a) A single relation example. We first identify the topic entity with _entity linking_ and then detect the relation asked by the question with _relation detection_ (from all relations connecting the topic entity). Based on the detected entity and relation, we form a query to search the KB for the correct answer “_Love Will Find a Way_”. (b) A more complex question containing two entities. By using “_Grant Show_” as the topic entity, we could detect a chain of relations “_starring\({}_{roles-series}\)_” pointing to the answer. An additional _constraint detection_ takes the other entity “_2008_” as a constraint, to filter the correct answer “_SwingTown_” from all candidates found by the topic entity and relation.

 

KBP2015 has 74 relations although it considers open-domain Wikipedia relations. All are much fewer than thousands of relations in KBQA. As a result, few work in this field focuses on dealing with large number of relations or unseen relations. Yu et al. (2016) proposed to use relation embeddings in a low-rank tensor method. However their relation embeddings are still trained in supervised way and the number of relations is not large in the experiments.

Relation Detection in KBQA SystemsRelation detection for KBQA also starts with feature-rich approaches Yao and Van Durme (2014); Bast and Haussmann (2015) towards usages of deep networks Yih et al. (2015); Xu et al. (2016); Dai et al. (2016) and attention models Yin et al. (2016); Golub and He (2016). Many of the above relation detection research could naturally support large relation vocabulary and open relation sets (especially for QA with OpenIE KB like ParaLex Fader et al. (2013)), in order to fit the goal of open-domain question answering.

Different KBQA data sets have different levels of requirement about the above open-domain capacity. For example, most of the gold test relations in WebQuestions can be observed during training, thus some prior work on this task adopted the close domain assumption like in the general RE research. While for data sets like SimpleQuestions and ParaLex, the capacity to support large relation sets and unseen relations becomes more necessary. To the end, there are two main solutions: (1) use pre-trained relation embeddings (e.g. from TransE Bordes et al. (2013)), like Dai et al. (2016); (2) factorize the relation names to sequences and formulate relation detection as a **sequence matching and ranking** task. Such factorization works because that the relation names usually comprise meaningful word sequences. For example, Yin et al. (2016) split relations to word sequences for single-relation detection. Liang et al. (2016) also achieve good performance on WebQSP with word-level relation representation in an end-to-end neural programmer model. Yih et al. (2015) use character tri-grams as inputs on both question and relation sides. Golub and He (2016) propose a generative framework for single-relation KBQA which predicts relation with a character-level sequence-to-sequence model.

Another difference between relation detection in KBQA and general RE is that general RE research assumes that the two argument entities are both available. Thus it usually benefits from features Nguyen and Grishman (2014); Gormley et al. (2015) or attention mechanisms Wang et al. (2016) based on the entity information (e.g. entity types or entity embeddings). For relation detection in KBQA, such information is mostly missing because: (1) one question usually contains single argument (the topic entity) and (2) one KB entity could have multiple types (type vocabulary size larger than 1,500). This makes KB entity typing itself a difficult problem so no previous used entity information in the relation detection model.3

Footnote 3: Such entity information has been used in KBQA systems as features for the final answer re-rankers.

## 3 Background: Different Granularity in KB Relations

Previous research Yih et al. (2015); Yin et al. (2016) formulates KB relation detection as a sequence matching problem. However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem. Here we give an overview of two types of relation sequence representations commonly used in previous work.

* their relation embeddings \(\mathbf{h}^{r}\mathrm{s}\) will be random vectors thus are not comparable to question embeddings \(\mathbf{h}^{q}\mathrm{s}\).

**(2) Relation as Word Sequence** (_word-level_). In this case, the relation is treated as a sequence of words from the tokenized relation name. It has better generalization, but suffers from the lack of global information from the original relation names. For example in Figure 1(b), when doing only word-level matching, it is difficult to rank the target relation "_starring\({}_{roles}\)_" higher compared to the incorrect relation "_plays\({}_{produced}\)_". This is because the incorrect relation contains word "_plays_", which is more similar to the question 

(containing word "_play_") in the embedding space. On the other hand, if the target relation co-occurs with questions related to "_tv appearance_" in training, by treating the whole relation as a token (i.e. relation id), we could better learn the correspondence between this token and phrases like "_tv show_" and "_play on_".

The two types of relation representation contain different levels of abstraction. As shown in Table 1, the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity. Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score. Section 4 gives the details of our proposed approach.

## 4 Improved KB Relation Detection

This section describes our hierarchical sequence matching with residual learning approach for relation detection. In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations.

### Relation Representations from Different Granularity

We provide our model with both types of relation representation: word-level and relation-level. Therefore, the input relation becomes \({\bf r}=\{r^{word}_{1},\cdots,r^{word}_{M_{1}}\}\cup\{r^{rel}_{1},\cdots,r^{ rel}_{M_{2}}\}\), where the first \(M_{1}\) tokens are words (e.g. {_episode, written_}), and the last \(M_{2}\) tokens are relation names, e.g., {_episode_written_} or {_starting_roles, series_} (when the target is a chain like in Figure 1(b)). We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations \([{\bf B}^{word}_{1:M_{1}}:{\bf B}^{rel}_{1:M_{2}}]\) (each row vector \(\mathbf{\beta}_{i}\) is the concatenation between forward/backward representations at \(i\)). We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations. We apply _one_ max-pooling on these two sets of vectors and get the final relation representation \({\bf h}^{r}\).

### Different Abstractions of Questions Representations

From Table 1, we can see that different parts of a relation could match different contexts of question texts. Usually relation names could match longer phrases in the question and relation words could match short phrases. Yet different words might match phrases of different lengths.

As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity. We deal with this problem by applying deep BiLTSMs on questions. The first-layer of BiLSTM works on the word embeddings of question words \({\bf q}=\{q_{1},\cdots,q_{N}\}\) and gets hidden representations \(\boldsymbol{\Gamma}^{(1)}_{1:N}=[\boldsymbol{\gamma}^{(1)}_{1};\cdots; \boldsymbol{\gamma}^{(1)}_{N}]\). The second-layer BiLSTM works on \(\boldsymbol{\Gamma}^{(1)}_{1:N}\) to get the second set of hidden representations \(\boldsymbol{\Gamma}^{(2)}_{1:N}\). Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer.

Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations. This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem.

\begin{table}
\begin{tabular}{c|c c c} \hline  & **Relation Token** & \multicolumn{2}{c}{**Question 1**} & **Question 2** \\ \cline{3-5}  & what t episodes were \textless{}e\textgreater{} the writer of & what episode was written by \textless{}e\textgreater{} \\ \hline
**relation-level** & episodes\_written & _tv episodes were \textless{}e\textgreater{} the writer of & _episode was written by \textless{}e\textgreater{}_ \\ \cline{2-5}
**word-level** & episodes & _tv episodes_ & _episode_ \\ \cline{2-5}  & written & _the writer of_ & _written_ \\ \hline \end{tabular}
\end{table}
Table 1: An example of KB relation (_episodes_written_) with two types of relation tokens (relation names and words), and two questions asking this relation. The topic entity is replaced with token \textless{}e\textgreater{} which could give the position information to the deep networks. The italics show the evidence phrase for each relation token in the question.

 



### Hierarchical Matching between Relation and Question

Now we have question contexts of different lengths encoded in \({\bf\Gamma}_{1:N}^{(1)}\) and \({\bf\Gamma}_{1:N}^{(2)}\). Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (_Hierarchical Matching_). This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations. For example in Table 1, the relation word _written_ could be matched to either the same single word in the question or a much longer phrase _be the writer of_.

We could perform the above hierarchical matching by computing the similarity between each layer of \({\bf\Gamma}\) and \({\bf h}^{r}\) separately and doing the (weighted) sum between the two scores. However this does not give significant improvement (see Table 2). Our analysis in Section 6.2 shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model. This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0. (2) The training of deeper architectures itself is more difficult.

To overcome the above difficulties, we adopt the idea from Residual Networks He et al. (2016) for hierarchical matching by adding shortcut connections between two BiLSTM layers. We proposed two ways of such _Hierarchical Residual Matching_: (1) Connecting each \({\mathbf{\gamma}}_{i}^{(1)}\) and \({\mathbf{\gamma}}_{i}^{(2)}\), resulting in a \({\mathbf{\gamma}}_{i}^{\prime}={\mathbf{\gamma}}_{i}^{(1)}+{\mathbf{\gamma}}_{i}^{(2)}\) for each position \(i\). Then the final question representation \({\bf h}^{q}\) becomes a max-pooling over all \({\mathbf{\gamma}}_{i}^{\prime}\)s, 1\(\leq\)i\(\leq\)N. (2) Applying max-pooling on \({\bf\Gamma}_{1:N}^{(1)}\) and \({\bf\Gamma}_{1:N}^{(2)}\) to get \({\bf h}_{max}^{(1)}\) and \({\bf h}_{max}^{(2)}\), respectively, then setting \({\bf h}^{q}={\bf h}_{max}^{(1)}+{\bf h}_{max}^{(2)}\). Finally we compute the matching score of \({\bf r}\) given \({\bf q}\) as \(s_{rel}({\bf r};{\bf q})=cos({\bf h}^{r},{\bf h}^{q})\).

Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other. This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier.

During training we adopt a ranking loss to maximizing the margin between the gold relation \({\bf r}^{+}\) and other relations \({\bf r}^{-}\) in the candidate pool \(R\).

\[l_{\rm rel}=\max\{0,\gamma-s_{\rm rel}({\bf r}^{+};{\bf q})+s_{\rm rel}({\bf r }^{-};{\bf q})\}\]

where \(\gamma\) is a constant parameter. Fig 2 summarizes the above _Hierarchical Residual BiLSTM (**HR-BiLSTM**) model.

Figure 2: The proposed Hierarchical Residual BiLSTM (HR-BiLSTM) model for relation detection. Note that without the dotted arrows of shortcut connections between two layers, the model will only compute the similarity between the second-layer of questions representations and the relation, thus is not doing hierarchical matching.



Remark:Another way of hierarchical matching consists in relying on **attention mechanism**, e.g. [16], to find the correspondence between different levels of representations. This performs below the HR-BiLSTM (see Table 2).

## 5 KBQA Enhanced by Relation Detection

This section describes our KBQA pipeline system. We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build.

Following previous work [30; 34], our KBQA system takes an existing entity linker to produce the top-\(K\) linked entities, \(EL_{K}(q)\), for a question \(q\) ("_initial entity linking_"). Then we generate the KB queries for \(q\) following the four steps illustrated in Algorithm 1.

```
Input : Question \(q\), Knowledge Base \(KB\), the initial top-\(K\) entity candidates \(EL_{K}(q)\) Output : Top query tuple \((\hat{e},\hat{r},\{c,r_{c}\})\)
1Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in \(EL_{K}(q)\); use the relation scores to re-rank \(EL_{K^{\prime}}(q)\) and generate a shorter list \(EL^{\prime}_{K^{\prime}}(q)\) containing the top-\(K^{\prime}\) entity candidates (Section 5.1)
2Relation Detection: Detect relation(s) using the _referenced question text_ in which the topic entity is replaced by a special token <=\(>\) (Section 5.2)
3Query Generation: Combine the scores from step 1 and 2, and select the top pair \((\hat{e},\hat{r})\) (Section 5.3)
4Constraint Detection (optional): Compute similarity between \(q\) and any neighbor entity \(c\) of the entities along \(\hat{r}\) (connecting by a relation \(r_{c}\) ), add the high scoring \(c\) and \(r_{c}\) to the query (Section 5.4).
```

**Algorithm 1**kBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional _entity re-ranking_ step after the _initial entity linking_. We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems. For example, on SimpleQuestions the best reported linker could only get 72.7% top-1 accuracy on identifying topic entities. This is usually due to the ambiguities of entity names, e.g. in Fig 1(a), there are _TV writer_ and _baseball player_ "_Mike Kelley_", which is impossible to distinguish with only entity name matching.

Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the _initial entity linking_ with relations detected in questions.

Sections 5.1 and 5.2 elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection. The KBQA end task, as a result, benefits from this process.

### Entity Re-Ranking

In this step, we use the _raw question text_ as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in \(EL_{K}(q)\). We call this step _relation detection on entity set_ since it does not work on a single topic entity as the usual settings. We use the HR-BiLSTM as described in Sec. 4. For each question \(q\), after generating a score \(s_{rel}(r;q)\) for each relation using HR-BiLSTM, we use the top \(l\) best scoring relations (\(R_{q}^{l}\)) to re-rank the original entity candidates. Concretely, for each entity \(e\) and its associated relations \(R_{e}\), given the original entity linker score \(s_{linker}\), and the score of the most confident relation \(r\in R_{q}^{l}\cap R_{e}\), we sum these two scores to re-rank the entities:

\[s_{\mathrm{remark}}(e;q)= \alpha\cdot s_{\mathrm{linker}}(e;q)\] \[+ (1-\alpha)\cdot\max_{r\in R_{q}^{l}\cap R_{e}}s_{\mathrm{rel}}( r;q).\]

Finally, we select top \(K^{\prime}<K\) entities according to score \(s_{rerank}\) to form the re-ranked list \(EL^{{}^{\prime}}_{K^{\prime}}(q)\).

We use the same example in Fig 1(a) to illustrate the idea. Given the input question in the example, a relation detector is very likely to assign high scores to relations such as "_episodes_written_", "_author_of_" and "_profession_". Then, according to the connections of entity candidates in KB, we find that the TV writer "_Mike Kelley_" will be scored higher than the baseball player "_Mike Kelley_", because the former has the relations "_episodes_written_" and "_profession_". This method can be viewed as exploiting entity-relation collocation for entity linking.

### Relation Detection

In this step, for each candidate entity \(e\in EL^{\prime}_{K}(q)\), we use the question text as the input to a relation detector to score all the relations \(r\in R_{e}\) that are associated to the entity \(e\) in the KB.4 Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate \(e\)'s entity mention in 

\(q\) with a token "\(<\!\!e\!>\)". This helps the model better distinguish the relative position of each word compared to the entity. We use the HR-BiLSTM model to predict the score of each relation \(r\in R_{e}\): \(s_{rel}(r;e,q)\).

### Query Generation

Finally, the system outputs the \(<\)entity, relation (or core-chain)\(>\) pair \((\hat{e},\hat{r})\) according to:

\[s(\hat{e},\hat{r};q)=\max_{e\in EL^{\prime}_{K^{\prime}}(q),r\in R _{e}}\left(\beta\cdot s_{\mathrm{remark}}(e;q)\right.\\ \left.+(1-\beta)\cdot s_{\mathrm{rel}}(r;e,q)\right),\]

where \(\beta\) is a hyperparameter to be tuned.

### Constraint Detection

Similar to Yih et al. (2015), we adopt an additional constraint detection step based on text matching. Our method can be viewed as entity-linking on a KB sub-graph. It contains two steps: (1) **Sub-graph generation**: given the top scored query generated by the previous 3 steps6, for each node \(v\) (answer node or the CVT node like in Figure 1(b)), we collect all the nodes \(c\) connecting to \(v\) (with relation \(r_{c}\)) with any relation, and generate a sub-graph associated to the original query. (2) **Entity-linking on sub-graph nodes**: we compute a matching score between each \(n\)-gram in the input question (without overlapping the topic entity) and entity name of \(c\) (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints). If the matching score is larger than a threshold \(\theta\) (tuned on training set), we will add the constraint entity \(c\) (and \(r_{c}\)) to the query by attaching it to the corresponding node \(v\) on the core-chain.

Footnote 6: The two resources have been downloaded from [https://github.com/Gorov/SimpleQuestions-EntityLinking](https://github.com/Gorov/SimpleQuestions-EntityLinking)

## 6 Experiments

### Task Introduction & Settings

We use the SimpleQuestions Bordes et al. (2015) and WebQSP Yih et al. (2016) datasets. Each question in these datasets is labeled with the gold semantic parse. Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task.

SimpleQuestions (SQ):It is a single-relation KBQA task. The KB we use consists of a Freebase subset with 2M entities (FB2M) Bordes et al. (2015), in order to compare with previous research. Yin et al. (2016) also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set. For the KBQA evaluation, we also start with their entity linking results7. Therefore, our results can be compared with their reported results on both tasks.

Footnote 7: [https://github.com/scottyih/STAGG](https://github.com/scottyih/STAGG)

WebQSP (WQ):A multi-relation KBQA task. We use the entire Freebase KB for evaluation purposes. Following yih2016multi, we use S-MART Yang and Chang (2015) entity-linking outputs.8 In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set.9 For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length \(\leq\) 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples.

Footnote 9: The dataset is available at [https://github.com/Gorov/SimpleQuestions-EntityLinking](https://github.com/Gorov/SimpleQuestions-EntityLinking).

We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400})10; (2) learning rate ({0.1, 0.5, 1.0, 2.0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section 4.3); and (4) the number of training epochs.

For both the relation detection experiments and the second-step relation detection in KBQA, we have _entity replacement_ first (see Section 5.2 and Figure 1). All word vectors are initialized with 300-\(d\) pretrained word embeddings Mikolov et al. (2013). The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e.g. TransE) usually support limited sets of relation names. We leave the usage of pre-trained relation embeddings to future work.

### Relation Detection Results

Table 2 shows the results on two relation detection tasks. The AMPCNN result is from Yin et al. (2016), which yielded state-of-the-art scores by outperforming several attention-based meth



ods. We re-implemented the BiCNN model from [25], where both questions and relations are represented with the word hash trick on character tri-grams. The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions. Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p \(<0.001\) and 0.01 compared to the best baseline _BiLSTM w/ words_ on SQ and WQ respectively).

Note that using only relation names instead of words results in a weaker baseline BiLSTM model. The model yields a significant performance drop on SimpleQuestions (91.2% to 88.9%). However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions.

Ablation Test:The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM. First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93.3% vs. 91.2/88.8%). Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section 4.3). For the attention-based baseline, we tried the model from [7] and its one-way variations, where the one-way model gives better results10. Note that residual learning significantly helps on WebQSP (80.65% to 82.53%), while it does not help as much on SimpleQuestions. On SimpleQuestions, even removing the deep layers only causes a small drop in performance. WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching.

Footnote 10: We also tried to apply the same attention method on deep BiLSTM with residual connections, but it does not lead to better results compared to HR-BiLSTM. We hypothesize that the idea of hierarchical matching with attention mechanism may work better for long sequences, and the new advanced attention mechanisms [22, 23] might help hierarchical matching. We leave the above directions to future work.

Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop. Yet on SimpleQuestions the gap is much smaller. We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies.

AnalysisNext, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores. We use WebQSP for the analysis purposes. First, we have the hypothesis that _training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable_. This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored. For example, one run gives us weights of -75.39/0.14 for the two layers (we take exponential for the final weighted sum). It also gives much lower training accuracy (91.94%) compared to HR-BiLSTM (95.67%), suffering from training difficulty.

Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, _training deep BiLSTMs is more difficult without shortcut connections_. Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy. In the experiments a two-layer BiLSTM converges to 94.99%, even lower than the 95.25% achieved by a

\begin{table}
\begin{tabular}{l|c|r|r} \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c|}{**Accuracy**} \\  & **Relation Input Views** & **SimpleQuestions** & **WebQSP** \\ \hline AMPCNN [25] & words & 91.3 & – \\ BiCNN [25] & char-3-gram & 90.0 & 77.74 \\ BiLSTM w/ words & words & 91.2 & 79.32 \\ BiLSTM w/ relation names & rel\_names & 88.9 & 78.96 \\ \hline Hier-Res–BiLSTM (HR-BiLSTM) & words + rel\_names & **93.3** & **82.53** \\ w/o ref\_name & words & 91.3 & 81.69 \\ w/o ref\_words & rel\_names & 88.8 & 79.68 \\ w/o residual learning (weighted sum on two layers) & words + rel\_names & 92.5 & 80.65 \\ replacing residual with attention [7] & words + rel\_names & 92.6 & 81.38 \\ single-layer BiLSTM question encoder & words + rel\_names & 92.8 & 78.41 \\ replacing BiLSTM with CNN (HR-CNN) & words + rel\_names & 92.9 & 79.08 \\ \hline \end{tabular}
\end{table}
Table 2: Accuracy on the SimpleQuestions and WebQSP relation detection tasks (test sets). The top shows performance of baselines. On the bottom we give the results of our proposed model together with the ablation tests.

 

single-layer BiLSTM. Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty.

Finally, we hypothesize that _HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction_. To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states. This decreases test accuracy to 76.11%. It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem. This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM.

### KBQA End-Task Results

Table 3 compares our system with two published baselines (1) STAGG [25], the state-of-the-art on WebQSP11 and (2) AMPCNN [24], the state-of-the-art on SimpleQuestions. Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset. In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3).

Footnote 11: The STAGG score on SQ is from [8].

Compared to the _baseline relation detector_ (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row). Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art.

The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores. Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways. Appendix C gives the detailed performance of the re-ranking step. (2) In contrast to the conclusion in [25], constraint detection is crucial for our system12. This is probably because our joint performance on topic entity and core-chain detection is more accurate (77.5% top-1 accuracy), leaving a huge potential (77.5% vs. 58.0%) for the constraint detection module to improve.

Footnote 12: Note that another reason is that we are evaluating on accuracy here. When evaluating on F1 the gap will be smaller.

Finally, like STAGG, which uses multiple relation detectors (see Yih et al. (2015) for the three models used), we also try to use the top-3 relation detectors from Section 6.2. As shown on the last row of Table 3, this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP.

## 7 Conclusion

KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks. We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations. Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts. For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems. For example, our model could be integrated into the decoder in [9], to provide better sequence prediction. We will also investigate new emerging datasets like GraphQuestions [26] and ComplexQuestions [8] to handle more characteristics of general QA.

\begin{table}
\begin{tabular}{l|c|c} \hline  & \multicolumn{2}{c}{**Accuracy**} \\ \cline{2-3}
* \\ \hline Baseline: Our Method w/ & & \\ baseline relation detector & 75.1 & 60.0 \\ \hline Our Method & **77.0** & 63.0 \\ w/o entity re-ranking & 74.9 & 60.6 \\ w/o constraints &
- & 58.0 \\ \hline Our Method (multi-detectors) & **78.7** & **63.9** \\ \hline \end{tabular}
\end{table}
Table 3: KBQA results on SimpleQuestions (SQ) and WebQSP (RQ) test sets. The numbers in _green_ color are directly comparable to our results since we start with the same entity linking results.
 



## References

* [Bao et al.2016] Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLING 2016 Organizing Committee, Osaka, Japan, pages 2503-2514.
* Bast and Haussmann (2015) Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_. ACM, pages 1431-1440.
* Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington, USA, pages 1533-1544.
* Bordes et al. [2015] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. _arXiv preprint arXiv:1506.02075_ .
* Bordes et al. [2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In _Advances in Neural Information Processing Systems_. pages 2787-2795.
* Dai et al. [2016] Zihang Dai, Lei Li, and Wei Xu. 2016. Cfo: Conditional focused neural question answering with large-scale knowledge bases. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 800-810.
* Santos et al. [2015] Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Beijing, China, pages 626-634.
* Fader et al. [2013] Anthony Fader, Luke S Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In _ACL (1)_. Citeseer, pages 1608-1618.
* Golub and He [2016] David Golub and Xiaodong He. 2016. Character-level question answering with attention. _arXiv preprint arXiv:1604.00727_ .
* Gormley et al. [2015] Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich compositional embedding models. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 1774-1784.
* He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. pages 770-778.
* Liang et al. [2016] Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. _arXiv preprint arXiv:1611.00020_ .
* Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. pages 3111-3119.
* Nguyen and Grishman [2014] Thien Huu Nguyen and Ralph Grishman. 2014. Employing word representations and regularization for domain adaptation of relation extraction. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Baltimore, Maryland, pages 68-74.
* Parikh et al. [2016] Ankur Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 2249-2255.
* Rink and Harabagiu [2010] Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and semantic resources. In _Proceedings of the 5th International Workshop on Semantic Evaluation_. Association for Computational Linguistics, Uppsala, Sweden, pages 256-259.
* Su et al. [2016] Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. 2016. On generating characteristic-rich question sets for qa evaluation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 562-572. [https://aclweb.org/anthology/D16-1054](https://aclweb.org/anthology/D16-1054).
* Sun et al. [2011] Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011. Semi-supervised relation extraction with large-scale word clustering. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Portland, Oregon, USA, pages 521-529.
* Vu et al. [2016] Ngoc Thang Vu, Heike Adel, Pankaj Gupta, and Hinrich Schutze. 2016. Combining recurrent and convolutional neural networks for relation classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 534-539

* Wang et al. (2016) Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention cnns. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 1298-1307.
* Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with lstm. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 1442-1451. [http://www.aclweb.org/anthology/N16-1170](http://www.aclweb.org/anthology/N16-1170).
* Wang et al. (2017) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. _arXiv preprint arXiv:1702.03814_ .
* Xu et al. (2016) Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 2326-2336.
* Yang and Chang (2015) Yi Yang and Ming-Wei Chang. 2015. S-mart: Novel tree-based structured learning algorithms applied to tweet entity linking. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Beijing, China, pages 504-513.
* Yao et al. (2014) Xuchen Yao, Jonathan Berant, and Benjamin Van Durme. 2014. Freebase qa: Information extraction or semantic parsing? _ACL 2014_ page 82.
* Yao and Van Durme (2014) Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In _ACL (1)_. Citeseer, pages 956-966.
* Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _Association for Computational Linguistics (ACL)_.
* Yih et al. (2014) Wen-tau Yih, Xiaodong He, and Christopher Meek. 2014. Semantic parsing for single-relation question answering. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Baltimore, Maryland, pages 643-648.
* Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 201-206.
* Yin et al. (2016) Wenpeng Yin, Mo Yu, Bing Xiang, Bowen Zhou, and Hinrich Schutze. 2016. Simple question answering by attentive convolutional neural network. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1746-1756.
* Yu et al. (2016) Mo Yu, Mark Dredze, Raman Arora, and Matthew R. Gormley. 2016. Embedding lexical features via low-rank tensors. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 1019-1029. [http://www.aclweb.org/anthology/N16-1117](http://www.aclweb.org/anthology/N16-1117).
* Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_. Dublin City University and Association for Computational Linguistics, Dublin, Ireland, pages 2335-2344.
* Zhou et al. (2005) GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In _Ascorization for Computational Linguistics_. pages 427-434.
* Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 207-212.