prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is te core component for KBQA?
Context 1: Introduction
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5   For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB  Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question  The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to 
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system  Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration  As a result, there is a significant gap between general relation detection studies and KB-specific relation detection  First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100  In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types  Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data  For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples  Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation  This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection  Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks 
This paper improves KB relation detection to cope with the problems mentioned above  First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching  Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations  Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information  Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching 
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection  Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model  This step is important to deal with the ambiguities normally present in entity linking results  (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking  The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e g , multiple entities in the question)  Finally the highest scored query from the above steps is used to query the KB for answers 
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks 
Background: Different Granularity in KB Relations
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem  However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem  Here we give an overview of two types of relation sequence representations commonly used in previous work 
(1) Relation Name as a Single Token (relation-level)  In this case, each relation name is treated as a unique token  The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations  For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s 
(2) Relation as Word Sequence (word-level)  In this case, the relation is treated as a sequence of words from the tokenized relation name  It has better generalization, but suffers from the lack of global information from the original relation names  For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”  This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space  On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i e  relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on” 
The two types of relation representation contain different levels of abstraction  As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity  Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score  Section "Improved KB Relation Detection" gives the details of our proposed approach 
Improved KB Relation Detection
This section describes our hierarchical sequence matching with residual learning approach for relation detection  In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations 
Relation Representations from Different Granularity
We provide our model with both types of relation representation: word-level and relation-level  Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e g  {episode, written}), and the last $M_2$ tokens are relation names, e g , {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b))  We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ )  We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations  We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$  
Different Abstractions of Questions Representations
From Table 1 , we can see that different parts of a relation could match different contexts of question texts  Usually relation names could match longer phrases in the question and relation words could match short phrases  Yet different words might match phrases of different lengths 
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity  We deal with this problem by applying deep BiLSTMs on questions  The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$   The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$   Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer 
Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations  This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem 
Hierarchical Matching between Relation and Question
Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$   Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching)  This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations  For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of 
We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores  However this does not give significant improvement (see Table 2 )  Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model  This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0  (2) The training of deeper architectures itself is more difficult 
To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers  We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$   Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$   (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3   Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6  
Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other  This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier 
During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$  
$$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq  12)
where $\gamma $ is a constant parameter  Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model 
Another way of hierarchical matching consists in relying on attention mechanism, e g  BIBREF24 , to find the correspondence between different levels of representations  This performs below the HR-BiLSTM (see Table 2 ) 
KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system  We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build 
Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”)  Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection"  
[htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" )  KBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking  We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems  For example, on SimpleQuestions the best reported linker could only get 72 7% top-1 accuracy on identifying topic entities  This is usually due to the ambiguities of entity names, e g  in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching 
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions 
Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection  The KBQA end task, as a result, benefits from this process 
Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$   We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings  We use the HR-BiLSTM as described in Sec  "Improved KB Relation Detection"   For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates  Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities:
$$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q) \nonumber $$   (Eq  15)
Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$  
We use the same example in Fig 1 (a) to illustrate the idea  Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”  Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”  This method can be viewed as exploiting entity-relation collocation for entity linking 
Relation Detection
In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB  Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”  This helps the model better distinguish the relative position of each word compared to the entity  We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$  
Query Generation
Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to:
$$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right  \nonumber \\ &\left + (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq  19)
where $\beta $ is a hyperparameter to be tuned 
Constraint Detection
Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching  Our method can be viewed as entity-linking on a KB sub-graph  It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query  (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints)  If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain 
Experiments
Task Introduction & Settings
We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets  Each question in these datasets is labeled with the gold semantic parse  Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task 
SimpleQuestions (SQ): It is a single-relation KBQA task  The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research  yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set  For the KBQA evaluation, we also start with their entity linking results  Therefore, our results can be compared with their reported results on both tasks 
WebQSP (WQ): A multi-relation KBQA task  We use the entire Freebase KB for evaluation purposes  Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs  In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set  For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples 
We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0 1, 0 5, 1 0, 2 0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs 
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 )  All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27   The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e g  TransE) usually support limited sets of relation names  We leave the usage of pre-trained relation embeddings to future work 
Relation Detection Results
Table 2 shows the results on two relation detection tasks  The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods  We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams  The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions  Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0 001 and 0 01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively) 
Note that using only relation names instead of words results in a weaker baseline BiLSTM model  The model yields a significant performance drop on SimpleQuestions (91 2% to 88 9%)  However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions 
The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM  First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93 3% vs  91 2/88 8%)  Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" )  For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results  Note that residual learning significantly helps on WebQSP (80 65% to 82 53%), while it does not help as much on SimpleQuestions  On SimpleQuestions, even removing the deep layers only causes a small drop in performance  WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching 
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop  Yet on SimpleQuestions the gap is much smaller  We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies 
Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores  We use WebQSP for the analysis purposes  First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable  This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored  For example, one run gives us weights of -75 39/0 14 for the two layers (we take exponential for the final weighted sum)  It also gives much lower training accuracy (91 94%) compared to HR-BiLSTM (95 67%), suffering from training difficulty 
Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections  Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy  In the experiments a two-layer BiLSTM converges to 94 99%, even lower than the 95 25% achieved by a single-layer BiLSTM  Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty 
Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction  To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states  This decreases test accuracy to 76 11%  It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem  This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM 
KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions  Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset  In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ) 
Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row)  Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art 
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores  Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways  Appendix C gives the detailed performance of the re-ranking step  (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system  This is probably because our joint performance on topic entity and core-chain detection is more accurate (77 5% top-1 accuracy), leaving a huge potential (77 5% vs  58 0%) for the constraint detection module to improve 
Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results"   As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP 
Conclusion
KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks  We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations  Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts  For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems  For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction  We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA 
Context 2: Introduction
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5   For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB  Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question  The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to 
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system  Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration  As a result, there is a significant gap between general relation detection studies and KB-specific relation detection  First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100  In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types  Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data  For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples  Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation  This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection  Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks 
This paper improves KB relation detection to cope with the problems mentioned above  First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching  Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations  Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information  Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching 
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection  Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model  This step is important to deal with the ambiguities normally present in entity linking results  (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking  The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e g , multiple entities in the question)  Finally the highest scored query from the above steps is used to query the KB for answers 
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks 
Background: Different Granularity in KB Relations
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem  However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem  Here we give an overview of two types of relation sequence representations commonly used in previous work 
(1) Relation Name as a Single Token (relation-level)  In this case, each relation name is treated as a unique token  The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations  For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s 
(2) Relation as Word Sequence (word-level)  In this case, the relation is treated as a sequence of words from the tokenized relation name  It has better generalization, but suffers from the lack of global information from the original relation names  For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”  This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space  On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i e  relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on” 
The two types of relation representation contain different levels of abstraction  As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity  Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score  Section "Improved KB Relation Detection" gives the details of our proposed approach 
Improved KB Relation Detection
This section describes our hierarchical sequence matching with residual learning approach for relation detection  In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations 
Relation Representations from Different Granularity
We provide our model with both types of relation representation: word-level and relation-level  Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e g  {episode, written}), and the last $M_2$ tokens are relation names, e g , {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b))  We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ )  We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations  We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$  
Different Abstractions of Questions Representations
From Table 1 , we can see that different parts of a relation could match different contexts of question texts  Usually relation names could match longer phrases in the question and relation words could match short phrases  Yet different words might match phrases of different lengths 
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity  We deal with this problem by applying deep BiLSTMs on questions  The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$   The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$   Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer 
Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations  This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem 
Hierarchical Matching between Relation and Question
Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$   Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching)  This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations  For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of 
We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores  However this does not give significant improvement (see Table 2 )  Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model  This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0  (2) The training of deeper architectures itself is more difficult 
To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers  We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$   Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$   (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3   Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6  
Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other  This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier 
During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$  
$$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq  12)
where $\gamma $ is a constant parameter  Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model 
Another way of hierarchical matching consists in relying on attention mechanism, e g  BIBREF24 , to find the correspondence between different levels of representations  This performs below the HR-BiLSTM (see Table 2 ) 
KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system  We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build 
Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”)  Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection"  
[htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" )  KBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking  We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems  For example, on SimpleQuestions the best reported linker could only get 72 7% top-1 accuracy on identifying topic entities  This is usually due to the ambiguities of entity names, e g  in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching 
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions 
Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection  The KBQA end task, as a result, benefits from this process 
Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$   We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings  We use the HR-BiLSTM as described in Sec  "Improved KB Relation Detection"   For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates  Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities:
$$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q) \nonumber $$   (Eq  15)
Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$  
We use the same example in Fig 1 (a) to illustrate the idea  Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”  Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”  This method can be viewed as exploiting entity-relation collocation for entity linking 
Relation Detection
In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB  Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”  This helps the model better distinguish the relative position of each word compared to the entity  We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$  
Query Generation
Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to:
$$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right  \nonumber \\ &\left + (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq  19)
where $\beta $ is a hyperparameter to be tuned 
Constraint Detection
Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching  Our method can be viewed as entity-linking on a KB sub-graph  It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query  (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints)  If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain 
Experiments
Task Introduction & Settings
We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets  Each question in these datasets is labeled with the gold semantic parse  Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task 
SimpleQuestions (SQ): It is a single-relation KBQA task  The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research  yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set  For the KBQA evaluation, we also start with their entity linking results  Therefore, our results can be compared with their reported results on both tasks 
WebQSP (WQ): A multi-relation KBQA task  We use the entire Freebase KB for evaluation purposes  Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs  In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set  For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples 
We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0 1, 0 5, 1 0, 2 0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs 
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 )  All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27   The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e g  TransE) usually support limited sets of relation names  We leave the usage of pre-trained relation embeddings to future work 
Relation Detection Results
Table 2 shows the results on two relation detection tasks  The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods  We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams  The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions  Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0 001 and 0 01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively) 
Note that using only relation names instead of words results in a weaker baseline BiLSTM model  The model yields a significant performance drop on SimpleQuestions (91 2% to 88 9%)  However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions 
The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM  First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93 3% vs  91 2/88 8%)  Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" )  For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results  Note that residual learning significantly helps on WebQSP (80 65% to 82 53%), while it does not help as much on SimpleQuestions  On SimpleQuestions, even removing the deep layers only causes a small drop in performance  WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching 
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop  Yet on SimpleQuestions the gap is much smaller  We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies 
Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores  We use WebQSP for the analysis purposes  First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable  This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored  For example, one run gives us weights of -75 39/0 14 for the two layers (we take exponential for the final weighted sum)  It also gives much lower training accuracy (91 94%) compared to HR-BiLSTM (95 67%), suffering from training difficulty 
Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections  Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy  In the experiments a two-layer BiLSTM converges to 94 99%, even lower than the 95 25% achieved by a single-layer BiLSTM  Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty 
Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction  To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states  This decreases test accuracy to 76 11%  It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem  This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM 
KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions  Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset  In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ) 
Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row)  Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art 
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores  Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways  Appendix C gives the detailed performance of the re-ranking step  (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system  This is probably because our joint performance on topic entity and core-chain detection is more accurate (77 5% top-1 accuracy), leaving a huge potential (77 5% vs  58 0%) for the constraint detection module to improve 
Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results"   As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP 
Conclusion
KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks  We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations  Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts  For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems  For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction  We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA 
Context 3: Introduction
Understanding the emotions expressed in a text or message is of high relevance nowadays  Companies are interested in this to get an understanding of the sentiment of their current customers regarding their products and the sentiment of their potential customers to attract new ones  Moreover, changes in a product or a company may also affect the sentiment of a customer  However, the intensity of an emotion is crucial in determining the urgency and importance of that sentiment  If someone is only slightly happy about a product, is a customer willing to buy it again? Conversely, if someone is very angry about customer service, his or her complaint might be given priority over somewhat milder complaints 
BIBREF0 present four tasks in which systems have to automatically determine the intensity of emotions (EI) or the intensity of the sentiment (Valence) of tweets in the languages English, Arabic, and Spanish  The goal is to either predict a continuous regression (reg) value or to do ordinal classification (oc) based on a number of predefined categories  The EI tasks have separate training sets for four different emotions: anger, fear, joy and sadness  Due to the large number of subtasks and the fact that this language does not have many resources readily available, we only focus on the Spanish subtasks  Our work makes the following contributions:
Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets  This paper will first focus on the datasets, the data generation procedure, and the techniques and tools used  Then we present the results in detail, after which we perform a small error analysis on the largest mistakes our model made  We conclude with some possible ideas for future work 
Data
For each task, the training data that was made available by the organizers is used, which is a selection of tweets with for each tweet a label describing the intensity of the emotion or sentiment BIBREF1   Links and usernames were replaced by the general tokens URL and @username, after which the tweets were tokenized by using TweetTokenizer  All text was lowercased  In a post-processing step, it was ensured that each emoji is tokenized as a single token 
Word Embeddings
To be able to train word embeddings, Spanish tweets were scraped between November 8, 2017 and January 12, 2018  We chose to create our own embeddings instead of using pre-trained embeddings, because this way the embeddings would resemble the provided data set: both are based on Twitter data  Added to this set was the Affect in Tweets Distant Supervision Corpus (DISC) made available by the organizers BIBREF0 and a set of 4 1 million tweets from 2015, obtained from BIBREF2   After removing duplicate tweets and tweets with fewer than ten tokens, this resulted in a set of 58 7 million tweets, containing 1 1 billion tokens  The tweets were preprocessed using the method described in Section SECREF6   The word embeddings were created using word2vec in the gensim library BIBREF3 , using CBOW, a window size of 40 and a minimum count of 5  The feature vectors for each tweet were then created by using the AffectiveTweets WEKA package BIBREF4  
Translating Lexicons
Most lexical resources for sentiment analysis are in English  To still be able to benefit from these sources, the lexicons in the AffectiveTweets package were translated to Spanish, using the machine translation platform Apertium BIBREF5  
All lexicons from the AffectiveTweets package were translated, except for SentiStrength  Instead of translating this lexicon, the English version was replaced by the Spanish variant made available by BIBREF6  
For each subtask, the optimal combination of lexicons was determined  This was done by first calculating the benefits of adding each lexicon individually, after which only beneficial lexicons were added until the score did not increase anymore (e g  after adding the best four lexicons the fifth one did not help anymore, so only four were added)  The tests were performed using a default SVM model, with the set of word embeddings described in the previous section  Each subtask thus uses a different set of lexicons (see Table TABREF1 for an overview of the lexicons used in our final ensemble)  For each subtask, this resulted in a (modest) increase on the development set, between 0 01 and 0 05 
Translating Data
The training set provided by BIBREF0 is not very large, so it was interesting to find a way to augment the training set  A possible method is to simply translate the datasets into other languages, leaving the labels intact  Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish  This new set of “Spanish” data was then added to our original training set  Again, the machine translation platform Apertium BIBREF5 was used for the translation of the datasets

ground_truth: answer questions by obtaining information from KB tuples 

hierarchical matching between questions and relations with residual learning

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On which benchmarks they achieve the state of the art?
Context 3: 
Algorithms Used
Three types of models were used in our system, a feed-forward neural network, an LSTM network and an SVM regressor  The neural nets were inspired by the work of Prayas BIBREF7 in the previous shared task  Different regression algorithms (e g  AdaBoost, XGBoost) were also tried due to the success of SeerNet BIBREF8 , but our study was not able to reproduce their results for Spanish 
For both the LSTM network and the feed-forward network, a parameter search was done for the number of layers, the number of nodes and dropout used  This was done for each subtask, i e  different tasks can have a different number of layers  All models were implemented using Keras BIBREF9   After the best parameter settings were found, the results of 10 system runs to produce our predictions were averaged (note that this is different from averaging our different type of models in Section SECREF16 )  For the SVM (implemented in scikit-learn BIBREF10 ), the RBF kernel was used and a parameter search was conducted for epsilon  Detailed parameter settings for each subtask are shown in Table TABREF12   Each parameter search was performed using 10-fold cross validation, as to not overfit on the development set 
Semi-supervised Learning
One of the aims of this study was to see if using semi-supervised learning is beneficial for emotion intensity tasks  For this purpose, the DISC BIBREF0 corpus was used  This corpus was created by querying certain emotion-related words, which makes it very suitable as a semi-supervised corpus  However, the specific emotion the tweet belonged to was not made public  Therefore, a method was applied to automatically assign the tweets to an emotion by comparing our scraped tweets to this new data set 
First, in an attempt to obtain the query-terms, we selected the 100 words which occurred most frequently in the DISC corpus, in comparison with their frequencies in our own scraped tweets corpus  Words that were clearly not indicators of emotion were removed  The rest was annotated per emotion or removed if it was unclear to which emotion the word belonged  This allowed us to create silver datasets per emotion, assigning tweets to an emotion if an annotated emotion-word occurred in the tweet 
Our semi-supervised approach is quite straightforward: first a model is trained on the training set and then this model is used to predict the labels of the silver data  This silver data is then simply added to our training set, after which the model is retrained  However, an extra step is applied to ensure that the silver data is of reasonable quality  Instead of training a single model initially, ten different models were trained which predict the labels of the silver instances  If the highest and lowest prediction do not differ more than a certain threshold the silver instance is maintained, otherwise it is discarded 
This results in two parameters that could be optimized: the threshold and the number of silver instances that would be added  This method can be applied to both the LSTM and feed-forward networks that were used  An overview of the characteristics of our data set with the final parameter settings is shown in Table TABREF14   Usually, only a small subset of data was added to our training set, meaning that most of the silver data is not used in the experiments  Note that since only the emotions were annotated, this method is only applicable to the EI tasks 
Ensembling
To boost performance, the SVM, LSTM, and feed-forward models were combined into an ensemble  For both the LSTM and feed-forward approach, three different models were trained  The first model was trained on the training data (regular), the second model was trained on both the training and translated training data (translated) and the third one was trained on both the training data and the semi-supervised data (silver)  Due to the nature of the SVM algorithm, semi-supervised learning does not help, so only the regular and translated model were trained in this case  This results in 8 different models per subtask  Note that for the valence tasks no silver training data was obtained, meaning that for those tasks the semi-supervised models could not be used 
Per task, the LSTM and feed-forward model's predictions were averaged over 10 prediction runs  Subsequently, the predictions of all individual models were combined into an average  Finally, models were removed from the ensemble in a stepwise manner if the removal increased the average score  This was done based on their original scores, i e  starting out by trying to remove the worst individual model and working our way up to the best model

ground_truth: SimpleQuestions, WebQSP

WebQSP, SimpleQuestions

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were the scores of their system?
Context 3:  We only consider it an increase in score if the difference is larger than 0 002 (i e  the difference between 0 716 and 0 718)  If at some point the score does not increase and we are therefore unable to remove a model, the process is stopped and our best ensemble of models has been found  This process uses the scores on the development set of different combinations of models  Note that this means that the ensembles for different subtasks can contain different sets of models  The final model selections can be found in Table TABREF17  
Results and Discussion
Table TABREF18 shows the results on the development set of all individuals models, distinguishing the three types of training: regular (r), translated (t) and semi-supervised (s)  In Tables TABREF17 and TABREF18 , the letter behind each model (e g  SVM-r, LSTM-r) corresponds to the type of training used  Comparing the regular and translated columns for the three algorithms, it shows that in 22 out of 30 cases, using translated instances as extra training data resulted in an improvement  For the semi-supervised learning approach, an improvement is found in 15 out of 16 cases  Moreover, our best individual model for each subtask (bolded scores in Table TABREF18 ) is always either a translated or semi-supervised model  Table TABREF18 also shows that, in general, our feed-forward network obtained the best results, having the highest F-score for 8 out of 10 subtasks 
However, Table TABREF19 shows that these scores can still be improved by averaging or ensembling the individual models  On the dev set, averaging our 8 individual models results in a better score for 8 out of 10 subtasks, while creating an ensemble beats all of the individual models as well as the average for each subtask  On the test set, however, only a small increase in score (if any) is found for stepwise ensembling, compared to averaging  Even though the results do not get worse, we cannot conclude that stepwise ensembling is a better method than simply averaging 
Our official scores (column Ens Test in Table TABREF19 ) have placed us second (EI-Reg, EI-Oc), fourth (V-Reg) and fifth (V-Oc) on the SemEval AIT-2018 leaderboard  However, it is evident that the results obtained on the test set are not always in line with those achieved on the development set  Especially on the anger subtask for both EI-Reg and EI-Oc, the scores are considerably lower on the test set in comparison with the results on the development set  Therefore, a small error analysis was performed on the instances where our final model made the largest errors 
Error Analysis
Due to some large differences between our results on the dev and test set of this task, we performed a small error analysis in order to see what caused these differences  For EI-Reg-anger, the gold labels were compared to our own predictions, and we manually checked 50 instances for which our system made the largest errors 
Some examples that were indicative of the shortcomings of our system are shown in Table TABREF20   First of all, our system did not take into account capitalization  The implications of this are shown in the first sentence, where capitalization intensifies the emotion used in the sentence  In the second sentence, the name Imperator Furiosa is not understood  Since our texts were lowercased, our system was unable to capture the named entity and thought the sentence was about an angry emperor instead  In the third sentence, our system fails to capture that when you are so angry that it makes you laugh, it results in a reduced intensity of the angriness  Finally, in the fourth sentence, it is the figurative language me infla la vena (it inflates my vein) that the system is not able to understand 
The first two error-categories might be solved by including smart features regarding capitalization and named entity recognition  However, the last two categories are problems of natural language understanding and will be very difficult to fix 
Conclusion
To conclude, the present study described our submission for the Semeval 2018 Shared Task on Affect in Tweets  We participated in four Spanish subtasks and our submissions ranked second, second, fourth and fifth place  Our study aimed to investigate whether the automatic generation of additional training data through translation and semi-supervised learning, as well as the creation of stepwise ensembles, increase the performance of our Spanish-language models  Strong support was found for the translation and semi-supervised learning approaches; our best models for all subtasks use either one of these approaches  These results suggest that both of these additional data resources are beneficial when determining emotion intensity (for Spanish)  However, the creation of a stepwise ensemble from the best models did not result in better performance compared to simply averaging the models  In addition, some signs of overfitting on the dev set were found  In future work, we would like to apply the methods (translation and semi-supervised learning) used on Spanish on other low-resource languages and potentially also on other tasks 

ground_truth: column Ens Test in Table TABREF19