Acknowledgments	This work is supported by the National Natural Science Foundation of China (Nos. 61772264 and 61872172). We would like to thank Yao Zhao for his help in preparing evaluation.

Related Work	Alongside with entity and relation linking, existing KBQA systems often leverage formal query generation for complex question answering  ,  . Based on our investigation, the query generation approaches can be roughly divided into two kinds: template-based and semantic parsing-based. Template-based approaches transform the input question into a formal query by employing pre-collected query templates.  (  ) collect different natural language expressions for the same query intention from question-answer pairs.  (  ) re-implement and evaluate the query generation module in NLIWOD, which selects an existing template by some simple features such as the number of entities and relations in the input question. Recently, several query decomposition methods are studied to enlarge the coverage of the templates.  (  ) present a KBQA system named QUINT, which collects query templates for specific dependency structures from question-answer pairs. Furthermore, it rewrites the dependency parsing results for questions with conjunctions, and then performs sub-question answering and answer stitching.  (  ) decompose questions by using a huge number of triple-level templates extracted by distant supervision.

Query Structure Ranking	In this step, we use a combinational function to score each query structure in the training data for the input question. Since the prediction result for each query substructure is independent, the score for query structure INLINEFORM0 is measured by joint probability, which is DISPLAYFORM0  Assume that INLINEFORM0 , INLINEFORM1 , we have INLINEFORM2 . Thus, INLINEFORM3 should be 1 in the ideal condition. On the other hand, INLINEFORM4 , INLINEFORM5 should be 0. Thus, we have INLINEFORM6 , and INLINEFORM7 , we have INLINEFORM8 .

Introduction	Knowledge-based question answering (KBQA) aims to answer natural language questions over knowledge bases (KBs) such as DBpedia and Freebase. Formal query generation is an important component in many KBQA systems  ,  ,  , especially for answering complex questions. Given entity and relation linking results, formal query generation aims to generate correct executable queries, e.g., SPARQL queries, for the input natural language questions. An example question and its formal query are shown in Figure FIGREF1 . Generally speaking, formal query generation is expected to include but not be limited to have the capabilities of (i) recognizing and paraphrasing different kinds of constraints, including triple-level constraints (e.g., “movies" corresponds to a typing constraint for the target variable) and higher level constraints (e.g., subgraphs). For instance, “the same ... as" represents a complex structure shown in the middle of Figure FIGREF1 ; (ii) recognizing and paraphrasing aggregations (e.g., “how many" corresponds to Count); and (iii) organizing all the above to generate an executable query  ,  . There are mainly two kinds of query generation approaches for complex questions.

(i) Template-based approaches choose a pre-collected template for query generation  ,  . Such approaches highly rely on the coverage of templates, and perform unstably when some complex templates have very few natural language questions as training data. (ii) Approaches based on semantic parsing and neural networks learn entire representations for questions with different query structures, by using a neural network following the encode-and-compare framework  ,  . They may suffer from the lack of training data, especially for long-tail questions with rarely appeared structures. Furthermore, both above approaches cannot handle questions with unseen query structures, since they cannot generate new query structures. To cope with the above limitations, we propose a new query generation approach based on the following observation: the query structure for a complex question may rarely appear, but it usually contains some substructures that frequently appeared in other questions. For example, the query structure for the question in Figure FIGREF1 appears rarely, however, both “how many movies" and “the same ... as" are common expressions, which correspond to the two query substructures in dashed boxes. To collect such frequently appeared substructures, we automatically decompose query structures in the training data.

Instead of directly modeling the query structure for the given question as a whole, we employ multiple neural networks to predict query substructures contained in the question, each of which delivers a part of the query intention. Then, we select an existing query structure for the input question by using a combinational ranking function. Also, in some cases, no existing query structure is appropriate for the input question. To cope with this issue, we merge query substructures to build new query structures. The contributions of this paper are summarized below:

Preliminaries	An entity is typically denoted by a URI and described with a set of properties and values. A fact is an INLINEFORM0 triple, where the value can be either a literal or another entity. A KB is a pair INLINEFORM1 , where INLINEFORM2 denotes the set of entities and INLINEFORM3 denotes the set of facts. A formal query (or simply called query) is the structured representation of a natural language question executable on a given KB. Formally, a query is a pair INLINEFORM0 , where INLINEFORM1 denotes the set of vertices, and INLINEFORM2 denotes the set of labeled edges. A vertex can be either a variable, an entity or a literal, and the label of an edge can be either a built-in property or a user-defined one. For simplicity, the set of all edge labels are denoted by INLINEFORM3 . In this paper, the built-in properties include Count, Avg, Max, Min, MaxAtN, MinAtN and IsA (rdf:type), where the former four are used to connect two variables. For example, INLINEFORM4 represents that INLINEFORM5 is the counting result of INLINEFORM6 .

MaxAtN and MinAtN take the meaning of Order By in SPARQL  . For instance, INLINEFORM7 means Order By Desc INLINEFORM8 Limit 1 Offset 1. To classify various queries with similar query intentions and narrow the search space for query generation, we introduce the notion of query structures. A query structure is a set of structurally-equivalent queries. Let INLINEFORM0 and INLINEFORM1 denote two queries. INLINEFORM2 is structurally-equivalent to INLINEFORM3 , denoted by INLINEFORM4 , if and only if there exist two bijections INLINEFORM5 and INLINEFORM6 such that: The query structure for INLINEFORM0 is denoted by INLINEFORM1 , which contains all the queries structurally-equivalent to INLINEFORM2 . For graphical illustration, we represent a query structure by a representative query among the structurally-equivalent ones and replace entities and literals with different kinds of placeholders. An example of query and query structure is shown in the upper half of Figure FIGREF9 . For many simple questions, two query structures, i.e., INLINEFORM0 INLINEFORM1 and INLINEFORM2 INLINEFORM3 , are sufficient.

However, for complex questions, a diversity of query structures exist and some of them share a set of frequently-appeared substructures, each of which delivers a part of the query intention. We give the definition of query substructures as follows. Let INLINEFORM0 and INLINEFORM1 denote two query structures. INLINEFORM2 is a query substructure of INLINEFORM3 , denoted by INLINEFORM4 , if and only if INLINEFORM5 has a subgraph INLINEFORM6 such that INLINEFORM7 . Furthermore, if INLINEFORM8 , we say that INLINEFORM9 has INLINEFORM10 , and INLINEFORM11 is contained in INLINEFORM12 . For example, although the query structures for the two questions in Figures FIGREF1 and FIGREF9 are different, they share the same query substructure INLINEFORM0 INLINEFORM1 INLINEFORM2 , which corresponds to the phrase “how many movies". Note that, a query substructure can be the query structure of another question. The goal of this paper is to leverage a set of frequent query (sub-)structures to generate formal queries for answering complex questions.

The Proposed Approach	In this section, we present our approach, SubQG, for query generation. We first introduce the framework and general steps with a running example (Section SECREF10 ), and then describe some important steps in detail in the following subsections.

Framework	Figure FIGREF11 depicts the framework of SubQG, which contains an offline training process and an online query generation process. Offline. The offline process takes as input a set of training data in form of INLINEFORM0 pairs, and mainly contains three steps: 1. Collect query structures. For questions in the training data, we first discover the structurally-equivalent queries, and then extract the set of all query structures, denoted by INLINEFORM0 . 2. Collect frequent query substructures. We decompose each query structure INLINEFORM0 to get the set for all query substructures. Let INLINEFORM1 be a non-empty subset of INLINEFORM2 , and INLINEFORM3 be the set of vertices used in INLINEFORM4 . INLINEFORM5 should be a query substructure of INLINEFORM6 according to the definition. So, we can generate all query substructures of INLINEFORM7 from each subset of INLINEFORM8 . Disconnected query substructures would be ignored since they express discontinuous meanings and should be split into smaller query substructures. If more than INLINEFORM9 queries in training data have substructure INLINEFORM10 , we consider INLINEFORM11 as a frequent query substructure.

The set for all frequent query substructures is denoted by INLINEFORM12 . 3. Train query substructure predictors. We train a neural network for each query substructure INLINEFORM0 , to predict the probability that INLINEFORM1 has INLINEFORM2 (i.e., INLINEFORM3 ) for input question INLINEFORM4 , where INLINEFORM5 denotes the formal query for INLINEFORM6 . Details for this step are described in Section SECREF13 . Online. The online query generation process takes as input a natural language question INLINEFORM0 , and mainly contains four steps: 1. Predict query substructures. We first predict the probability that INLINEFORM0 for each INLINEFORM1 , using the query substructure predictors trained in the offline step. An example question and four query substructures with highest prediction probabilities are shown in the top of Figure FIGREF12 . 2. Rank existing query structures. To find an appropriate query structure for the input question, we rank existing query structures ( INLINEFORM0 ) by using a scoring function, see Section SECREF20 . 3. Merge query substructures.

Consider the fact that the target query structure INLINEFORM0 may not appear in INLINEFORM1 (i.e., there is no query in the training data that is structurally-equivalent to INLINEFORM2 ), we design a method (described in Section SECREF22 ) to merge question-contained query substructures for building new query structures. The merged results are ranked using the same function as existing query structures. Several query structures (including the merged results and the existing query structures) for the example question are shown in the middle of Figure FIGREF12 . 4. Grounding and validation. We leverage the query structure ranking result, alongside with the entity/relation linking result from some existing black box systems  to generate executable formal query for the input question. For each query structure, we try all possible combinations of the linking results according to the descending order of the overall linking score, and perform validation including grammar check, domain/range check and empty query check. The first non-empty query passing all validations is considered as the output for SubQG. The grounding and validation results for the example question are shown in the bottom of Figure FIGREF12 .

Query Substructure Prediction	In this step, we employ an attention based Bi-LSTM network  to predict INLINEFORM0 for each frequent query substructure INLINEFORM1 , where INLINEFORM2 represents the probability of INLINEFORM3 . There are mainly three reasons that we use a predictor for each query substructure instead of a multi-tag predictor for all query substructures: (i) a query substructure usually expresses part of the meaning of input question. Different query substructures may focus on different words or phrases, thus, each predictor should have its own attention matrix; (ii) multi-tag predictor may have a lower accuracy since each tag has unbalanced training data; (iii) single pre-trained query substructure predictor from one dataset can be directly reused on another without adjusting the network structure, however, the multi-tag predictor need to adjust the size of the output layer and retrain when the set of frequent query substructures changes. The structure of the network is shown in Figure FIGREF14 . Before the input question is fed into the network, we replace all entity mentions with INLINEFORM0 Entity INLINEFORM1 using EARL  , to enhance the generalization ability.

Given the question word sequence { INLINEFORM2 }, we first use a word embedding matrix to convert the original sequence into word vectors { INLINEFORM3 }, followed by a BiLSTM network to generate the context-sensitive representation { INLINEFORM4 } for each word, where DISPLAYFORM0  Then, the attention mechanism takes each INLINEFORM0 as input, and calculates a weight INLINEFORM1 for each INLINEFORM2 , which is formulated as follows: DISPLAYFORM0   where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . Next, we get the representation for the whole question INLINEFORM3 as the weighted sum of INLINEFORM4 : DISPLAYFORM0  The output of the network is a probability DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 . The loss function minimized during training is the binary cross-entropy: DISPLAYFORM0  where INLINEFORM0 denotes the set of training data.

Query Substructure Merging	We proposed a method, shown in Algorithm SECREF22 , to merge question-contained query substructures to build new query structures. In the initialization step, it selects some query substructures of high scores as candidates, since the query substructure may directly be the appropriate query structure for the input question. In each iteration, the method merges each question-contained substructures with existing candidates, and the merged results of high scores are used as candidates in the next iteration. The final output is the union of all the results from at most INLINEFORM0 iterations. [!t] Query substructure merging textit Question INLINEFORM1 , freq. query substructures INLINEFORM2 INLINEFORM3 INLINEFORM4 (*[f] INLINEFORM5 is maximum iterations) INLINEFORM6 to INLINEFORM7 INLINEFORM8 INLINEFORM9 INLINEFORM10 INLINEFORM11 INLINEFORM12  When merging different query substructures, we allow them to share some vertices of the same kind (variable, entity, etc.) or edge labels, except the variables which represent aggregation results. Thus, the merged result of two query substructures is a set of query structures instead of one.

Conclusion	In this paper, we introduced SubQG, a formal query generation approach based on frequent query substructures. SubQG firstly utilizes multiple neural networks to predict query substructures contained in the question, and then ranks existing query structures using a combinational function. Moreover, SubQG merges query substructures to build new query structures for questions without appropriate query structures in the training data. Our experiments showed that SubQG achieved superior results than the existing approaches, especially for complex questions. In future work, we plan to add support for other complex questions whose queries require Union, Group By, or numerical comparison. Also, we are interested in mining natural language expressions for each query substructures, which may help current parsing approaches.

Also, the following restrictions are used to filter the merged results: The merged results should be connected; The merged results have INLINEFORM0 triples; The merged results have INLINEFORM0 aggregations; An example for merging two query substructures is shown in Figure FIGREF26 .

Experiments and Results	In this section, we introduce the query generation datasets and state-of-the-art systems that we compare. We first show the end-to-end results of the query generation task, and then perform detailed analysis to show the effectiveness of each module. Question sets, source code and experimental results are available online.

Experimental Setup	We employed the same datasets as  (  ) and  (  ): (i) the large-scale complex question answering dataset (LC-QuAD)  , containing 3,253 questions with non-empty results on DBpedia (2016-04), and (ii) the fifth edition of question answering over linked data (QALD-5) dataset  , containing 311 questions with non-empty results on DBpedia (2015-10). Both datasets are widely used in KBQA studies  ,  , and have become benchmarks for some annual KBQA competitions. We did not employ the WebQuestions  dataset, since approximately 85% of its questions are simple. Also, we did not employ the ComplexQuestions  and ComplexWebQuestions  dataset, since the existing works on these datasets have not reported the formal query generation result, and it is difficult to separate the formal query generation component from the end-to-end KBQA systems in these works. All the experiments were carried out on a machine with an Intel Xeon E3-1225 3.2GHz processor, 32 GB of RAM, and an NVIDIA GTX1080Ti GPU. For the embedding layer, we used random embedding.

For each dataset, we performed 5-fold cross-validation with the train set (70%), development set (10%), and test set (20%). The threshold INLINEFORM0 for frequent query substructures is set to 30, the maximum iteration number INLINEFORM1 for merging is set to 2, INLINEFORM2 in Algorithm SECREF22 is set to INLINEFORM3 , the maximum triple number INLINEFORM4 for merged results is set to 5, and the maximum aggregation number INLINEFORM5 is set to 2. Other detailed statistics are shown in Table TABREF33 .

End-to-End Results	We compared SubQG with several existing approaches. SINA  and NLIWOD conduct query generation by predefined rules and existing templates. SQG  firstly generates candidate queries by finding valid walks containing all of entities and properties mentioned in questions, and then ranks them based on Tree-LSTM similarity. CompQA  is a KBQA system which achieved state-of-the-art performance on WebQuesions and ComplexQuestions over Freebase. We re-implemented its query generation component for DBpedia, which generates candidate queries by staged query generation, and ranks them using an encode-and-compare network. The average F1-scores for the end-to-end query generation task are reported in Table TABREF35 . All these results are based on the gold standard entity/relation linking result as input. Our approach SubQG outperformed all the comparative approaches on both datasets. Furthermore, as the results shown in Table TABREF36 , it gained a more significant improvement on complex questions compared with CompQA.

https://github.com/dice-group/NLIWOD Both SINA and NLIWOD did not employ a query ranking mechanism, i.e., their accuracy and coverage are limited by the rules and templates. Although both CompQA and SQG have a strong ability of generating candidate queries, they perform not quite well in query ranking. According to our observation, the main reason is that these approaches tried to learn entire representations for questions with different query structures (from simple to complex) using a single network, thus, they may suffer from the lack of training data, especially for the questions with rarely appeared structures. As a contrast, our approach leveraged multiple networks to learn predictors for different query substructures, and ranked query structures using combinational function, which gained a better performance. The results on QALD-5 dataset is not as high as the result on LC-QuAD. This is because QALD-5 contains 11% of very difficult questions, requiring complex filtering conditions such as Regex and numerical comparison. These questions are currently beyond our approach's ability. Also, the size of training data is significant smaller.

Detailed Analysis	We compared the following settings of SubQG: Rank w/o substructures. We replaced the query substructure prediction and query structure ranking module, by choosing an existing query structure in the training data for the input question, using a BiLSTM multiple classification network. Rank w/ substructures We removed the merging module described in Section SECREF22 . This setting assumes that the appropriate query structure for an input question exists in the training data. Merge query substructures This setting ignored existing query structures in the training data, and only considered the merged results of query substructures. As the results shown in Table TABREF39 , the full version of SubQG achieved the best results on both datasets. Rank w/o substructures gained a comparatively low performance, especially when there is inadequate training data (on QALD-5). Compared with Rank w/ substructures, SubQG gained a further improvement, which indicates that the merging method successfully handled questions with unseen query structures. Table TABREF40 shows the accuracy of some alternative networks for query substructure prediction (Section SECREF13 ). By removing the attention mechanism (replaced by unweighted average), the accuracy declined approximately 3%.

Adding additional part of speech tag sequence of the input question gained no significant improvement. We also tried to replace the attention based BiLSTM with the network in  , which encodes questions with a convolutional layer followed by a max pooling layer. This approach did not perform well since it cannot capture long-term dependencies. We simulated the real KBQA environment by considering noisy entity/relation linking results. We firstly mixed the correct linking result for each mention with the top-5 candidates generated from EARL  , which is a joint entity/relation linking system with state-of-the-art performance on LC-QuAD. The result is shown in the second row of Table TABREF42 . Although the precision for first output declined 11.4%, in 85% cases we still can generate correct answer in top-5. This is because SubQG ranked query structures first and considered linking results in the last step. Many error linking results can be filtered out by the empty query check or domain/range check. We also test the performance of our approach only using the EARL linking results. The performance dropped dramatically in comparison to the first two rows.

The main reason is that, for 82.8% of the questions, EARL provided partially correct results. If we consider the remaining questions, our system again have 73.2% and 84.8% of correctly-generated queries in top-1 and top-5 output, respectively. We tested the performance of SubQG with different sizes of training data. The results on LC-QuAD dataset are shown in Figure FIGREF44 . With more training data, our query substructure based approaches obtained stable improvements on both precision and recall. Although the merging module impaired the overall precision a little bit, it shows a bigger improvement on recall, especially when there is very few training data. Generally speaking, equipped with the merging module, our substructure based query generation approach showed the best performance. We analyzed 100 randomly sampled questions that SubQG did not return correct answers. The major causes of errors are summarized as follows: Query structure errors (71%) occurred due to multiple reasons. Firstly, 21% of error cases have entity mentions that are not correctly detected before query substructure prediction, which highly influenced the prediction result.

Secondly, in 39% of the cases a part of substructure predictors provided wrong prediction, which led to wrong structure ranking results. Finally, in the remaining 11% of the cases the correct query structure did not appear in the training data, and they cannot be generated by merging substructures. Grounding errors (29%) occurred when SubQG generated wrong queries with correct query structures. For example, for the question “Was Kevin Rudd the prime minister of Julia Gillard", SubQG cannot distinguish INLINEFORM0 from INLINEFORM1 INLINEFORM2 , since both triples exist in DBpedia. We believe that extra training data are required for fixing this problem.

Compared with these approaches, our approach predicts all kinds of query substructures (usually 1 to 4 triples) contained in the question, making full use of the training data. Also, our merging method can handle questions with unseen query structures, having a larger coverage and a more stable performance. Semantic parsing-based approaches translate questions into formal queries using bottom up parsing  or staged query graph generation  . gAnswer  ,  builds up semantic query graph for question analysis and utilize subgraph matching for disambiguation. Recent studies combine parsing based approaches with neural networks, to enhance the ability for structure disambiguation.  (  ),  (  ) and  (  ) build query graphs by staged query generation, and follow an encode-and-compare framework to rank candidate queries with neural networks. These approaches try to learn entire representations for questions with different query structures by using a single network. Thus, they may suffer from the lack of training data, especially for questions with rarely appeared structures. By contrast, our approach utilizes multiple networks to learn predictors for different query substructures, which can gain a stable performance with limited training data.

Also, our approach does not require manually-written rules, and performs stably with noisy linking results.

Leveraging Frequent Query Substructures to Generate Formal Queries for Complex Question Answering	Formal query generation aims to generate correct executable queries for question answering over knowledge bases (KBs), given entity and relation linking results. Current approaches build universal paraphrasing or ranking models for the whole questions, which are likely to fail in generating queries for complex, long-tail questions. In this paper, we propose SubQG, a new query generation approach based on frequent query substructures, which helps rank the existing (but nonsignificant) query structures or build new query structures. Our experiments on two benchmark datasets show that our approach significantly outperforms the existing ones, especially for complex questions. Also, it achieves promising performance with limited training data and noisy entity/relation linking results.

Question: What are their evaluation metrics?
Answer: average F1-score, accuracy