Evaluation on the Domain-specific KB	The last experiment is on our in-house KB in the power tool domain. It contains 67 distinct predicates, 293 distinct subjects and 279 distinct objects respectively. For the 67 predicates, we hand-craft 163 templates. Here we use the same language model as in our first experiment, and learn a skip-gram model  on Wikipedia for evaluating domain relevance. We generate 12,228 seed questions from which 20,000 more questions are expanded with Google. Shown in Table 4 are some expanded questions from which we can see that most of them are grammatical and relevant to the power tool domain. In addition, most questions are informative and correspond to a specific answer, except the one “do I need a hammer drill” that lacks context information. Finally, in addition to the simple factoid questions, our system generates many complex questions such as “how to cut a groove in wood without a router”.

On the other hand, the question part of the 10,000 pairs are human generated, which requires a large amount of human effort. In addition, the grammaticality and naturalness of generated questions can not be guaranteed (as seen in Table 1 ). We propose a system for generating questions from KB that significantly reduces the human effort by leveraging the massive web resources. Given a KB, a small set of question templates are first hand-crafted based on the predicates in the KB. These templates consist of a transcription of the predicate in the KB (e.g. performsActivity $\Rightarrow $ how to) and placeholders for the subject (#X#) and the object (#Y#). A seed question set is then generated by applying the templates on the KB. The seed question set is further expanded through a search engine (e.g., Google, Bing), by iteratively forming each generated question as a search query to retrieve more related question candidates. Finally a selection step is applied by estimating the fluency and domain relevance of each question candidate. The only human labor in this work is the question template construction.

System	Shown in Figure 1 , our system contains the sub-modules of question template construction, seed question generation, question expansion and selection. Given an input KB, a small set of question templates is first constructed such that each template is associated with a predicate, then a seed question set is generated by applying the template set on the input KB, before finally more questions are generated from related questions that are iteratively retrieved from a search engine with already-obtained questions as search queries (section "Experiments" ). Taking our in-house KB of power tool domain as an example, template “how to use #X#” is first constructed for predicate “performsActivity”. In addition, seed question “how to use jigsaw” is generated by applying the template on triple “ $\langle $ jigsaw, performsActivity, CurveCut $\rangle $ ”, before finally questions (Figure 2 ) are retrieved from Google with the seed question.

Evaluation on Freebase	We first compare our system with serban-EtAl:2016:P16-1 on 500 randomly selected triples from Freebase  . For the 500 triples, we hand-crafted 106 templates, as these triples share only 53 distinct predicates (we made 2 templates for each predicate on average). 991 seed questions are generated by applying the templates on the triples, and 1529 more questions are retrieved from Google. To evaluate the fluency of the candidate questions, we train a 4-gram language model (LM) on gigaword (LDC2011T07) with Kneser Ney smoothing. Using the averaged language model score as index, the top 500 questions are selected to compare with the results from serban-EtAl:2016:P16-1. We ask three native English speakers to evaluate the fluency and the naturalness of both results based on a 4-point scheme where 4 is the best. We show the averaged human rate in Table 2 , where we can see that our questions are more grammatical and natural than serban-EtAl:2016:P16-1.

Question Generation from a Knowledge Base with Web Exploration	Question generation from a knowledge base (KB) is the task of generating questions related to the domain of the input KB. We propose a system for generating fluent and natural questions from a KB, which significantly reduces the human effort by leveraging massive web resources. In more detail, a seed question set is first generated by applying a small number of hand-crafted templates on the input KB, then more questions are retrieved by iteratively forming already obtained questions as search queries into a standard search engine, before finally questions are selected by estimating their fluency and domain relevance. Evaluated by human graders on 500 random-selected triples from Freebase, questions generated by our system are judged to be more fluent than those of \newcite{serban-EtAl:2016:P16-1} by human graders.

Domain Relevance	We test our domain-relevance evaluating method on the web snippet dataset, which is a commonly-used for domain classification of short documents. It contains 10,060 training and 2,280 test snippets (short documents) in 8 classes (domains), and each snippet has 18 words on average. There have been plenty of prior results  ,  ,  on the dataset. Shown in Table 3 , we compare our domain-relevance evaluation method (section "Experiments" ) with previous state-of-the-art methods: phan2008learning first derives latent topics with LDA  from Wikipedia, then uses the topics as appended features to expand the short text. chen2011short further expanded phan2008learning by using multi-granularity topics. ma-EtAl:2015:VSM-NLP adopts a Bayesian model that the probability a document $D$ belongs to a topic $t$ equals to the prior of $t$ times the probability each word $w$ in $D$ comes from $t$ .

Introduction	Question generation is important as questions are useful for student assessment or coaching purposes in educational or professional contexts, and a large-scale corpus of question and answer pairs is also critical to many NLP tasks including question answering, dialogue interaction and intelligent tutoring systems. There has been much literature so far  ,  ,  ,  ,  ,  ,  studying question generation from text. Recently people are becoming interested in question generation from KB, since large-scale KBs, such as Freebase  and DBPedia  , are freely available, and entities and their relations are already present in KBs but not for texts. Question generation from KB is challenging as function words and morphological forms for entities are abstracted away when a KB is created. To tackle this challenge, previous work  ,  relies on massive human-labeled data. Treating question generation as a machine translation problem, serban-EtAl:2016:P16-1 train a neural machine translation (NMT) system with 10,000 $\langle $ triple, question $\rangle $ pairs. At test time, input triples are “translated” into questions with the NMT system.

Our method first concatenates training documents of the same domain into one “domain document”, then calculates each document embedding by averaging word embeddings within it, before finally assigns the label of the nearest (cosine similarity) “domain document” to each test document. Simple as it is, our method outperforms all previous methods proving its effectiveness. The reason can be that word embeddings captures the similarity between distinct words (such as “finance” and “economy”), while it is hard for traditional methods. On the order hand, LDA only learns probabilities of words belonging to topics.

Our system does not require a large number of templates because: (1) the iterative question expansion can produce a large number of questions even with a relatively small number of seed questions, as we see in the experiments, (2) multiple entities in the KB share the same predicates. Another advantage is that our system can easily generate updated questions as web is self-updating consistently. In our experiment, we compare with serban-EtAl:2016:P16-1 on 500 random selected triples from Freebase  . Evaluated by 3 human graders, questions generated by our system are significantly better then serban-EtAl:2016:P16-1 on grammaticality and naturalness.

The naturalness score is less than the grammatical score for both methods. It is because naturalness is a more strict metric since a natural question should also be grammatical. Shown in Table 1 , we compare our questions with serban-EtAl:2016:P16-1 where questions in the same line describe the same entity. We can see that our questions are grammatical and natural as these questions are what people usually ask on the web. On the other hand, questions from serban-EtAl:2016:P16-1 are either ungrammatical (such as “who was someone who was involved in the leukemia ?” and “whats the title of a book of the subject of the bible ?”), unnatural (“what 's one of the mountain where can you found in argentina in netflix ?”) or confusing (“who was someone who was involved in the leukemia ?”).

Conclusion	We presented a system to generate natural language questions from a knowledge base. By leveraging rich web information, our system is able to generate domain-relevant questions in wide scope, while human effort is significantly reduced. Evaluated by human graders, questions generated by our system are significantly better than these from serban-EtAl:2016:P16-1 on 500 random-selected triples from Freebase. We also demonstrated generated questions from our in-house KB of power tool domain, which are fluent and domain-relevant in general. Our current system only generates questions without answers, leaving automatic answer mining as our future work.

Experiments	We perform three experiments to evaluate our system qualitatively and quantitatively. In the first experiment, we compare our end-to-end system with the previous state-of-the-art method  on Freebase  , a domain-general KB. In the second experiment, we validate our domain relevance evaluation method on a standard dataset about short document classification. In the final experiment, we run our end-to-end system on a highly specialized in-house KB and present sample results, showing that our system is capable of generating questions from domain specific KBs.

We apply thresholds $t_{rel}$ and $t_{flu}$ for domain relevance and fluency respectively, and filter out questions whose scores are below these thresholds.

The questions collected from the web search engine may not be fluent or domain relevant; especially the domain relevance drops significantly as the iteration goes on. Here we adopt a skip-gram model  and a language model for evaluating the domain relevance and fluency of the expanded questions, respectively. For domain relevance, we take the seed question set as the in-domain data $D_{in}$ , the domain relevance of expanded question $q$ is defined as:  $$\textsc {Rel}(q) = \cos (v(q),v(D_{in}))$$   (Eq. 7)  where $v(\cdot )$ is the document embedding defined as the averaged word embedding within the document. For fluency, we define the averaged language model score as:  $$\textsc {AvgLM}(q) = \frac{\textsc {Lm}(q)}{\textsc {Len}(q)}$$   (Eq. 8)  where $\textsc {Lm}(\cdot )$ is the general-domain language model score (log probability), and $\textsc {Len}(\cdot )$ is the word count.

Question expansion and selection	[t] seed question set $S$ candidate questions $E$ $E \leftarrow S$ $Q \leftarrow S$ $I \leftarrow 0$ len $(Q) > 0$ and $I < I_{max}$ $I = I + 1$ $q_{cur}$ $\leftarrow $ $E$0 .Pop() $E$1 in WebExp $E$2 not $E$3 .contains $E$4 $E$5 .Append( $E$6 ) $E$7 .Push( $E$8 ) Question expansion method Shown in Algorithm "Experiments" , the expanded question set $E$ is initialized as the seed question set (Line 1). In each iteration, an already-obtained question is expanded from web and the retrieved questions are added to $E$ if $E$ does not contain them (Lines 6-10). As there may be a large number of questions generated in the loop, we limit the maximum number of iterations with $I_{max}$ (Line 4).

Knowledge Base	A knowledge base (KB) can be viewed as a directed graph, in which nodes are entities (such as “jigsaw” and “CurveCut”) and edges are relations of entities (such as “performsActivity”). A KB can also be viewed as a list of triples in the format of $\langle $ subject, predicate, object $\rangle $ , where subjects and objects are entities, and predicates are relations.

Question: How many hand-crafted templates did they have to make?
Answer: 269.