Paraphrase Generation from Latent-Variable PCFGs for Semantic Parsing

Abstract: One of the limitations of semantic parsing approaches to open-domain question answering is the lexicosyntactic gap between natural language questions and knowledge base entries -- there are many ways to ask a question, all with the same answer. In this paper we propose to bridge this gap by generating paraphrases of the input question with the goal that at least one of them will be correctly mapped to a knowledge-base query. We introduce a novel grammar model for paraphrase generation that does not require any sentence-aligned paraphrase corpus. Our key idea is to leverage the flexibility and scalability of latent-variable probabilistic context-free grammars to sample paraphrases. We do an extrinsic evaluation of our paraphrases by plugging them into a semantic parser for Freebase. Our evaluation experiments on the WebQuestions benchmark dataset show that the performance of the semantic parser significantly improves over strong baselines.

Semantic parsers map sentences onto logical forms that can be used to query databases BIBREF0 , BIBREF1 , instruct robots BIBREF2 , extract information BIBREF3 , or describe visual scenes BIBREF4 . In this paper we consider the problem of semantically parsing questions into Freebase logical forms for the goal of question answering. Current systems accomplish this by learning task-specific grammars BIBREF5 , strongly-typed CCG grammars BIBREF6 , BIBREF7 , or neural networks without requiring any grammar BIBREF8 . These methods are sensitive to the words used in a question and their word order, making them vulnerable to unseen words and phrases. Furthermore, mismatch between natural language and Freebase makes the problem even harder. For example, Freebase expresses the fact that “Czech is the official language of Czech Republic” (encoded as a graph), whereas to answer a question like “What do people in Czech Republic speak?” one should infer people in Czech Republic refers to Czech Republic and What refers to the language and speak refers to the predicate official language.
We address the above problems by using paraphrases of the original question. Paraphrasing has shown to be promising for semantic parsing BIBREF9 , BIBREF10 , BIBREF11 . We propose a novel framework for paraphrasing using latent-variable PCFGs (L-PCFGs). Earlier approaches to paraphrasing used phrase-based machine translation for text-based QA BIBREF12 , BIBREF13 , or hand annotated grammars for KB-based QA BIBREF10 . We find that phrase-based statistical machine translation (MT) approaches mainly produce lexical paraphrases without much syntactic diversity, whereas our grammar-based approach is capable of producing both lexically and syntactically diverse paraphrases. Unlike MT based approaches, our system does not require aligned parallel paraphrase corpora. In addition we do not require hand annotated grammars for paraphrase generation but instead learn the grammar directly from a large scale question corpus.
The main contributions of this paper are two fold. First, we present an algorithm (§ "Paraphrase Generation Using Grammars" ) to generate paraphrases using latent-variable PCFGs. We use the spectral method of narayan-15 to estimate L-PCFGs on a large scale question treebank. Our grammar model leads to a robust and an efficient system for paraphrase generation in open-domain question answering. While CFGs have been explored for paraphrasing using bilingual parallel corpus BIBREF14 , ours is the first implementation of CFG that uses only monolingual data. Second, we show that generated paraphrases can be used to improve semantic parsing of questions into Freebase logical forms (§ "Semantic Parsing using Paraphrasing" ). We build on a strong baseline of reddylargescale2014 and show that our grammar model competes with MT baseline even without using any parallel paraphrase resources.
Our paraphrase generation algorithm is based on a model in the form of an L-PCFG. L-PCFGs are PCFGs where the nonterminals are refined with latent states that provide some contextual information about each node in a given derivation. L-PCFGs have been used in various ways, most commonly for syntactic parsing BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 .
In our estimation of L-PCFGs, we use the spectral method of narayan-15, instead of using EM, as has been used in the past by matsuzaki-2005 and petrov-2006. The spectral method we use enables the choice of a set of feature functions that indicate the latent states, which proves to be useful in our case. It also leads to sparse grammar estimates and compact models.
The spectral method works by identifying feature functions for “inside” and “outside” trees, and then clusters them into latent states. Then it follows with a maximum likelihood estimation step, that assumes the latent states are represented by clusters obtained through the feature function clustering. For more details about these constructions, we refer the reader to cohen-13 and narayan-15.
The rest of this section describes our paraphrase generation algorithm.
We define our paraphrase generation task as a sampling problem from an L-PCFG $G_{\mathrm {syn}}$ , which is estimated from a large corpus of parsed questions. Once this grammar is estimated, our algorithm follows a pipeline with two major steps.
We first build a word lattice $W_q$ for the input question $q$ . We use the lattice to constrain our paraphrases to a specific choice of words and phrases that can be used. Once this lattice is created, a grammar $G_{\mathrm {syn}}^{\prime }$ is then extracted from $G_{\mathrm {syn}}$ . This grammar is constrained to the lattice.
We experiment with three ways of constructing word lattices: naïve word lattices representing the words from the input question only, word lattices constructed with the Paraphrase Database BIBREF14 and word lattices constructed with a bi-layered L-PCFG, described in § "Bi-Layered L-PCFGs" . For example, Figure 1 shows an example word lattice for the question What language do people in Czech Republic speak? using the lexical and phrasal rules from the PPDB.
Once $G_{\mathrm {syn}}^{\prime }$ is generated, we sample paraphrases of the input question $q$ . These paraphrases are further filtered with a classifier to improve the precision of the generated paraphrases.
We train the L-PCFG $G_{\mathrm {syn}}$ on the Paralex corpus BIBREF9 . Paralex is a large monolingual parallel corpus, containing 18 million pairs of question paraphrases with 2.4M distinct questions in the corpus. It is suitable for our task of generating paraphrases since its large scale makes our model robust for open-domain questions. We construct a treebank by parsing 2.4M distinct questions from Paralex using the BLLIP parser BIBREF25 .
Given the treebank, we use the spectral algorithm of narayan-15 to learn an L-PCFG for constituency parsing to learn $G_{\mathrm {syn}}$ . We follow narayan-15 and use the same feature functions for the inside and outside trees as they use, capturing contextual syntactic information about nonterminals. We refer the reader to narayan-15 for more detailed description of these features. In our experiments, we set the number of latent states to 24.
Once we estimate $G_{\mathrm {syn}}$ from the Paralex corpus, we restrict it for each question to a grammar $G_{\mathrm {syn}}^{\prime }$ by keeping only the rules that could lead to a derivation over the lattice. This step is similar to lexical pruning in standard grammar-based generation process to avoid an intermediate derivation which can never lead to a successful derivation BIBREF26 , BIBREF27 .
Sampling a question from the grammar $G_{\mathrm {syn}}^{\prime }$ is done by recursively sampling nodes in the derivation tree, together with their latent states, in a top-down breadth-first fashion. Sampling from the pruned grammar $G_{\mathrm {syn}}^{\prime }$ raises an issue of oversampling words that are more frequent in the training data. To lessen this problem, we follow a controlled sampling approach where sampling is guided by the word lattice $W_q$ . Once a word $w$ from a path $e$ in $W_q$ is sampled, all other parallel or conflicting paths to $e$ are removed from $W_q$ . For example, generating for the word lattice in Figure 1 , when we sample the word citizens, we drop out the paths “human beings”, “people's”, “the population”, “people” and “members of the public” from $W_q$ and accordingly update the grammar. The controlled sampling ensures that each sampled question uses words from a single start-to-end path in $W_q$ . For example, we could sample a question what is Czech Republic 's language? by sampling words from the path (what, language, do, people 's, in, Czech, Republic, is speaking, ?) in Figure 1 . We repeat this sampling process to generate multiple potential paraphrases.
The resulting generation algorithm has multiple advantages over existing grammar generation methods. First, the sampling from an L-PCFG grammar lessens the lexical ambiguity problem evident in lexicalized grammars such as tree adjoining grammars BIBREF27 and combinatory categorial grammars BIBREF28 . Our grammar is not lexicalized, only unary context-free rules are lexicalized. Second, the top-down sampling restricts the combinatorics inherent to bottom-up search BIBREF29 . Third, we do not restrict the generation by the order information in the input. The lack of order information in the input often raises the high combinatorics in lexicalist approaches BIBREF30 . In our case, however, we use sampling to reduce this problem, and it allows us to produce syntactically diverse questions. And fourth, we impose no constraints on the grammar thereby making it easier to maintain bi-directional (recursive) grammars that can be used both for parsing and for generation BIBREF31 .
As mentioned earlier, one of our lattice types is based on bi-layered PCFGs introduced here.
In their traditional use, the latent states in L-PCFGs aim to capture syntactic information. We introduce here the use of an L-PCFG with two layers of latent states: one layer is intended to capture the usual syntactic information, and the other aims to capture semantic and topical information by using a large set of states with specific feature functions.
To create the bi-layered L-PCFG, we again use the spectral algorithm of narayan-15 to estimate a grammar $G_{\mathrm {par}}$ from the Paralex corpus. We use the word alignment of paraphrase question pairs in Paralex to map inside and outside trees of each nonterminals in the treebank to bag of word features. The number of latent states we use is 1,000.
Once the two feature functions (syntactic in $G_{\mathrm {syn}}$ and semantic in $G_{\mathrm {par}}$ ) are created, each nonterminal in the training treebank is assigned two latent states (cluster identifiers). Figure 2 shows an example annotation of trees for three paraphrase questions from the Paralex corpus. We compute the parameters of the bi-layered L-PCFG $G_{\mathrm {layered}}$ with a simple frequency count maximum likelihood estimate over this annotated treebank. As such, $G_{\mathrm {layered}}$ is a combination of $G_{\mathrm {syn}}$ and $G_{\mathrm {par}}$ , resulting in 24,000 latent states (24 syntactic x 1000 semantic).
Consider an example where we want to generate paraphrases for the question what day is nochebuena. Parsing it with $G_{\mathrm {layered}}$ will lead to the leftmost hybrid structure as shown in Figure 2 . The assignment of the first latent states for each nonterminals ensures that we retrieve the correct syntactic representation of the sentence. Here, however, we are more interested in the second latent states assigned to each nonterminals which capture the paraphrase information of the sentence at various levels. For example, we have a unary lexical rule (NN-*-142 day) indicating that we observe day with NN of the paraphrase type 142. We could use this information to extract unary rules of the form (NN-*-142 $w$ ) in the treebank that will generate words $w$ which are paraphrases to day. Similarly, any node WHNP-*-291 in the treebank will generate paraphrases for what day, SBARQ-*-403, for what day is nochebuena. This way we will be able to generate paraphrases when is nochebuena and when is nochebuena celebrated as they both have SBARQ-*-403 as their roots.
To generate a word lattice $W_q$ for a given question $q$ , we parse $q$ with the bi-layered grammar $G_{\mathrm {layered}}$ . For each rule of the form $X$ - $m_1$ - $m_2 \rightarrow w$ in the bi-layered tree with $X \in {\cal P}$ , $m_1 \in \lbrace  1, \ldots , 24 \rbrace $ , $m_2 \in \lbrace  1, \ldots , 1000 \rbrace $ and $q$0 a word in $q$1 , we extract rules of the form $q$2 - $q$3 - $q$4 from $q$5 such that $q$6 . For each such $q$7 , we add a path $q$8 parallel to $q$9 in the word lattice.
Our sampling algorithm overgenerates paraphrases which are incorrect. To improve its precision, we build a binary classifier to filter the generated paraphrases. We randomly select 100 distinct questions from the Paralex corpus and generate paraphrases using our generation algorithm with various lattice settings. We randomly select 1,000 pairs of input-sampled sentences and manually annotate them as “correct” or “incorrect” paraphrases. We train our classifier on this manually created training data. We follow madnani2012, who used MT metrics for paraphrase identification, and experiment with 8 MT metrics as features for our binary classifier. In addition, we experiment with a binary feature which checks if the sampled paraphrase preserves named entities from the input sentence. We use WEKA BIBREF32 to replicate the classifier of madnani2012 with our new feature. We tune the feature set for our classifier on the development data.
In this section we describe how the paraphrase algorithm is used for converting natural language to Freebase queries. Following reddylargescale2014, we formalize the semantic parsing problem as a graph matching problem, i.e., finding the Freebase subgraph (grounded graph) that is isomorphic to the input question semantic structure (ungrounded graph).
This formulation has a major limitation that can be alleviated by using our paraphrase generation algorithm. Consider the question What language do people in Czech Republic speak?. The ungrounded graph corresponding to this question is shown in Figure 3 . The Freebase grounded graph which results in correct answer is shown in Figure 3 . Note that these two graphs are non-isomorphic making it impossible to derive the correct grounding from the ungrounded graph. In fact, at least 15% of the examples in our development set fail to satisfy isomorphic assumption. In order to address this problem, we use paraphrases of the input question to generate additional ungrounded graphs, with the aim that one of those paraphrases will have a structure isomorphic to the correct grounding. Figure 3 and Figure 3 are two such paraphrases which can be converted to Figure 3 as described in sec:groundedGraphs.
For a given input question, first we build ungrounded graphs from its paraphrases. We convert these graphs to Freebase graphs. To learn this mapping, we rely on manually assembled question-answer pairs. For each training question, we first find the set of oracle grounded graphs—Freebase subgraphs which when executed yield the correct answer—derivable from the question's ungrounded graphs. These oracle graphs are then used to train a structured perceptron model. These steps are discussed in detail below.
We use GraphParser BIBREF7 to convert paraphrases to ungrounded graphs. This conversion involves three steps: 1) parsing the paraphrase using a CCG parser to extract syntactic derivations BIBREF33 , 2) extracting logical forms from the CCG derivations BIBREF34 , and 3) converting the logical forms to an ungrounded graph. The ungrounded graph for the example question and its paraphrases are shown in Figure 3 , Figure 3 and Figure 3 , respectively.
The ungrounded graphs are grounded to Freebase subgraphs by mapping entity nodes, entity-entity edges and entity type nodes in the ungrounded graph to Freebase entities, relations and types, respectively. For example, the graph in Figure 3 can be converted to a Freebase graph in Figure 3 by replacing the entity node Czech Republic with the Freebase entity CzechRepublic, the edge (speak.arg $_2$ , speak.in) between $x$ and Czech Republic with the Freebase relation (location.country.official_language.2, location.country.official_language.1), the type node language with the Freebase type language.human_language, and the target node remains intact. The rest of the nodes, edges and types are grounded to null. In a similar fashion, Figure 3 can be grounded to Figure 3 , but not Figure 3 to Figure 3 . If no paraphrase is isomorphic to the target grounded grounded graph, our grounding fails.
We use a linear model to map ungrounded graphs to grounded ones. The parameters of the model are learned from question-answer pairs. For example, the question What language do people in Czech Republic speak? paired with its answer $\lbrace \textsc {CzechLanguage}\rbrace $ . In line with most work on question answering against Freebase, we do not rely on annotated logical forms associated with the question for training and treat the mapping of a question to its grounded graph as latent.
Let $q$ be a question, let $p$ be a paraphrase, let $u$ be an ungrounded graph for $p$ , and let $g$ be a grounded graph formed by grounding the nodes and edges of $u$ to the knowledge base $\mathcal {K}$ (throughout we use Freebase as the knowledge base). Following reddylargescale2014, we use beam search to find the highest scoring tuple of paraphrase, ungrounded and grounded graphs $(\hat{p}, \hat{u}, \hat{g})$ under the model $\theta \in \mathbb {R}^n$ : $
({\hat{p},\hat{u},\hat{g}}) = \operatornamewithlimits{arg\,max}_{(p,u,g)} \theta \cdot \Phi (p,u,g,q,\mathcal {K})\,,
$ 
where $\Phi (p, u, g, q, \mathcal {K}) \in \mathbb {R}^n$ denotes the features for the tuple of paraphrase, ungrounded and grounded graphs. The feature function has access to the paraphrase, ungrounded and grounded graphs, the original question, as well as to the content of the knowledge base and the denotation $|g|_\mathcal {K}$ (the denotation of a grounded graph is defined as the set of entities or attributes reachable at its target node). See sec:details for the features employed. The model parameters are estimated with the averaged structured perceptron BIBREF35 . Given a training question-answer pair $(q,\mathcal {A})$ , the update is: $
\theta ^{t+1} \leftarrow \theta ^{t} + \Phi (p^+, u^+, g^+, q,
\mathcal {K}) - \Phi (\hat{p}, \hat{u}, \hat{g}, q, \mathcal {K})\,,
$ 
where $({p^+,u^+,g^+})$ denotes the tuple of gold paraphrase, gold ungrounded and grounded graphs for $q$ . Since we do not have direct access to the gold paraphrase and graphs, we instead rely on the set of oracle tuples, $\mathcal {O}_{\mathcal {K}, \mathcal {A}}(q)$ , as a proxy: $
(p^{+},u^{+},{g^{+}}) = \operatornamewithlimits{arg\,max}_{(p,u,g) \in \mathcal {O}_{\mathcal {K},\mathcal {A}}(q)} \theta \cdot \Phi ({p,u,g,q,\mathcal {K}})\,,
$ 
where $\mathcal {O}_{\mathcal {K}, \mathcal {A}}(q)$ is defined as the set of tuples ( $p$ , $u$ , $g$ ) derivable from the question $q$ , whose denotation $|g|_\mathcal {K}$ has minimal $F_1$ -loss against the gold answer $\mathcal {A}$ . We find the oracle graphs for each question a priori by performing beam-search with a very large beam.
Below, we give details on the evaluation dataset and baselines used for comparison. We also describe the model features and provide implementation details.
We evaluate our approach on the WebQuestions dataset BIBREF5 . WebQuestions consists of 5,810 question-answer pairs where questions represents real Google search queries. We use the standard train/test splits, with 3,778 train and 2,032 test questions. For our development experiments we tune the models on held-out data consisting of 30% training questions, while for final testing we use the complete training data. We use average precision (avg P.), average recall (avg R.) and average F $_1$ (avg F $_1$ ) proposed by berantsemantic2013 as evaluation metrics.
We use GraphParser without paraphrases as our baseline. This gives an idea about the impact of using paraphrases.
We compare our paraphrasing models with monolingual machine translation based model for paraphrase generation BIBREF24 , BIBREF36 . In particular, we use Moses BIBREF37 to train a monolingual phrase-based MT system on the Paralex corpus. Finally, we use Moses decoder to generate 10-best distinct paraphrases for the test questions.
For WebQuestions, we use 8 handcrafted part-of-speech patterns (e.g., the pattern (DT)?(JJ.? $\mid $ NN.?){0,2}NN.? matches the noun phrase the big lebowski) to identify candidate named entity mention spans. We use the Stanford CoreNLP caseless tagger for part-of-speech tagging BIBREF38 . For each candidate mention span, we retrieve the top 10 entities according to the Freebase API. We then create a lattice in which the nodes correspond to mention-entity pairs, scored by their Freebase API scores, and the edges encode the fact that no joint assignment of entities to mentions can contain overlapping spans. We take the top 10 paths through the lattice as possible entity disambiguations. For each possibility, we generate $n$ -best paraphrases that contains the entity mention spans. In the end, this process creates a total of $10n$ paraphrases. We generate ungrounded graphs for these paraphrases and treat the final entity disambiguation and paraphrase selection as part of the semantic parsing problem.
We use the features from reddylargescale2014. These include edge alignments and stem overlaps between ungrounded and grounded graphs, and contextual features such as word and grounded relation pairs. In addition to these features, we add two new real-valued features – the paraphrase classifier's score and the entity disambiguation lattice score.
We use beam search to infer the highest scoring graph pair for a question. The search operates over entity-entity edges and entity type nodes of each ungrounded graph. For an entity-entity edge, there are two operations: ground the edge to a Freebase relation, or skip the edge. Similarly, for an entity type node, there are two operations: ground the node to a Freebase type, or skip the node. We use a beam size of 100 in all our experiments.
In this section, we present results from five different systems for our question-answering experiments: original, mt, naive, ppdb and bilayered. First two are baseline systems. Other three systems use paraphrases generated from an L-PCFG grammar. naive uses a word lattice with a single start-to-end path representing the input question itself, ppdb uses a word lattice constructed using the PPDB rules, and bilayered uses bi-layered L-PCFG to build word lattices. Note that naive does not require any parallel resource to train, ppdb requires an external paraphrase database, and bilayered, like mt, needs a parallel corpus with paraphrase pairs. We tune our classifier features and GraphParser features on the development data. We use the best setting from tuning for evaluation on the test data.
We described a grammar method to generate paraphrases for questions, and applied it to a question answering system based on semantic parsing. We showed that using paraphrases for a question answering system is a useful way to improve its performance. Our method is rather generic and can be applied to any question answering system.
The authors would like to thank Nitin Madnani for his help with the implementation of the paraphrase classifier. We would like to thank our anonymous reviewers for their insightful comments. This research was supported by an EPSRC grant (EP/L02411X/1), the H2020 project SUMMA (under grant agreement 688139), and a Google PhD Fellowship for the second author.

Figure and table captions:
Figure 1: An example word lattice for the question What language do people in Czech Republic speak? using the lexical and phrasal rules from the PPDB.
Figure 2: Trees used for bi-layered L-PCFG training. The questions what day is nochebuena, when is nochebuena and when is nochebuena celebrated are paraphrases from the Paralex corpus. Each nonterminal is decorated with a syntactic label and two identifiers, e.g., for WP-7-254, WP is the syntactic label assigned by the BLLIP parser, 7 is the syntactic latent state, and 254 is the semantic latent state.
Figure 3: Ungrounded graphs for an input question and its paraphrases along with its correct grounded graph. The green squares indicate NL or Freebase entities, the yellow rectangles indicate unary NL predicates or Freebase types, the circles indicate NL or Freebase events, the edge labels indicate binary NL predicates or Freebase relations, and the red diamonds attach to the entity of interest (the answer to the question).
Table 1: Oracle statistics and results on the WebQuestions development set.
Table 2: Results on WebQuestions test dataset.

Questions about this paper:
How many paraphrases are generated per question?

Answers:
10*n paraphrases, where n depends on the number of paraphrases that contain the entity mention spans