Venue: NAACL 2016

# Probabilistic Models for Learning a Semantic Parser Lexicon

Jayant Krishnamurthy

Allen Institute for Artificial Intelligence

2157 N. Northlake Way, Suite 110

Seattle, WA 98103

jayantk@allenai.org

###### Abstract

We introduce several probabilistic models for learning the lexicon of a semantic parser. Lexicon learning is the first step of training a semantic parser for a new application domain and the quality of the learned lexicon significantly affects both the accuracy and efficiency of the final semantic parser. Existing work on lexicon learning has focused on heuristic methods that lack convergence guarantees and require significant human input in the form of lexicon templates or annotated logical forms. In contrast, our probabilistic models are trained directly from question/answer pairs using EM and our simplest model has a concave objective that guarantees convergence to a global optimum. An experimental evaluation on a set of 4th grade science questions demonstrates that our models improve semantic parser accuracy (35-70% error reduction) and efficiency (4-25x more sentences per second) relative to prior work despite using less human input. Our models also obtain competitive results on geo880 without any dataset-specific engineering.

## 1 Introduction

Semantic parsing has recently gained popularity as a technique for mapping from natural language to a formal meaning representation language, e.g., in order to answer questions against a database Zelle and Mooney (1993); Zettlemoyer and Collins (2005). In order to train a semantic parser, one must first provide a _lexicon_, which is a mapping from words in the language to statements in the meaning representation language. This mapping defines the grammar of the parser and thereby determines the set of meaning representations that can be produced for any given sentence. Therefore, a good lexicon is necessary to achieve both high accuracy and parsing speed. However, the lexicon is unobserved in real semantic parsing applications, leading us to ask: _how do we learn a lexicon for a semantic parser?_

This paper presents several novel probabilistic models for learning a semantic parser lexicon. Existing lexicon learning algorithms are heuristic in nature and require either annotated logical forms or manually-specified lexicon entry templates during training. In contrast, our models do not require such templates and can be trained from question/answer pairs and other forms of weak supervision. Training consists of optimizing an objective function with Expectation Maximization (EM), thereby guaran

Figure 1: Parse tree of a training example and the lexicon entries derived from it.



teeing convergence to a local optimum. Furthermore, the objective function for our simplest model is _concave_, guaranteeing convergence to a global optimum. Our approach generates a probabilistic context-free grammar that represents the space of correct semantic parses for each question; once trained, our approach derives lexicon entries from the most likely parse of each question (Figure 1).

We present an experimental evaluation of our lexicon learning models on a data set of food chain questions from a 4th grade science domain. These questions concern relations between organisms in an ecosystem and have challenging lexical diversity and question length. Our models improve semantic parser accuracy (35-70% error reduction) over prior work despite using less human input. Furthermore, our best model produces a lexicon that contains 40x fewer entries than the most accurate baseline, resulting in a semantic parser that is 4x faster. Our models also obtain competitive results on geo880 without any dataset-specific engineering.

## 2 Prior Work

Work on lexicon learning falls into two categories:

**Pipelined approaches** build a lexicon before training the parser, either by manually defining it Lee et al. (2014); Angeli et al. (2012) or by using a collection of heuristics. The heuristics often take the form of _lexicon templates_, which are rules that create lexicon entries by pattern-matching training examples Liang et al. (2011); Krishnamurthy and Mitchell (2012); Krishnamurthy and Kollar (2013). These approaches require new lexicon templates for each application. More complex heuristic algorithms have been proposed based on word alignments Wong and Mooney (2006); Wong and Mooney (2007) or common substructures in the meaning representation Chen and Mooney (2011); these algorithms all require annotated logical forms.

**Joint approaches** simultaneously learn a lexicon and the parameters of a semantic parser. Typically, these algorithms use lexicon templates to generate a set of lexicon entries for each example, then heuristically select a subset of these entries to include in the global lexicon while training the parser Zettlemoyer and Collins (2005); Artzi and Zettlemoyer (2013b); Artzi et al. (2014). UBL takes a different approach that performs top-down, iterative splits of an annotated logical form for each training example Kwiatkowski et al. (2010); Kwiatkowski et al. (2011). Artzi et al. (2015) combine templates with top-down splitting. The heuristic search performed by these algorithms can be difficult to control and we empirically found that these algorithms often selected overly-specific lexicon entries (see Section 4.4).

Other work has avoided the lexicon learning problem altogether by searching over all possible meaning representations Kate and Mooney (2006); Clarke et al. (2010); Goldwasser et al. (2011); Berant and Liang (2014); Pasupat and Liang (2015); Reddy et al. (2014). The challenge of this approach is that the space of meaning representations for a sentence can be very large, making parsing less efficient and learning more difficult. A practical compromise is to combine a (possibly minimal) lexicon with flexible parsing operations Liang et al. (2011); Zettlemoyer and Collins (2007); Poon (2013); Parikh et al. (2015).

Our lexicon learning models are closely related to machine translation word alignment models Brown et al. (1993) - our key insight is that _lexicon learning is equivalent to word alignment where the tokenization of one of the sentences is unobserved_. Thus, our models simultaneously "tokenize" the logical form - using a splitting process similar to UBL - and align the resulting logical form "tokens" to words.

## 3 Probabilistic Models for Lexicon Learning

This section describes our lexicon learning models. For concreteness, we focus on learning a lexicon for a Combinatory Categorial Grammar (CCG) semantic parser with lambda calculus logical forms as the meaning representation; however, our models are applicable to other semantic parsing formalisms and meaning representation languages.

Our models learn a CCG lexicon from a data set of question/label pairs \(\{(\textbf{w}^{i},L^{i})\}_{i=1}^{n}\). Each question is a sequence of words, \(\textbf{w}^{i}=[w_{1}^{i},w_{2}^{i},...]\), and each label is a _set_ of logical forms, \(L^{i}=\{\ell_{1}^{i},...\}\). Labeling each question with a set of logical forms generalizes many weak supervision settings, including ambiguous supervision Kate and Mooney (2007) and ques 

tion/answer pairs (Liang et al., 2011).1 The output of learning is a collection of lexicon entries \(w:=C:\ell\) mapping word \(w\) to syntactic category \(C\) and logical form \(\ell\).

Footnote 1: In the second case, the set \(L\) can be generated by enumerating logical forms and evaluating each one to determine if it produces the correct answer. See Section 5 for a discussion of the benefits and limitations of this process.

Our models are generative models of questions given a label, \(P({\bf w}|L)\). The key component of each model is a probabilistic context-free grammar (PCFG) over _correct_ logical form derivations. A parse tree in this grammar simultaneously represents (1) the choice of a logical form \(\ell\in L\), (2) the way \(\ell\) is constructed from smaller parts, and (3) the alignment between these parts and words in the question. Training each model amounts to learning the rule probabilities of this grammar, including which logical forms are likely to generate which words. Parsing an example with the trained grammar produces an alignment between words and logical forms that is used to construct a lexicon.

### Logical Form Derivation Grammar

The logical form derivation grammar is a PCFG constructed to represent the set of correct logical form derivations - i.e., correct semantic parses - of each training example. The grammar's nonterminals are logical forms and its binary production rules represent ways that pairs of logical forms can combine in the semantic parser. The grammar's terminals are words and its terminal production rules represent lexicon entries. The complete grammar is a union of many smaller grammars, each of which is constructed to represent the logical form derivations of a single example. Figure 2 shows a training example and a portion of the grammar generated for it, and Figure 1 shows a parse tree in the grammar.

Our algorithm for constructing the PCFG for a training example \(({\bf w},L)\) uses a top-down approach that iteratively splits logical forms in \(L\). Assume we are given a procedure split\((f)\) that outputs a list of ways to split \(f\) into a pair of logical forms \((g,h)\). Grammar generation performs the following steps:

1. **Model weak supervision.** Add \(L\) to the grammar as a nonterminal and add a unary rule \(L\to\ell\) for all \(\ell\in L\).
2. **Enumerate logical form splits.** For all \(\ell\in L\), perform a depth-first search over logical forms starting at \(\ell\). To explore a logical form \(f\) during the search, use split\((f)\) to produce a collection of \(g,h\) pairs. For each \(g,h\) pair, add the binary rules \(f\to g~{}h\) and \(f\to h~{}g\) to the grammar, then add \(g\) and \(h\) to the search queue for later exploration.
3. **Create lexicon entries.** Add a terminal rule \(f\to w\) to \(G\) for every word in the question, \(w\in{\bf w}\), and logical form \(f\) encountered during the search above.
4. **Allow word skipping.** Add a special skip nonterminal, along with the rules \(f\to f\)skip, \(f\rightarrow\textsc{skip}~{}f\) and skip\(\to w\) for all logical forms \(f\) and words \(w\in{\bf w}\).

The split procedure required above depends on the meaning representation language, but is application-independent. For our lambda calculus representation, split\((f)\) returns a list of logical forms \(g,h\) such that \(f=g(h)\). We use similar constraints as Kwiatkowski et al. (2011) to keep the number of splits manageable. Note that split could also include composition by returning \(g,h\)

Figure 2: Training example (top) and several of the rules in its logical form derivation grammar (bottom).

 

pairs such that \(f=\lambda x.g(h(x))\); however, we did not explore this possibility in this paper.

An important property of this grammar is that it _excludes many logical form derivations that cannot lead to the label_. For example, the grammar in Figure 2 does not let us apply \(\lambda x.{\tt eats}(x,{\tt bass})\) to Bass - even though this operation would be permitted by CCG - because there is no way to reach the label \(L\) from the result \({\tt eats}({\tt bass},{\tt bass})\). This property reduces the number of possible parses of a question relative to a CCG parser with the same lexicon entries, making parsing more efficient.

The logical form derivation grammar \(G\) is constructed by applying the above process to every example in the data set. Let \(P(t|L;\theta)\) denote the probability of generating tree \(t\) from \(G\) given \({\tt root}(t)=L\) and parameters \(\theta\). This probability factors into a product of production rule probabilities:

\[P(t|L;\theta) =\prod_{(f\to g~{}h)\in t}P(f\to g~{}h;\theta)\times\] \[\prod_{(f\to w)\in t}P(f\to w;\theta)\]

In the above equation, \(P(f\to g~{}h;\theta)\) and \(P(f\to w;\theta)\) represent the conditional probability of selecting a production rule given the nonterminal \(f\). We use \(P({\bf w},t|L;\theta)\) to denote \(P(t|L;\theta)\) where the terminals of \(t\) are equal to the question \({\bf w}\). In the following, sums over trees \(t\) are implicitly over all trees permitted by \(G\).

### Independent Model

The independent model assumes that each word \(w_{j}\) of a question \({\bf w}\) is generated independently from a parse tree \(t\) chosen uniformly at random given the label \(L\). This simple model allows two words in the same question to be generated by different trees. The probability of a question given a label is:

\[P({\bf w}|L;\theta)=\prod_{j=1}^{|{\bf w}|}\sum_{f}P(f\to w_{j};\theta)\#(f, j,L,|{\bf w}|)\]

The final term \(\#(f,j,L,|{\bf w}|)\) is the fraction of trees with root \(L\) and \(|{\bf w}|\) terminals where the \(j\)th terminal symbol is generated by nonterminal \(f\). This term appears due to the assumption that trees are drawn uniformly at random. The parameters \(\theta\) of this model are the terminal production rule probabilities, which are modeled as a conditional probability table: \(P(f\to w;\theta)=\theta_{f,w}\) where \(\sum_{w}\theta_{f,w}=1\).

The independent model is a generalization of IBM Model 1 (Brown et al., 1993) to the lexicon learning problem, and - like IBM Model 1 - its loglikelihood function is _concave_ (see Appendix A). Therefore, the EM algorithm will converge to a global optimum of the data loglikelihood under this model.

### Coupled Model

The coupled model generates the entire question \({\bf w}\) from a single parse tree \(t\) that is generated given \(L\). This model removes the previous model's naive assumption that each word is generated independently. The probability of a question given a label under this model is:

\[P({\bf w}|L;\theta)=\sum_{t}P({\bf w},t|L;\theta)\]

Theoretically, we could learn both of the production rule distributions that compose \(P({\bf w},t|L;\theta)\) in this formulation. However, in practice, the large number of nonterminals makes it challenging to learn a conditional probability table for the binary production rules. Therefore, we again assume the trees are drawn uniformly at random and only learn a conditional probability table for the terminal production rules.

### Coupled Loglinear Model

The coupled loglinear model replaces the conditional probability tables of the coupled model with loglinear models. Loglinear models can share parameters across different - but intuitively similar - production rules. For example, both \(\lambda x.{\tt eats}(x,{\tt bass})\) and \(\lambda x.{\tt eats}(x,{\tt frog})\) should have similar distributions over production rules when bass is appropriately replaced by frog. We can produce this effect with loglinear models by assigning similar feature vectors to these rules.

This model uses _three_ locally-normalized loglinear models to parameterize the distribution over production rules. First, a rule model decides whether or not to apply a terminal production rule. Next, given 

this model's response, a second loglinear model decides which of the chosen kind of rules to apply. This approach ensures that each nonterminal symbol has a proper conditional probability distribution over rules. The production rule distributions are parameterized as:

\[P(f\to w;\theta) =\frac{\exp(\theta_{r}^{T}\phi_{r}(f)+\theta_{t}^{T}\phi_{t}(f,w))}{Z_{ r}Z_{t}(f)}\] \[P(f\to g~{}h;\theta) =\frac{\exp(\theta_{n}^{T}\phi_{n}(f,g,h))}{Z_{r}Z_{n}(f)}\]

In the above equation, the \(r,t\) and \(n\) subscripts indicate terms of the rule, terminal and nonterminal models, respectively. The \(Z_{r},Z_{t}(f)\) and \(Z_{n}(f)\) terms denote the partition functions of each model, and \(\phi_{r},\phi_{n}\) and \(\phi_{t}\) are functions mapping nonterminals and production rules to feature vectors. We use indicator features for logical form patterns, where each pattern is produced by replacing all of a logical form's subexpressions below a certain depth with their types.

### Training with Expectation Maximization

We train all three models by maximizing data log-likelihood with EM Dempster et al. (1977). Training the independent model is equivalent to training a mixture of multinomials where each word of each question has its own prior over cluster assignments. Let \(\theta^{m}\) represent the model parameters on the \(m\)th training iteration. The E-step calculates expected count of each terminal production rule:

\[E_{f,w}\leftarrow\sum_{i,j:w_{j}^{i}=w}P(f\to w_{j}^{i};\theta^{ m})\#(f,j,L^{i},|\textbf{w}^{i}|)\]

The term \(\#(f,j,L^{i},|\textbf{w}^{i}|)\) - representing the fraction of trees where nonterminal \(f\) generates the \(j\)th word of question \(i\) - can be calculated by parsing each example once using the inside/outside algorithm. The M-step re-estimates the terminal production rule probabilities using these expected counts:

\[\theta_{f,w}^{m+1}\leftarrow\frac{E_{f,w}}{\sum_{w^{\prime}}E_{ f,w^{\prime}}}\]

Training the coupled models is a standard application of EM to learning the parameters of a latent probabilistic CFG. The E-step calculates the expected number of occurrences of each production rule in each example:

\[E_{f,g,h} \leftarrow\sum_{i,t}\#(f\to g~{}h,t)P(t|\textbf{w}^{i},L^{ i};\theta^{m})\] \[E_{f,w} \leftarrow\sum_{i,t}\#(f\to w,t)P(t|\textbf{w}^{i},L^{i}; \theta^{m})\]

In the above equation, the function \(\#(f\to w;t)\) returns the number of occurrences of \(f\to w\) in \(t\). These expected counts can be computed efficiently using the inside/outside algorithm.2 The M-step of the coupled model is the same as that of the independent model above. The M-step of the coupled log-linear model solves an optimization problem to fit the loglinear models to the computed expectations Berg-Kirkpatrick et al. (2010):

Footnote 2: A further efficiency improvement is to note that, when parsing an example, it is sufficient to use the subset of \(G\) that was generated for it.

\[\theta^{m+1}\leftarrow\operatorname*{arg\,max}_{\theta} \sum_{f,w}E_{f,w}\log P(f\to w;\theta)+\] \[\sum_{f,g,h}E_{f,g,h}\log P(f\to g~{}h;\theta)\]

This problem factors into three separate optimization problems: a binary logistic regression for the rule model, and estimating two conditional distributions over nonterminal and terminal production rules. We use L-BFGS to solve these problems.

### Producing a Lexicon

Given parameters \(\theta\) and a data set \(\{(\mathbf{w}^{i},L^{i})\}_{i=1}^{n}\), we produce a CCG lexicon from the terminal production rules of the most probable parse of each example. First, we parse each question \(\mathbf{w}^{i}\) conditioning on the parse tree root being \(L^{i}\). Second, we generate lexicon entries from the highest scoring parse tree for each example. We identify the nonterminal \(f\) that generates each word \(w\in\mathbf{w}\) and, if \(f\neq\textsc{skip}\), create a lexicon entry \(w:=C:f\). As in previous work, we derive the syntactic category \(C\) from the semantic type of the logical form Kwiatkowski et 

al., 2011). The argument directions of \(C\) are determined by walking up the tree and noting the relative position of each of its arguments. Figure 1 shows an example of a predicted parse tree and the lexicon entries generated from it.

## 4 Evaluation

We compare our lexicon learning models against several baselines on two data sets: foodchains, containing 4th grade science food chain questions, and geo880, containing geography questions. These data sets each present different challenges for lexicon learning: foodchains has more difficult language - long questions and more lexical variation - while geo880 has more complex logical forms. Our results demonstrate that our models perform better than several baselines with difficult language while simultaneously performing reasonably well with complex logical forms.

Code, data and other supplementary material for this paper is available at [http://www.allenai.org/paper-appendix/naacl2016-lexicon](http://www.allenai.org/paper-appendix/naacl2016-lexicon).

### Data

We collected a new data set, foodchains, that contains 774 food chain questions designed to imitate actual questions from the New York State Grade 4 Regents Exam. Each example in the data set consists of a natural language question and a food chain, which is a list of organisms that eat each other. The questions are multiple choice and the answer options are either animals from the food chain or a direction of change, e.g., "increase." Each question also has a logical form annotated by the first author, which is necessary to train some of the baseline systems. The denotation of each predicate - and therefore logical form - is a deterministic function of the food chain. Figure 3 shows some examples from this data set.

foodchains was created using Mechanical Turk. We first manually created 25 distinct food chains of various lengths containing different organisms and a set of question templates - questions with a blank - based on real Regents questions. In the first task, Turk workers were shown a randomly-selected food chain, question template, and answer, and were asked to complete the question by filling in the blank. In the second task, workers paraphrased questions from the first task, thereby eliminating the templated structure and increasing lexical variation (similar to Wang et al., (2015)). In the third task, a worker validated each question by answering it.

Statistics of foodchains are presented in Table 1 alongside corresponding statistics of geo880 (Zelle and Mooney, 1996; Tang and Mooney, 2001; Zettlemoyer and Collins, 2005). foodchains differs from geo880 in two significant and interesting ways. First, although the data set contains relatively few predicates, there are many ways to reference each predicate - for example, consider the diversity in references to decrease in Figure 3. Second, the questions are long but contain many uninformative

\begin{table}
\begin{tabular}{l r r} \hline \hline  & foodchains & geo880 \\ \hline Examples & 774 & 880 \\ Word types & 446 & 279 \\ Word types w/o entity names & 357 & 157 \\ Tokens per question & 11.6 & 7.56 \\ Predicates in ontology & 15 & 38 \\ Constants per logical form & 4.0 & 7.7 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Data statistics for foodchains and geo880.

 

words that can safely be ignored by the parser.

### Methodology

We compare lexicon learning algorithms by performing an end-to-end evaluation, measuring the question answering accuracy of a CCG semantic parser trained with the learned lexicon. The parser has a rich set of features, including lexicon entry features, dependency features, and dependency distance features. The parser is also permitted to skip words in the question for a learned per-word cost. We train the parser by optimizing data loglikelihood with 100 epochs of stochastic gradient descent.

All experiments on foodchains are performed using 5-fold cross validation. All questions about a single food chain appear in the same fold, ensuring that the questions in the held-out fold reference unseen food chains.

### Comparing Probabilistic Models

Our initial experiment compares the three probabilistic models proposed in Section 3 on foodchains. We generated three lexicons by training each model using 10 iterations of EM. We used a smoothing parameter of 0.1 when estimating conditional probability tables, and an L2 regularization parameter of \(10^{-6}\) when estimating loglinear models. We also initialized the coupled model with the optimum of the independent model. All of these models are trained without labeled logical forms, instead using an automatically enumerated set of logical forms that evaluate to the correct answer.

Table 2 presents the result of this evaluation. All three models perform roughly similarly, with the coupled loglinear model slightly outperforming the others. The competitive performance of the independent model is interesting because its concave objective function is easy to optimize. The remaining experiments compare against the coupled loglinear model, which we dub pal, short for "Probabilistic Alignments for Lexicon learning."

### Lexicon Learning Baselines

Our second experiment compares pal with four baseline lexicon learning algorithms. The first baseline, POS, defines a set of lexicon entries for each word in the training set based on its part-of-speech tag Liang et al. (2011). We iteratively developed these templates to cover the data set, and the lexicon generated by these templates can correctly parse \(96\%\) of the examples in foodchains. The remaining three baselines, ZC2007 Zettlemoyer and Collins (2007), UBL Kwiatkowski et al. (2011), and ADP2014 Artzi et al. (2014), are joint lexicon and parameter learning algorithms. We used the UW SPF Artzi and Zettlemoyer (2013a) implementations of UBL and ZC2007. For these two models, we trained our parser using the learned lexicon to ensure consistency of implementation details.3 We initialized UBL's parameters using GIZA++ Och and Ney (2003) as in the original paper. The lexicon entry templates for ZC2007 and ADP2014 are derived from the POS templates to ensure coverage of the questions; these algorithms allow each template to apply to 1-4 word phrases.

Footnote 3: We also postprocessed the lexicon entries to improve performance, for example, by removing lexicon entries that skip words. In both cases, our parser was more accurate than the UW SPF parser.

Table 3 compares the accuracy of semantic parsers trained with these baseline approaches to pal. Our model outperforms all of the baselines, beating the most accurat

\begin{table}
\begin{tabular}{l c} \hline \hline Model & Accuracy \\ \hline Independent Model (§3.2) & 78.7\% \\ Coupled Model (§3.3) & 79.0\% \\ Coupled Loglinear Model (§3.4) & **81.7\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of semantic parser accuracy on foodchains when trained using our three proposed probabilistic models for lexicon learning.

\begin{table}
\begin{tabular}{l l r r r} \hline \hline  & Logical & Lexicon & \multicolumn{1}{c}{pal} \\ Model & forms? & Templates? & Accuracy & Err. Red. \\ \hline pal & **No** & **No** & **81.7\%** & – \\ \hline POS & **No** & Yes & 70.5\% & 38.0\% \\ UBL & Yes & **No** & 40.6\% & 69.1\% \\ ZC2007 & Yes & Yes & 49.4\% & 63.8\% \\ ADP2014 & **No** & Yes & 32.4\% & 72.9\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Semantic parser accuracy comparison for several lexicon learning algorithms on foodchains. The middle two columns note the human input required by each algorithm and the final column notes the relative error reduction of pal over each baseline.

 

than 10 points. Note that all of these baselines also use more human input than our model, either in the form of lexicon templates or logical forms. Table 4 compares the average number of lexicon entries and semantic parser speed of the baselines with our models. pal produces the most compact lexicon and second fastest parser, which is 4x faster than POS, the baseline with the highest accuracy. The correlation between lexicon size and parse time is imperfect due to word frequency and co-occurrence effects.

The three joint lexicon and parameter learning algorithms perform poorly on our data set for two reasons. First, the long question length increases the difficulty of finding good lexicon entries. The algorithms with lexicon templates were frequently unable to find a correct parse for long questions, even with a large beam size - we ran ADP2014 with a beam size of 10000. (Note that POS does not suffer from this problem because it only generates lexicon entries for a few parts of speech, so most of the words in a question are ignored by default.) Second, these algorithms' discriminative objectives inherently prefer lexicon entries with highly specific word sequences. This preference interacts poorly with the uninformative words in the data set, leading these algorithms to produce many lexicon entries for long phrases that do not generalize well. The lexicon sizes in Table 4 are suggestive of this problem for ZC2007 and UBL; ADP2014 also has this problem, but its voting mechanism prunes lexicon entries much more aggressively. We tried to solve this problem for ZC2007 and ADP2014 by restricting lexicon templates to apply to at most 1 word, but this change actually reduced accuracy. An investigation of this phenomenon found that reducing the length of the templates made it even more difficult for these models to find correct parses for long questions.

In contrast to these baselines, our models do not suffer from either of these problems because the logical form derivation grammar restricts the search to _correct_ derivations and our generative objective prefers frequently-occurring lexicon entries. Our models actually consider a _larger_ space of possible lexicon entries than ZC2007 and ADP2014 with 1 word templates, yet find better lexicon entries.

### Geo880 Evaluation

We performed an additional evaluation on geo880 to demonstrate that our models can work with more complex logical forms. geo880 is a good data set for this evaluation because its logical forms contain, on average, about twice as many constants as foodchains. We applied pal to generate a lexicon for this data set using its included logical form labels, then trained a CCG semantic parser with this lexicon. Table 5 compares the accuracy of this parser with previous CCG lexicon learning results on this data set using the standard 600/280 train/test split. The comparison to pal is inexact because both prior systems use CCG parsers with special extensions that improve performance on this data set. UBL uses factored lexicon entries that generalize better to certain infrequent entries, and ZC2007 includes relaxed parsing operators that fix common parsing errors. Examining the errors made by our parser, we found many cases where these extensions would help. We therefore trained another CCG parser using a lexicon generated by postprocessing pal's lexicon to include factored lexicon entries. This parser achieves an accuracy close to previous work.

\begin{table}
\begin{tabular}{l r r} \hline \hline Model & Lexicon Size & Parse Time (ms) \\ \hline POS & 4,282 & 7.8 \\ UBL & 3,356 & 14.9 \\ ZC2007 & 2,949 & 43.4 \\ ADP2014 & 401 & 1.1 \\ \hline Independent Model & 410 & 6.7 \\ Coupled Model & 318 & 3.1 \\ pal & 184 & 1.8 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Lexicon size excluding entity names and parse time per question on foodchains averaged across folds per model. Our models produce smaller lexicons that lead to faster semantic parsers.

\begin{table}
\begin{tabular}{l l} \hline \hline Model & Accuracy \\ \hline UBL [2] & 88.6\% \\ ZC2007 [10] & 86.1\% \\ \hline pal & 81.8\% \\ w/ factored lexicon & 85.4\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Logical form accuracy on geo880 compared to previously reported CCG parsing results. Both UBL and ZC2007 use special CCG extensions to improve performance; adding one of these to pal brings its accuracy near that of these systems.

 



## 5 Discussion

We introduce several probabilistic models for learning a semantic parser lexicon that can be trained from question/answer pairs and other forms of weak supervision. Our experimental results demonstrate that our models improve semantic parser accuracy and efficiency relative to prior work on data sets with more challenging language, despite using less human input. Furthermore, we find that our independent model is nearly as effective as more complex models, but has a concave objective function that guarantees training converges to a global optimum.

A possible complaint about our approach is that, when training from question/answer pairs, it is not practical to enumerate all logical forms that produce the correct answer. We believe this compliant is misguided because enumerating logical forms is unavoidable in the question/answer setting. _Every_ algorithm uses an enumerate-and-test approach to identify correct logical forms; this process occurs in the gradient computation of semantic parser training and in template-based lexicon learning algorithms such as ADP2014. The critical question is not _whether_ enumeration is used, but rather _how_ logical forms are enumerated. Many strategies are possible and different strategies are likely to be effective on different data sets. In fact, the failure of template-based algorithms on foodchains is largely a failure of their template-guided enumeration strategy to find correct logical forms. Choosing an enumeration strategy and related questions - e.g., does semantic parser parameter learning affect the enumeration? - are empirical questions that must be decided on a task-specific basis.

A recent trend in semantic parsing has been to avoid lexicon learning, instead directly searching the space of all possible logical forms. However, we think that lexicon learning still serves a valuable purpose. Fundamentally, a lexicon constrains the space of logical forms searched by a semantic parser; these constraints improve efficiency and can improve accuracy as long as they do not exclude correct logical forms. Thus, a promising approach to building a semantic parser is to automatically learn a lexicon using one of our models, then (perhaps) manually specify new parsing operations to correct any problems with the learned lexicon. We plan to apply this approach in the future to construct semantic parsers for more challenging tasks.

#### Acknowledgments

We gratefully acknowledge Oyvind Tafjord, Matt Gardner, Peter Turney, Oren Etzioni and the anonymous reviewers for their helpful comments.

## References

* Angeli et al. (2012) Gabor Angeli, Christopher D. Manning, and Daniel Jurafsky. 2012. Parsing time: Learning to interpret time expressions. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Artzi and Zettlemoyer (2013a) Yoav Artzi and Luke Zettlemoyer. 2013a. UW SPF: The University of Washington Semantic Parsing Framework.
* Artzi and Zettlemoyer (2013b) Yoav Artzi and Luke Zettlemoyer. 2013b. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics_.
* Artzi et al. (2014) Yoav Artzi, Dipanjan Das, and Slav Petrov. 2014. Learning compact lexicons for CCG semantic parsing. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_.
* Artzi et al. (2015) Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015. Broad-coverage CCG semantic parsing with AMR. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Berant and Liang (2014) Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_.
* Berg-Kirkpatrick et al. (2010) Taylor Berg-Kirkpatrick, Alexandre Bouchard-Cote, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_.
* Brown et al. (1993) Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. _Computational Linguistics_.
* Chen and Mooney (2011) David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In _Proceedings of the 25th AAAI Conference on Artificial Intelligence_.
* Clarke et al. (2010) James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from

the world's response. In _Proceedings of the Fourteenth Conference on Computational Natural Language Learning_.
* Dempster et al. (1977) Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. _Journal of the royal statistical society. Series B (methodological)_, pages 1-38.
* Goldwasser et al. (2011) Dan Goldwasser, Roi Reichart, James Clarke, and Dan Roth. 2011. Confidence driven unsupervised semantic parsing. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics_.
* Kate and Mooney (2006) Rohit J. Kate and Raymond J. Mooney. 2006. Using string-kernels for learning semantic parsers. In _21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_.
* Kate and Mooney (2007) Rohit J. Kate and Raymond J. Mooney. 2007. Learning language semantics from ambiguous supervision. In _Proceedings of the 22nd Conference on Artificial Intelligence_.
* Volume 1_.
* Krishnamurthy and Mitchell (2012) Jayant Krishnamurthy and Tom M. Mitchell. 2012. Weakly supervised training of semantic parsers. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_.
* Kwiatkowski et al. (2010) Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_.
* Kwiatkowski et al. (2011) Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2011. Lexical generalization in CCG grammar induction for semantic parsing. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Lee et al. (2014) Kenton Lee, Yoav Artzi, Jesse Dodge, and Luke Zettlemoyer. 2014. Context-dependent semantic parsing for time expressions. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Liang et al. (2011) Percy Liang, Michael I. Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In _Proceedings of the Association for Computational Linguistics_.
* Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational Linguistics_.
* Parikh et al. (2015) Ankur P. Parikh, Hoifung Poon, and Kristina Toutanova. 2015. Grounded semantic parsing for complex knowledge extraction. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Poon (2013) Hoifung Poon. 2013. Grounded unsupervised semantic parsing. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Reddy et al. (2014) Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. _Transactions of the Association for Computational Linguistics_.
* Tang and Mooney (2001) Lappoon R. Tang and Raymond J. Mooney. 2001. Ussing multiple clause constructors in inductive logic programming for semantic parsing. In _Proceedings of the 12th European Conference on Machine Learning_.
* Wang et al. (2015) Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_.
* Wong and Mooney (2006) Yuk Wah Wong and Raymond J. Mooney. 2006. Learning for semantic parsing with statistical machine translation. In _Proceedings of the Human Language Technology Conference of the NAACL_.
* Wong and Mooney (2007) Yuk Wah Wong and Raymond J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics_.
* Zelle and Mooney (1993) John M. Zelle and Raymond J. Mooney. 1993. Learning semantic grammars with constructive inductive logic programming. In _Proceedings of the 11th National Conference on Artificial Intelligence_.
* Zelle and Mooney (1996) John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In _Proceedings of the Thirteenth National Conference on Artificial Intelligence_.
* Zettlemoyer and Collins (2005) Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: structured classification with probabilistic categorial grammars. In _UAI '05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence_.
* Zettlemoyer and Collins (2007) Luke S. Zettlemoyer and Michael Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In _Proceedings of the 2007 Joint Conference 

on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_.

## Appendix A: Proof of Concavity

The loglikelihood \(O(\theta)\) of the independent model is:

\[O(\theta)= \sum_{i=1}^{n}\log P(\mathbf{w}^{i}|L^{i};\theta)\] \[= \sum_{i=1}^{n}\sum_{j=1}^{|\mathbf{w}^{i}|}\log\sum_{f}\theta_{f, w_{j}^{i}}\#(f,j,L^{i},|\mathbf{w}^{i}|)\]

Each \(\log\) term above is concave in \(\theta\) because \(\log\) is a concave function applied to an affine function of \(\theta\). (Note that the \(\#(f,j,L^{i},|\mathbf{w}^{i}|)\) terms do not depend on \(\theta\).) Finally, \(O(\theta)\) is concave because it is a sum of concave functions.