Venue: NAACL 2019

# Iterative Search for Weakly Supervised Semantic Parsing

Pradeep Dasigi\({}^{\diamondsuit}\), Matt Gardner\({}^{\blacktriangledown}\), Shikhar Murty\({}^{\diamondsuit}\), Luke Zettlemoyer\({}^{\clubsuit}\), and Eduard Hovy\({}^{\spadesuit}\)

\({}^{\heartsuit}\)Allen Institute for Artificial Intelligence, Seattle, Washington \({}^{\blacktriangledown}\)Allen Institute for Artificial Intelligence, Irvine, California \({}^{\diamondsuit}\)Mila, Universite de Montreal \({}^{\clubsuit}\)University of Washington \({}^{\spadesuit}\)Carnegie Mellon University pradeepd@allenai.org

###### Abstract

Training semantic parsers from question-answer pairs typically involves searching over an exponentially large space of logical forms, and an unguided search can easily be misled by spurious logical forms that coincidentally evaluate to the correct answer. We propose a novel iterative training algorithm that alternates between searching for consistent logical forms and maximizing the marginal likelihood of the retrieved ones. This training scheme lets us iteratively train models that provide guidance to subsequent ones to search for logical forms of increasing complexity, thus dealing with the problem of spuriousness. We evaluate these techniques on two hard datasets: WikiTableQuestions (WTQ) and Cornell Natural Language Visual Reasoning (NLVR), and show that our training algorithm outperforms the previous best systems, on WTQ in a comparable setting, and on NLVR with significantly less supervision.

## 1 Introduction

Semantic parsing is the task of translating natural language utterances into machine-executable meaning representations, often called _programs_ or _logical forms_. These logical forms can be executed against some representation of the context in which the utterance occurs, to produce a _denotation_. This setup allows for complex reasoning over contextual knowledge, and it has been successfully used in several natural language understanding problems such as question answering Berant et al. (2013), program synthesis Yin and Neubig (2017) and building natural language interfaces Suhr et al. (2018).

Recent work has focused on training semantic parses via _weak supervision_ from denotations alone Liang et al. (2011); Berant et al. (2013). This is because obtaining logical form annotations is generally expensive (although recent work has addressed this issue to some extent Yih et al. (2016)), and not assuming full supervision lets us be agnostic about the logical form language. The second reason is more important in open-domain semantic parsing tasks where it may not be possible to arrive at a complete set of operators required by the task. However, training semantic parsers with weak supervision requires not only searching over an exponentially large space of logical forms Berant et al. (2013); Artzi and Zettlemoyer (2013); Pasupat and Liang (2015); Guu et al. (2017), _inter alia_) but also dealing with _spurious_ logical forms that evaluate to the correct denotation while not being semantically equivalent to the utterance. For example, if the denotations are binary, \(50\%\) of all syntactically valid logical forms evaluate to the correct answer, regardless of their semantics. This problem renders the training signal extremely noisy, making it hard for the model to learn anything without some additional guidance during search.

We introduce two innovations to improve learning from denotations. Firstly, we propose an iterative search procedure for gradually increasing the complexity of candidate logical forms for each training instance, leading to better training data and better parsing accuracy. This procedure is implemented via training our model with two interleaving objectives, one that involves searching for logical forms of limited complexity during training (online search), and another that maximizes the marginal likelihood of retrieved logical forms. Second, we include a notion of coverage over the question in the search step to guide the training algorithm towards logical forms that not only evaluate to the correct denotation, but also have some connection to the words in the utterance.

We demonstrate the effectiveness of these two techniques on two difficult reasoning tasks: Wik

iTableQuestions(WTQ) Pasupat and Liang (2015), an open domain task with significant lexical variation, and Cornell Natural Language Visual Reasoning (NLVR) Suhr et al. (2017), a closed domain task with binary denotations, and thus far less supervision. We show that: 1) interleaving online search and MML over retrieved logical forms (SS4) is a more effective training algorithm than each of those objectives alone; 2) coverage guidance during search (SS3) is helpful for dealing with weak supervision, more so in the case of NLVR where the supervision is weaker; 3) a combination of the two techniques yields \(44.3\%\) test accuracy on WTQ, outperforming the previous best single model in a comparable setting, and \(82.9\%\) test accuracy on NLVR, outperforming the best prior model, which also relies on greater supervision.

## 2 Background

### Weakly supervised semantic parsing

We formally define semantic parsing in a weakly supervised setup as follows. Given a dataset where the \(i^{th}\) instance is the triple \(\{x_{i},w_{i},d_{i}\}\), representing a sentence \(x_{i}\), the world \(w_{i}\) associated with the sentence, and the corresponding denotation \(d_{i}\), our goal is to find \(y_{i}\), the translation of \(x_{i}\) in an appropriate logical form language (see SS5.3), such that \([y_{i}]^{w_{i}}=d_{i}\); i.e., the execution of \(y_{i}\) in world \(w_{i}\) produces the correct denotation \(d_{i}\). A semantic parser defines a distribution over logical forms given an input utterance: \(p(Y|x_{i};\theta)\).

### Training algorithms

In this section we describe prior techniques for training semantic parsers with weak supervision: maximizing marginal likelihood, and reward-based methods.

#### 2.2.1 Maximum marginal likelihood

Most work on training semantic parsers from denotations maximizes the likelihood of the denotation given the utterance. The semantic parsing model itself defines a distribution over _logical forms_, however, not _denotations_, so this maximization must be recast as a _marginalization_ over logical forms that evaluate to the correct denotation:

\[\max_{\theta}\prod_{x_{i},d_{i}\in\mathcal{D}}\sum_{y_{i}\in\mathbb{V}\cdot[ \![y_{i}]\!]^{w_{i}}=d_{i}}p(y_{i}|x_{i};\theta) \tag{1}\]

This objective function is called _maximum marginal likelihood_ (MML). The inner summation is in general intractable to perform during training, so it is only approximated.

Most prior work Berant et al. (2013); Goldman et al. (2018), _inter alia_) approximate the intractable marginalization by summing over logical forms obtained via beam search during training. This typically results in frequent search failures early during training when model parameters are close to random, and in general may only yield spurious logical forms in the absence of any guidance. Since modern semantic parsers typically operate without a lexicon, new techniques are essential to provide guidance to the search procedure Goldman et al. (2018).

One way of providing this guidance during search is to perform some kind of heuristic search up front to find a set of logical forms that evaluate to the correct denotation, and use those logical forms to approximate the inner summation Liang et al. (2011); Krishnamurthy et al. (2017). The particulars of the heuristic search can have a large impact on performance; a smaller candidate set has lower noise, while a larger set makes it more likely that the correct logical form is in it, and one needs to strike the right balance. In this paper, we refer to the MML that does search during training as _dynamic MML_, and the one that does an offline search as _static MML_.

The main benefit of dynamic MML is that it adapts its training signal over time. As the model learns, it can increasingly focus its probability mass on a small set of very likely logical forms. The main benefit of static MML is that there is no need to search during training, so there is a consistent training signal even at the start of training, and it is typically more computationally efficient than dynamic MML.

#### 2.2.2 Reward-based methods

When training weakly supervised semantic parsers, it is often desirable to inject some prior knowledge into the training procedure by defining arbitrary reward or cost functions. There exists prior work that use such methods, both in a reinforcement learning setting Liang et al. (2017, 2018), and otherwise Iyyer et al. (2017); Guu et al. (2017). In our work, we define a customized cost function that includes a coverage term, and use a Minimum Bayes Risk (MBR) Goodman (1996); Goel and Byrne (2000); Smith and Eisner ( 

2006) training scheme, which we describe in SS3.

## 3 Coverage-guided search

Weakly-supervised training of semantic parsers relies heavily on lexical cues to guide the initial stages of learning to good logical forms. Traditionally, these lexical cues were provided in the parser's lexicon. Neural semantic parsers remove the lexicon, however, and so need another mechanism for obtaining these lexical cues. In this section we introduce the use of coverage to inject lexicon-like information into neural semantic parsers.

Coverage is a measure of relevance of the candidate logical form \(y_{i}\) to the input \(x_{i}\), in terms of how well the productions in \(y_{i}\) map to parts of \(x_{i}\). We use a small manually specified lexicon as a mapping from source language to the target language productions, and define coverage of \(y_{i}\) as the number of productions triggered by the input utterance, according to the lexicon, that are included in \(y_{i}\).

We use this measure of coverage to augment our loss function, and train using an MBR based algorithm as follows. We use beam search to train a model to minimize the expected value of a cost function \(\mathcal{C}\):

\[\min_{\theta}\sum_{i=1}^{N}\mathbb{E}_{\tilde{p}(y_{i}|x_{i};\tilde{p})} \mathcal{C}(x_{i},y_{i},w_{i},d_{i}) \tag{2}\]

where \(\tilde{p}\) is a _re-normalization1_ of the probabilities assigned to all logical forms on the beam.

Footnote 1: Note that without this re-normalization, and with a -1/0 cost function based on denotation accuracy, MBR will maximize the likelihood of correct logical forms on the beam, which is equivalent to dynamic MML.

We define the cost function \(\mathcal{C}\) as:

\[\mathcal{C}(x_{i},y_{i},w_{i},d_{i})=\lambda\mathcal{S}(y_{i},x_{i})+(1- \lambda)\mathcal{T}(y_{i},w_{i},d_{i}) \tag{3}\]

where the function \(\mathcal{S}\) measures the number of items that \(y_{i}\) is missing from the actions (or grammar production rules) triggered by the input utterance \(x_{i}\) given the lexicon; and the function \(\mathcal{T}\) measures the consistency of the evaluation of \(y_{i}\) in \(w_{i}\), meaning that it is \(0\) if \(\llbracket y_{i}\rrbracket^{w_{i}}=d_{i}\), or a value \(e\) otherwise. We set \(e\) as the maximum possible value of the coverage cost for the corresponding instance, to make the two costs comparable in magnitude. \(\lambda\) is a hyperparameter that gives the relative weight of the coverage cost.

## 4 Iterative search

In this section we describe the iterative technique for refining the set of candidate logical forms associated with each training instance.

As discussed in SS2.2, most prior work on weakly-supervised training of semantic parsers uses dynamic MML. This is particularly problematic in domains like NLVR, where the supervision signal is binary--it is very hard for dynamic MML to bootstrap its way to finding good logical forms. To solve this problem, we interleave static MML, which has a consistent supervision signal from the start of training, with the coverage-augmented MBR algorithm described in SS3.

In order to use static MML, we need an initial set of candidate logical forms. We obtain this candidate set using a bounded-length exhaustive search, filtered using heuristics based on the same lexical mapping used for coverage in SS3. A bounded-length search will not find logical forms for the entire training data, so we can only use a subset of the data for initial training. We train a model to convergence using static MML on these logical forms, then use that model to initialize coverage-augmented MBR training. This gives the model a good starting place for the dynamic learning algorithm, and the search at training time can look for logical forms that are longer than could be found with the bounded-length exhaustive search. We train MBR to convergence, then use beam search on the MBR model to find a new set of candidate logical forms for static MML on the training data. This set of logical forms can have a greater length than those in the initial set, because this search uses model scores to not exhaustively explore all possible paths, and thus will likely cover more of the training data. In this way, we can iteratively improve the candidate logical forms used for static training, which in turn improves the starting place for the online search algorithm.

Algorithm 1 concretely describes this process. Decode in the algorithm refers to running a beam search decoder that returns a set of consistent logical forms (i.e. \(\mathcal{T}=0\)) for each of the input utterances. We start off with a seed dataset \(\mathcal{D}^{0}\) for which consistent logical forms are available.

## 5 Datasets

We will now describe the two datasets we use in this work to evaluate our methods - Cornell NLVR 

and WikiTableQuestions.

### Cornell NLVR

Cornell NLVR is a language-grounding dataset containing natural language sentences provided along with synthetically generated visual contexts, and a label for each sentence-image pair indicating whether the sentence is true or false in the given context. Figure 1 shows two example sentence-image pairs from the dataset (with the same sentence). The dataset also comes with structured representations of images, indicating the color, shape, size, and x- and y-coordinates of each of the objects in the image. While we show images in Figure 1 for ease of exposition, we use the structured representations in this work.

Following the notation introduced in SS2.1, \(x_{i}\) in this example is _There is a box with only one item that is blue_. The structured representations associated with the two images shown are two of the worlds (\(w^{1}_{i}\) and \(w^{2}_{i}\)), in which \(x_{i}\) could be evaluated. The corresponding labels are the denotations \(d^{1}_{i}\) and \(d^{2}_{i}\) that a translation \(y_{i}\) of the sentence \(x_{i}\) is expected to produce, when executed in the two worlds respectively. That the same sentence occurs with multiple worlds is an important property of this dataset, and we make use of it by defining the function \(\mathcal{T}\) to be \(0\) only if \(\forall_{w^{j}_{i},d^{j}_{i}}[\![y_{i}]\!]^{w^{j}_{i}}=d^{j}_{i}\).

### WikiTableQuestions

WikiTableQuestions is a question-answering dataset where the task requires answering complex questions in the context of Wikipedia tables. An example can be seen in Figure 2. Unlike NLVR, the answers are not binary. They can instead be cells in the table or the result of numerical or set-theoretic operations performed on them.

### Logical form languages

For NLVR, we define a typed variable-free functional query language, inspired by the GeoQuery language Zelle and Mooney (1996). Our language contains six basic types: box (referring to one of the three gray areas in Figure 1), object (referring to the circles, triangles and squares in Figure 1), shape, color, number and boolean. The constants in our language are color and shape names, the set of all boxes in an image, and the set of all objects in an image. The functions in our language include those for filtering objects and boxes, and making assertions, a higher order function for handling negati

Figure 1: Example from NLVR dataset showing an utterance associated with two worlds and corresponding binary denotations. Also shown are the logical form and the actions triggered by the lexicon from the utterance.

Figure 2: Example from WikiTableQuestions dataset showing an utterance, a world, associated denotation, corresponding logical form, and actions triggered by the lexicon.

 

querying objects in boxes. This type specification of constants and functions gives us a grammar with 115 productions, of which 101 are terminal productions (see Appendix A.1 for the complete set of rules in our grammar). Figure 1 shows an example of a complete logical form in our language.

For WTQ, we use the functional query language used by Liang et al. (2018) as the logical form language. Figure 2 shows an example logical form.

### Lexicons for coverage

The lexicon we use for the coverage measure described in SS3 contains under 40 rules for each logical form language. They mainly map words and phrases to constants and unary functions in the target language. The complete lexicons are shown in the Appendix. Figures 1 and 2 also show the actions triggered by the corresponding lexicons for the utterances shown. We find that small but precise lexicons are sufficient to guide the search process away from spurious logical forms. Moreover, as shown empirically in SS6.4, the model for NLVR does not learn much without this simple but crucial guidance.

## 6 Experiments

We evaluate both our contributions on NLVR and WikiTableQuestions.

### Model

In this work, we use a grammar-constrained encoder-decoder neural semantic parser for our experiments. Of the many variants of this basic architecture (see SS7), all of which are essentially seq2seq models with constrained outputs and/or re-parameterizations, we choose to use the parser of Krishnamurthy et al. (2017), as it is particularly well-suited to the WikiTableQuestions dataset, which we evaluate on.

The encoder in the model is a bi-directional recurrent neural network with Long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997) cells, and the decoder is a grammar-constrained decoder also with LSTM cells. Instead of directly outputting tokens in the logical form, the decoder outputs _production rules_ from a CFG-like grammar. These production rules sequentially build up an abstract syntax tree, which determines the logical form. The model also has an entity linking component for producing table entities in the logical forms; this component is only applicable to WikiTableQuestions, and we remove it when running experiments on NLVR. The particulars of the model are not the focus of this work, so we refer the reader to the original paper for more details.

In addition, we slightly modify the constrained decoding architecture from Krishnamurthy et al. (2017) to bias the predicted actions towards those that would decrease the value of \({\cal S}(y_{i},x_{i})\). This is done using a coverage vector, \(v^{\cal S}_{i}\) for each training instance that keeps track of the production rules triggered by \(x_{i}\), and gets updated whenever one of those desired productions is produced by the decoder. That is, \(v^{\cal S}_{i}\) is a vector of 1s and 0s, with 1s indicating the triggered productions that are yet to be produced by the decoder. This is similar to the idea of checklists used by Kiddon et al. (2016). The decoder in the original architecture scores output actions at each time step by computing a dot product of the predicted action representation with the embeddings of each of the actions. We add a weighted sum of all the actions that are yet to produced:

\[s^{a}_{i}=e^{a}.(p_{i}+\gamma*v^{\cal S}_{i}.E) \tag{4}\]

where \(s^{a}_{i}\) is the score of action \(a\) at time step \(i\), \(e^{a}\) is the embedding of that action, \(p_{i}\) is the predicted action representation, \(E\) is the set of embeddings of all the actions, and \(\gamma\) is a learned parameter for regularizing the bias towards yet-to-be produced triggered actions.

### Experimental setup

NlrrWe use the standard train-dev-test split for NLVR, containing 12409, 988 and 989 sentence-image pairs respectively. NLVR contains most of the sentences occurring in multiple worlds (with an average of 3.9 worlds per sentence). We set the word embedding and action embedding sizes to 50, and the hidden layer size of both the encoder and the decoder to 30. We initialized all the parameters, including the word and action embeddings using Glorot uniform initialization Glorot and Bengio (2010). We found that using pretrained word representations did not help. We added a dropout Srivastava et al. (2014) of 0.2 on the outputs of the encoder and the decoder and before predicting the next action, set the beam size to 10 both during training and at test time, and trained the model using ADAM Kingma and Ba (2014) with a learning rate of 0.001. All the hyperparameters are tuned on the validation set 

WikiTableQuestionsThis dataset comes with five different cross-validation folds of training data, each containing a different 80/20 split for training and development. We first show results aggregated from all five folds in SS6.3, and then show results from controlled experiments on fold 1. We replicated the model presented in Krishnamurthy et al. (2017), and only changed the training algorithm and the language used. We used a beam size of 20 for MBR during training and decoding, and 10 for MML during decoding, and trained the model using Stochastic Gradient Descent (Kiefer et al., 1952) with a learning rate of 0.1, all of which are tuned on the validation sets.

Specifics of iterative searchFor our iterative search algorithm, we obtain an initial set of candidate logical forms in both domains by exhaustively searching to a depth of \(10^{2}\). During search we retrieve the logical forms that lead to the correct denotations in all the corresponding worlds, and sort them based on their coverage cost using the coverage lexicon described in SS5.4, and choose the top-\(k\)3. At each iteration of the search step in our iterative training algorithm, we increase the maximum depth of our search with a step-size of 2, finding more complex logical forms and covering a larger proportion of the training data. While exhaustive search is prohibitively expensive beyond a fixed number of steps, our training process that uses beam search based approximation can go deeper.

Footnote 3: \(k\) is a hyperparameter that is chosen on the dev set at each iteration in iterative search, and is typically \(10\) or \(20\)

ImplementationWe implemented our model and training algorithms within the AllenNLP Gardner et al. (2018) toolkit. The code and models are publicly available at [https://github.com/allenai/iterative-search-semparse](https://github.com/allenai/iterative-search-semparse).

### Main results

WikiTableQuestionsTable 1 compares the performance of a single model trained using Iterative Search, with that of previously published single models. We excluded ensemble models since there are differences in the way ensembles are built for this task in previous work, either in terms of size or how the individual models were chosen. We show both best and average (over 5 folds) single model performance from Liang et al. (2018) (Memory Augmented Policy Optimization). The best model was chosen based on performance on the development set. Our single model performances are computed in the same way. Note that Liang et al. (2018) also use a lexicon similar to ours to prune the seed set of logical forms used to initialize their memory buffer.

In Table 2, we compare the performance of our iterative search algorithm with three baselines: 1) Static MML, as described in SS2.2.1 trained on the candidate set of logical forms obtained through the heuristic search technique described in SS6.2; 2) Iterative MML, also an iterative technique but unlike iterative search, we skip MBR and iteratively train static MML models while increasing the number of decoding steps; and 3) MAPO (Liang et al., 2018), the current best published system on WTQ. All four algorithms are trained and evaluated on the first fold, use the same language, and the bottom three use the same model and the same set of logical forms used to train static MML.

\begin{table}
\begin{tabular}{l c c} \hline \hline
* & 34.8 \\ Zhang et al. (2017) & 40.4 & 43.7 \\ Liang et al. (2018) (MAPO) (avg.) & 42.3 & 43.1 \\ Liang et al. (2018) (MAPO) (best) & 42.7 & 43.8 \\ Iterative Search (avg.) & 42.1 & 43.9 \\ Iterative Search (best) & **43.1** & **44.3** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of single model performances of Iterative Search with previously reported single model performances

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Algorithm** & **Dev acc.** & **Test acc.** \\ \hline MAPO & 42.1 & 42.7 \\ \hline Static MML & 40.0 & 42.2 \\ Iterative MML & 42.5 & 43.1 \\ Iterative Search & **43.0** & **43.8** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of iterative search with static MML, iterative MML, and the previous best result from (Liang et al., 2018), all trained on the official split 1 of WikiTableQuestions and tested on the official test set.

 

NlvrIn Table 3, we show a comparison of the performance of our _iterative coverage-guided search_ algorithm with the previously published approaches for NLVR. The first two rows correspond to models that are not semantic parsers. This shows that semantic parsing is a promising direction for this task. The closest work to ours is the weakly supervised parser built by Goldman et al. (2018). They build a lexicon similar to ours for mapping surface forms in input sentences to abstract clusters. But in addition to defining a lexicon, they also manually annotate complete sentences in this abstract space, and use those annotations to perform data augmentation for training a supervised parser, which is then used to initialize a weakly supervised parser. They also explicitly use the abstractions to augment the beam during decoding using caching, and a separately-trained discriminative re-ranker to re-order the logical forms on the beam. As a discriminative re-ranker is orthogonal to our contributions, we show their results with and without it, with "Abs. Sup." being more comparable to our work. Our model, which uses no data augmentation, no caching during decoding, and no discriminative re-ranker, outperforms their variant without reranking on the public test set, and outperforms their best model on the hidden test set, achieving a new state-of-the-art result on this dataset.

### Effect of coverage-guided search

To evaluate the contribution of coverage-guided search, we compare the the performance of the NLVR parser in two different settings: with and without coverage guidance in the cost function. We also compare the performance of the parser in the two settings, when initialized with parameters from an MML model trained to maximize the likelihood of the set of logical forms obtained from exhaustive search. Table 4 shows the results of this comparison. We measure accuracy and consistency of all four models on the publicly available test set, using the official evaluation script. Consistency here refers to the percentage of logical forms that produce the correct denotation in all the corresponding worlds, and is hence a stricter metric than accuracy. The cost weight (\(\lambda\) in Equation 3) was tuned based on validation set performance for the runs with coverage, and we found that \(\lambda=0.4\) worked best.

It can be seen that both with and without initialization, coverage guidance helps by a big margin, with the gap being even more prominent in the case where there is no initialization. When there is neither coverage guidance nor a good initialization, the model does not learn much from unguided search and get a test accuracy not much higher than the majority baseline of 56.2%.

We found that coverage guidance was not as useful for WTQ. The average value of the best performing \(\lambda\) was around \(0.2\), and higher values neither helped nor hurt performance.

### Effect of iterative search

To evaluate the effect of _iterative search_, we present the accuracy numbers from the search (S) and maximization (M) steps from different iterations in Tables 5 and 6, showing results on NLVR and WTQ, respectively. Additionally, we also show number of decoding steps used at each iterations, and the percentage of sentences in the training data for which we were able to obtain consistent logical forms from the S step, the set that was used in the M step of the same iteration. It can be seen in both tables that a better MML model gives a better initialization for MBR, and a better MBR model results in a larger set of utterances for which we can retrieve consistent logical forms, thus improving the subsequent MML model. The improvement for NLVR is more pronounced (a gain of 21% absolute) than for WTQ (a gain of 3% absolute), likely because the initial exhaustive search provides a much higher percentage of spurious logical forms for NLVR, and thus the starting place is relatively worse.

Complexity of Logical FormsWe analyzed the logical forms produced by our iterative search algorithm at different iterations to see how they differ. As expected, for NLVR, allowing greater depths lets the parser explore more complex logical forms. Table 7 shows examples from the validation set that indicate this trend.

## 7 Related Work

Most of the early methods used for training semantic parsers required the training data to come with annotated logical forms Zelle and Mooney (1996); Zettlemoyer and Collins (2005). The primary limitation of such methods is that manually producing these logical forms is expensive, making it hard to scale these methods across domains 

More recent research has focused on training semantic parsers with _weak supervision_(Liang et al., 2011; Berant et al., 2013), or trying to automatically infer logical forms from denotations [7]. However, matching the performance of a fully supervised semantic parser with only weak supervision remains a significant challenge [45].

The main contributions of this work deal with training semantic parsers with weak supervision, and we gave a detailed discussion of related training methods in SS2.2.

We evaluate our contributions on the NLVR and WikiTableQuestions datasets. Other work that evaluates on on these datasets include Goldman et al. (2018), Tan and Bansal (2018), Neelakantan et al. (2017), Krishnamurthy et al. (2017), Haug et al. (2018), and Liang et al. (2018). These prior works generally present modeling contributions that are orthogonal (and in some cases complementary) to the contributions of this paper. There has also been a lot of recent work on neural semantic parsing, most of which is also orthogonal to (and could probably benefit from) our contributions [69, 40, 45, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 61, 64, 65, 67, 69, 62, 68, 69, 60, 63, 61, 65, 64, 66, 67, 68, 69, 60, 64, 68, 69, 61, 62, 64, 63, 65, 66, 67, 68, 69, 62, 69, 63, 64, 65, 66, 68, 69, 60, 67, 69, 61, 68, 61, 69, 62, 63, 64, 65, 66, 67, 68, 69, 60, 68, 61, 69, 60, 61, 62, 64, 65, 67, 68, 69, 61, 63, 68, 62, 69, 64, 66, 67, 68, 69, 60, 61, 68, 62, 69, 61, 63, 64, 65, 66, 67, 68, 69, 61, 68, 69, 62, 63, 60, 64, 65, 66, 68, 67, 69, 61, 68, 69, 60, 62, 61, 63, 64, 65, 66, 67, 68, 69, 62, 68, 69, 60, 61, 64, 66, 68, 69, 61, 62, 63, 65, 67, 68, 69, 60, 61, 62, 64, 66, 68, 69, 61, 63, 67, 69, 62, 64, 65, 68, 66, 69, 61, 67, 68, 69, 60, 63, 64, 65, 67, 68, 69, 61, 69, 62, 63, 61, 64, 65, 66, 67, 68, 69, 60, 64, 68, 69, 62, 65, 69, 61, 66, 67, 68, 69, 63, 68, 64, 69, 65, 66, 67, 68, 69, 60, 68, 69, 61, 62, 63, 64, 65, 66, 67, 68, 69, 60, 69, 61, 62, 64, 65, 68, 69, 62, 66, 67, 68, 69, 60, 61, 62, 63, 64, 65, 66, 68, 69, 61, 63, 67, 69, 62, 68, 67, 69, 60, 64, 68, 65, 69, 61, 66, 62, 67, 68, 69, 63, 69, 64, 65, 66, 67, 68, 69, 60, 68, 69, 61, 62, 63, 64, 65, 67, 69, 62, 68, 69, 60, 61, 63, 65, 64, 66, 67, 68, 69, 62, 69, 63, 65, 66, 68, 69, 60, 67, 69, 61, 68, 62, 69, 63, 64, 65, 66, 67, 68, 69, 61, 69, 62, 63, 65, 69, 64, 66, 65, 67, 68, 69, 60, 69, 62, 61, 63, 66, 67, 68, 69, 60, 61, 62, 64, 65, 68, 69, 62, 63, 66, 69, 64, 66, 65, 67, 68, 69, 60, 69, 61, 62, 63, 64, 65, 66, 67, 68, 69, 61, 62, 65, 69, 63, 66, 68, 69, 60, 64, 65, 69, 66, 67, 68, 69, 61, 62, 69, 63, 64, 65, 66, 68, 69, 62, 67, 69, 63, 68, 69, 60, 64, 65, 66, 67, 68, 69, 61, 69, 62, 63, 65, 69, 64, 66, 65, 67, 68, 69, 60, 69, 61, 62, 63, 66, 64, 65, 66, 67, 68, 69, 62, 69, 60, 63, 65, 69, 61, 64, 66, 65, 66, 67, 68, 69, 62, 69, 60, 61, 62, 63, 64, 65, 66, 68, 69, 62, 67, 69, 63, 68, 69, 60, 64, 65, 66, 69, 61, 62, 65, 66, 67, 68, 69, 60, 61, 62, 63, 64, 65, 66, 69, 62, 66, 67, 68, 69, 60, 61, 62, 64, 65, 66, 69, 62, 63, 66, 67, 68, 69, 61, 64, 65, 69, 62, 66, 63, 66, 64, 65, 66, 67, 68, 69, 60, 69, 61, 62, 64, 65, 66, 69, 62, 66, 67, 68, 69, 60, 61, 62, 63, 64, 65, 66, 69, 62, 65, 66, 67, 68, 69, 60, 61, 62, 63, 64, 65, 66, 69, 60, 62, 66, 67, 68, 69, 61, 63, 68, 69, 62, 64, 65, 66, 67, 68, 69, 60, 61, 62, 69, 63, 64, 65, 66, 69, 61, 64, 65, 66, 67, 68, 69, 62, 69, 60, 63, 66, 65, 69, 61, 60, 62, 64, 65, 66, 67, 68, 69, 61, 63, 69, 62, 64, 65, 66, 67, 68, 69, 60, 62, 69, 61, 63, 64, 65, 66, 69, 62, 67, 68, 69, 60, 63, 68, 69, 61, 64, 65, 66, 67, 68, 69, 62, 69, 63, 69, 60, 64, 65, 66, 67, 68, 69, 60, 65, 69, 61, 66, 67, 68, 69, 60, 69, 62, 63, 64, 65, 66, 67, 68, 69, 60, 69, 61, 60, 62, 63, 65, 69, 62, 64, 66, 67, 68, 69, 60, 63, 69, 61, 64, 65, 66, 67, 68, 69, 60, 65, 69, 62, 68, 69, 60, 66, 67, 68, 69, 61, 69, 62, 63, 64, 65, 66, 67, 68, 69, 60, 69, 60, 61, 62, 63, 65, 69, 60, 62, 64, 65, 66, 67, 68, 69, 61, 63, 69, 62, 65, 69, 60, 63, 67, 68, 69, 60, 64, 65, 69, 60, 66, 67, 68, 69, 61, 69, 62, 63, 66, 69, 60, 67, 68, 69, 60, 69, 61, 60, 62, 63, 64, 65, 66, 67, 68, 69, 60, 69, 61, 60, 63, 68, 69, 60, 61, 64, 65, 67, 69, 62, 66, 67, 68, 69, 60, 69, 62, 69, 60, 63, 69, 60, 64, 65, 66, 67, 68, 69, 60, 67, 69, 61, 68, 69, 60, 



## 8 Conclusion

We have presented a new technique for training semantic parsers with weak supervision. Our key insights are that lexical cues are crucial for guiding search during the early stages of training, and that the particulars of the approximate marginalization in maximum marginal likelihood have a large impact on performance. To address the first issue, we used a simple coverage mechanism for including lexicon-like information in neural semantic parsers that do not have lexicons. For the second issue, we developed an iterative procedure that alternates between statically-computed and dynamically-computed training signals. Together these two contributions greatly improve semantic parsing performance, leading to new state-of-the-art results on NLVR and WikiTableQuestions. As these contributions are to the learning algorithm, they are broadly applicable to many models trained with weak supervision. One potential future work direction is investigating whether they extend to other structured prediction problems beyond semantic parsing.

### Acknowledgments

We would like to thank Jonathan Berant and Noah Smith for comments on earlier drafts and Chen Liang for helping us with implementation details of MAPO. Computations on beaker.org were supported in part by credits from Google Cloud.

## References

* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association of Computational Linguistics_, 1:49-62.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1533-1544.
* Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In _ACL'16_.
* Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. _CoRR_, abs/1803.07640.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_, pages 249-256.
* Goel and Byrne (2000) Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. _Computer Speech & Language_, 14(2):115-135.
* Goldman et al. (2018) Omer Goldman, Veronica Latcinnik, Udi Naveh, Amir Globerson, and Jonathan Berant. 2018. Weakly-supervised semantic parsing with abstract examples. In _ACL_.
* Goodman (1996) Joshua Goodman. 1996. Parsing algorithms and metrics. In _Proceedings of the 34th annual meeting on Association for Computational Linguistics_, pages 177-183. Association for Computational Linguistics.
* Guu et al. (2017) Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In _Association for Computational Linguistics (ACL)_.
* Haug et al. (2018) Till Haug, Octavian-Eugen Ganea, and Paulina Garraova. 2018. Neural multi-step reasoning for question answering on semi-structured tables. In _ECIR_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Iyyer et al. (2017) Mohit Iyyer, Wen tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In _Association for Computational Linguistics_.

\begin{table}
\begin{tabular}{l l} _There is a tower with four blocks_ \\  & (box\_exists (member\_count\_equals allboxes 4)) \\  & _Alteast one black triangle is not touching the edge_ \\  & (object\_exists (black (triangle (negate\_filter touchwall)all\_objects)))) \\  & _There is a yellow block as the top of a tower with exactly three blocks._ \\  & (object\_exists (yellow (top (object\_in\_box (member\_count\_equals all boxes 3))))) \\ _The tower with three blocks has a yellow block over a black block_ \\  & (object\_count\_greaterthanequal (yellow (above (black (object\_in\_box (member\_count equals all\_boxes 3))))) \\  & (member\_count\_equals all\_boxes 3)))))) \\ \end{tabular}
\end{table}
Table 7: Complexity of logical forms produced at different iterations, from iteration 0 to iteration 3; each logical form could not be produced at the previous iterations

* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _ACL'16_.
* Kiddon et al. (2016) Chloe Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 329-339.
* Kiefer et al. (1952) Jack Kiefer, Jacob Wolfowitz, et al. 1952. Stochastic estimation of the maximum of a regression function. _The Annals of Mathematical Statistics_, 23(3):462-466.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1516-1526.
* Kwiatkowski et al. (2011) Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2011. Lexical generalization in ccg grammar induction for semantic parsing. In _Proceedings of the conference on empirical methods in natural language processing_, pages 1512-1523. Association for Computational Linguistics.
* Liang et al. (2017) Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 23-33.
* Liang et al. (2018) Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis with generalization. _arXiv preprint arXiv:1807.02322_.
* Liang et al. (2011) Percy Liang, Michael I Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In _ACL_.
* Misra et al. (2018) Dipendra Misra, Ming-Wei Chang, Xiaodong He, and Wen tau Yih. 2018. Policy shaping and generalized update equations for semantic parsing from denotations. In _EMNLP_.
* Neelakantan et al. (2017) Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In _ICLR_.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, volume 1, pages 1470-1480.
* Pasupat and Liang (2016) Panupong Pasupat and Percy Liang. 2016. Inferring logical forms from denotations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 23-32.
* Rabinovich et al. (2017) Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In _ACL_.
* See et al. (2017)bigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. _ACL_.
* Smith and Eisner (2006) David A Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In _Proceedings of the COLING/ACL on Main conference poster sessions_, pages 787-794. Association for Computational Linguistics.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_, 15(1):1929-1958.
* Suhr et al. (2018) Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to excutable formal queries. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 2238-2249.
* Suhr et al. (2017)Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, volume 2, pages 217-223.
* Tan and Bansal (2018) Hao Tan and Mohit Bansal. 2018. Object ordering with bidirectional matchings for visual reasoning. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Tu et al. (2016)Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. _ACL_.
* Yih et al. (2016)Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In _ACL_.
* Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In _ACL'17_.
* Zelle and Mooney (1996) John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In _Proceedings of the thirteenth national conference on Artificial intelligence-Volume 2_, pages 1050-1055. AAAI Press.

 

Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In _UAI'05_.
* Zhang et al. (2017) Yuchen Zhang, Panupong Pasupat, and Percy Liang. 2017. Macro grammars and holistic triggering for efficient semantic parsing. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1214-1223.

## Appendix A Logical form language and lexicon for NLVR

**Basic Types:** bool (t),box (b),object (o),shape (s),color (c),number (n)

In the grammar and lexicon that follow we use the following placeholders,

```
quantifier\(\in\{\text{any},\text{all},\text{none}\}\)comparator\(\in\{\text{equals},\text{not\_equals},\) lesser,lesser_equals, greater,greater_equals\(\}\)color\(\in\{\text{yellow},\text{blue},\text{black}\}\)shape\(\in\{\text{square},\text{triangle},\text{circle}\}\)size\(\in\{\text{big},\text{medium},\text{small}\}\)location\(\in\{\text{above},\text{below},\text{top},\text{left},\) right,bottom,corner,wall\(\}\)number\(\in\{\text{1...9}\}\)

### Grammar Constants
``` b->all_boxes c->color_black, color_blue,color_yellow n->1,2,....,9 o->all_objects s->shape_circle, shape_square,shape_triangle Object filtering functions<o,o>->[location],[color], [shape],[size],same_color, same_shape,touch_object, touch_bottom, touch_top, touch_left, touch_right, touch_corner, touch_wall, Box filtering functions<b,<s,b>>-> member_shape_[quantifier]_equals<b,<c,b>>-> member_color_[quantifier]_equals<b,<n,b>>-> member_count_[comparator]member_colour_[comparator], member_shape_count_[comparator]<b,b>-> member_color_different, member_shape_different, member_shape_same Assertion functions<b,t>->box_exists<o,t>->object_exists<b,<n,t>>->box_count_[comparator]<o,<c,t>>-> object_color_[quantifier]_equals<o,<s,t>>-> object_shape_[quantifier]_equals<o,<n,t>>-> object_color_count_[comparator], object_shape_count_[comparator], object_count_[comparator] Other functions<b,o>->object_in_box<<o,o>,<o,O>>->negate_filter ```

### Lexicon for NLVR

there is a box \(\rightarrow\)box_exists there is a [other] \(\rightarrow\)object_exists box \(\ldots\)[color] \(\rightarrow\)color_[color] \(\rightarrow\)box \(\ldots\)[shape] \(\rightarrow\)shape_[shape] not \(\rightarrow\)negate_filter contains \(\rightarrow\)object_in_box t touch \(\ldots\)[location] \(\rightarrow\)touch_[location] [location] \(\rightarrow\)[location] [Shape] \(\rightarrow\)[shape] [color] \(\rightarrow\)[color] [size] \(\rightarrow\)[size] [number] \(\rightarrow\)[number]

## Appendix B Logical form language and lexicon for WikiTTableQuestions

We use the language from Liang et al. (2018). For coverage, in addition to triggering productions for numbers, and column names and cell strings in the table, we use the following lexicon for coverage 



### Lexicon for WikiTableQuestions

at least \(\rightarrow\) filter\({}_{\geq}\)
[greater|larger|more] than \(\rightarrow\) filter\({}_{\geq}\) at most \(\rightarrow\) filter\({}_{\leq}\) no [greater|larger|more] than \(\rightarrow\) filter\({}_{\leq}\) [next|below|after] \(\rightarrow\) next  [previous|above|before] \(\rightarrow\) previous  [first|top] \(\rightarrow\) top  [last|bottom] \(\rightarrow\) bottom same \(\rightarrow\) same_as  total \(\rightarrow\) sum difference \(\rightarrow\) diff average \(\rightarrow\) average [least|smallest|lowest|smallest] \(\rightarrow\) argmin  [most|longest|highest|largest] \(\rightarrow\) argmax  [what|when] ... [last|least] \(\rightarrow\) min  [what|when] ... [first|most] \(\rightarrow\) max  how many \(\rightarrow\) count