Venue: NAACL 2018

# Learning to Map Context-Dependent Sentences

to Executable Formal Queries

Alane Suhr\({}^{\dagger}\), Srinivasan Iyer\({}^{\ddagger}\), and Yoav Artzi\({}^{\dagger}\)

\({}^{\dagger}\) Dept. of Computer Science and Cornell Tech, Cornell University, New York, NY {suhr, yoav}@cs.cornell.edu

\({}^{\ddagger}\) Paul G. Allen School of Computer Science & Engineering, Univ. of Washington, Seattle, WA sviyer@cs.washington.edu

###### Abstract

We propose a context-dependent model to map utterances within an interaction to executable formal queries. To incorporate interaction history, the model maintains an interaction-level encoder that updates after each turn, and can copy sub-sequences of previously predicted queries during generation. Our approach combines implicit and explicit modeling of references between utterances. We evaluate our model on the ATIS flight planning interactions, and demonstrate the benefits of modeling context and explicit references.

show me flights from seatle to boston next monday[Table with 31 flights] on american airlines [Table with 5 flights] which ones arrive at 7pm [No flights returned] show me delta flights [Table with 5 flights]

## 1 Introduction

The meaning of conversational utterances depends strongly on the history of the interaction. Consider a user querying a flight database using natural language (Figure 1). Given a user utterance, the system must generate a query, execute it, and display results to the user, who then provides the next request. Key to correctly mapping utterances to executable queries is resolving references. For example, the second utterance implicitly depends on the first, and the reference _ones_ in the third utterance explicitly refers to the response to the second utterance. Within an interactive system, this information needs to be composed with mentions of database entries (e.g., _Seattle_, _next Monday_) to generate a formal executable representation. In this paper, we propose encoder-decoder models that directly map user utterances to executable queries, while considering the history of the interaction, including both previous utterances and their generated queries.

Reasoning about how the meaning of an utterance depends on the history of the interaction is critical to correctly respond to user requests. As interactions progress, users may omit previously-mentioned constraints and entities, and an increasing portion of the utterance meaning must be derived from the interaction history. Figure 2 shows SQL queries for the utterances in Figure 1. As the interaction progresses, the majority of the generated query is derived from the interaction history (underlined), rather than from the current utterance. A key challenge is resolving what past information is incorporated and how. For example, in the figure, the second utterance depends on the set of flights defined by the first, while adding a new constraint. The third utterance further refines this set by adding a constraint to the constraints from both previous utterances. In contrast, the fourth utterance refers only to the first one, and skips the two utterances in between.1 Correctly generating the fourth query requires understanding that the time constraint (_at 7pm_) can be ignored as it follows an airline constraint that has been replaced.

Footnote 1: An alternative explanation is that utterance four refers to utterance three, and deletes the time and airline constraints.

We study complementary methods to enable this type of reasoning. The first set of methods implicitly reason about references by modifying the encoder-decoder architecture to encode information from previous utterances for generation decisions. We experiment with attending over previous utterances and using an interaction-level recurrent encoder. We also study explicitly maintaining a set of refere

\(k_{1}\): _show one flights from seattle to boston next anomaly._

\(\bar{g}_{1}\):SELECT DISTINet flight.flight.flight.id FROM flight WHERE (flight.from,airport IN (SELECT airport.service.airport.code FROM Airport.service WHERE aircraft.service.city.code IN (SELECT city.city.code FROM city HHERER city.city.name = "SEATTLE")) AND (flight.toairport IN (SELECT airport.service.airport.code FROM airport.service.kitken Airport.service.city.code IN (SELECT city.city.code FROM city HREEES activity.city.same = "SOSTOPT")) AND (flight.tight.displays IN (SELECT days.days.code FROM day KREBE days.day.name IN (SELECT date.day.day.name FROM date.day WHERE date.day.year = 1993 AND date.day.month.number = 2 AND date.day.day.owner = 8));

\(\bar{g}_{2}\): on american antilines \(\bar{g}_{2}\): SELECT DISTINet flight.flight.flight.add FROM flight WHERE (flight.at airline.code - "JAY') AND (flight.from.airport IN (SELECT airport.service.airport.code FROM airport. service WHERE approx.service.city.code IN (SELECT city.city.code FROM city MNEKER city.city.name = SEATTLE")) AND (flight.toairport IN (SELECT sir.portservice.aiport.code FROM air.service WHERE approx.service.city.code INFO (SELECT city.city.code FROM city.service WHERE approx.service WHERE approx.service.city.city.code IN (SELECT city.city.code FROM cities with WHERE city.city.name = SOSTOPT")) AND (flight.tight.days IN (SELECT days.days.codes FROM days.day.name IN (SELECT date.day.day.name FROM date.day.where date.day.year = 1993 AND date.day.month.number = 2 AND date.day.day.number = 8)));

\(\bar{x}_{3}\): which case aarime /?m

**g**: (SELECT DISTINet flight.flight.flight FROM flight.WHERE (flight.at airline.code = "JAY') AND (flight.from.airport IN (SELECT airport.service.airport.code FROM airport.service WHERE airport.service.city.code IN (SELECT city.city.code city.city.code GROvity MNEKER city.city.city.name = SEATTLE")) AND (flight.toairport IN (SELECT air.portservice. airport.code FROM airport.service WHERE approx.service.city.code IN (SELECT city.cycle FROM CIR view WHERE city.city.name = SOSTOP")) AND (flight.flight.toairport IN (SELECT days.codes FROM days.day.name IN (SELECT date.day.day.name FROM date.day.year = 1993 AND date.day.month.number = 2 AND date.day.day.number = 6))) AND (flight.arrival.time = 1900));

**g**: (SELECT DISTINet flight.flight.if FROM flight.wHERE (flight.at airline.code = "JL") AND (flight.from.airport IN (SELECT airport.service.airport.code FROM airport.service WHERE airport.service.city.code IN (SELECT city.code (city.code FROM city.service.city.name = SOSTOP"))) AND (flight.toairport IN (SELECT air.portservice. airport.code FROM airport.service WHERE approx.service.city.code IN (SELECT city.city.code FROM city.service.city.name = SOSTOP")) AND (flight.toairport IN (SELECT day.keys.codes FROM date.day.month.number = 2 AND date.day.day.number = 8)));

**g**: (SELECT DISTINet flight.flight.if FROM flight.wHERE (flight.at airline.code = "JL") AND (flight.from.airport IN (SELECT airport.service.airport.code FROM airport.service WHERE approx.service.city.code IN (SELECT city.city.code (city.code (city.code (city.code (city.workshop.city.summary = SOSTOP")))) AND (flight.light.flow.loss IN (SELECT days.days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes.of errors (days.classes of errors (days.classesof errors (days.class errors (days.class errors (days. class errors (days. class errors (days. class error %)} + + + + 

ecution of \(\bar{y}_{i}\), the interaction history at turn \(i+1\) becomes \(\bar{I}[:i]=\langle(\bar{x}_{1},\bar{y}_{1}),\ldots,(\bar{x}_{i},\bar{y}_{i})\rangle.\)

ModelOur model is based on the recurrent neural network (RNN; Elman, 1990) encoder-decoder framework with attention (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015). We modify the model in three ways to reason about context from the interaction history by attending over previous utterances (Section 4.2), adding a turn-level recurrent encoder that updates after each turn (Section 4.3), and adding a mechanism to copy segments of queries from previous utterances (Section 4.4). We also design a scoring function to score values that are abstracted during pre-processing, including entities and times (Section 6). The full model selects between generating query tokens and copying complete segments from previous queries.

LearningWe assume access to a training set that contains \(N\) interactions \(\{\bar{I}^{(l)}\}_{l=1}^{N}\). We train using a token-level cross-entropy objective (Section 5). For models that use the turn-level encoder, we construct computational graphs for the entire interaction and back-propagate the loss for all queries together. Without the turn-level encoder, each utterance is processed separately.

EvaluationWe evaluate using a test set \(\{\bar{I}^{(l)}\}_{l=1}^{M}\) of \(M\) interactions. We measure the accuracy of each utterance for each test interaction against the annotated query and its execution result. For models that copy segments from previous queries, we evaluate using both predicted and gold previous queries.

## 3 Related Work

Mapping sentences to formal representations, commonly known as semantic parsing, has been studied extensively with linguistically-motivated compositional representations, including variable-free logic (e.g., Zelle and Mooney, 1996; Clarke et al., 2010), lambda calculus (e.g., Zettlemoyer and Collins, 2005; Artzi and Zettlemoyer, 2011; Kushman and Barzilay, 2013), and dependency-based compositional semantics (e.g., Liang et al., 2011; Berant et al., 2013). Recovering lambda-calculus representations was also studied with ATIS with focus on context-independent meaning using grammar-based approaches (Zettlemoyer and Collins, 2007; Kwiatkowski et al., 2011; Wang et al., 2014) and neural networks (Dong and Lapata, 2016; Jia and Liang, 2016).

Recovering context-independent executable representations has been receiving increasing attention. Mapping sentence in isolation to SQL queries has been studied with ATIS using statistical parsing Popescu et al. (2004); Poon (2013) and sequence-to-sequence models Iyer et al. (2017). Generating executable programs was studied with other domains and formal languages Giordani and Moschitti (2012); Ling et al. (2016); Zhong et al. (2017); Xu et al. (2017). Recently, various approaches were proposed to use the formal language syntax to constrain the search space Yin and Neubig (2017); Rabinovich et al. (2017); Krishnamurthy et al. (2017); Cheng et al. (2017) making all outputs valid programs. These contributions are orthogonal to ours, and can be directly integrated into our decoder.

Generating context-dependent formal representations has received less attention. Miller et al. (1996) used ATIS and mapped utterances to semantic frames, which were then mapped to SQL queries. For learning, they required full supervision, including annotated parse trees and contextual dependencies.2Zettlemoyer and Collins (2009) addressed the problem with lambda calculus, using a semantic parser trained separately with context-independent data. In contrast, we generate executable formal queries and require only interaction query annotations for training.

Footnote 2: Miller et al. (1996) provide limited details about their evaluation. Later work notes that they evaluate SQL query correctness (Zettlemoyer and Collins, 2009) with an accuracy of \(78.4\%\), higher than our results. However, the lack of details (e.g., if the metric is strict or relaxed) makes comparison difficult. In addition, we use significantly less supervision, and re-split the data to avoid scenario bias (Section 7).

Recovering context-dependent meaning was also studied with the SCONE Long et al. (2016) and SequentialQA Iyyer et al. (2017) corpora. We compare ATIS to these corpora in Section 7. Resolving explicit references, a part of our problem, has been studied as co-reference resolution Ng (2010). Context-dependent language understanding was also studied for dialogue systems, including with ATIS, as surveyed by Tur et al. (2010). More recently, encoder-decoder methods were applied to dialogue systems Peng et al. (2017); Li et al. (2017), including using hierarchical RNNs Serban et al. (2016, 2017), an architecture related to our turn-level encoder. These approaches use slot-filling frames with limited expressivity, while we focus on the original representation of unconstrained SQL queries.

 

Context-dependent Model

We base our model on an encoder-decoder architecture with attention [13, 24, 50]. At each interaction turn \(i\), given the current utterance \({\bar{x}_{i}}\) and the interaction history \(\bar{I}[:\ i-1]\), the model generates the formal query \({\bar{y}_{i}}\). Figure 3 illustrates our architecture. We describe the base architecture, and gradually add components.

### Base Encoder-Decoder Architecture

Our base architecture uses an encoder to process the user utterance \({\bar{x}_{i}}=\langle x_{i,1},\ldots,x_{i,|{\bar{x}_{i}}|}\rangle\) and a decoder to generate the output query \({\bar{y}_{i}}\) token-by-token. This architecture does not observe the interaction history \(\bar{I}[:\ i-1]\).

The encoder computes a hidden state \({\bf h}^{E}_{j}=[{\bf h}^{\overline{E}}_{j};{\bf h}^{\overline{E}}_{j}]\) for each token \(x_{i,j}\) using a bi-directional RNN. The forward RNN is defined by: 3

Footnote 3: We omit the memory cell (often denoted as \({\bf c}_{j}\)) from all LSTM descriptions. We use only the LSTM hidden state \({\bf h}_{j}\) in other parts of the architecture unless explicitly noted.

\[{\bf h}^{\overline{E}}_{j}=\textsc{LSTM}^{\overline{E}}\left(\phi^{*}(x_{i, j});{\bf h}^{\overline{E}}_{j-1}\right)~{}, \tag{1}\]

where \(\textsc{LSTM}^{\overline{E}}\) is a long short-term memory recurrence (LSTM; Hochreiter and Schmidhuber, 1997) and \(\phi^{x}\) is a learned embedding function for input tokens. The backward RNN recurs in the opposite direction with separate parameters.

We generate the query with an RNN decoder. The decoder state at step \(k\) is:

\[{\bf h}^{D}_{k}=\textsc{lstm}^{D}\left([\phi^{v}(y_{i,k-1});{\bf c}_{k-1}];{ \bf h}^{D}_{k-1}\right)~{},\]

where \(\textsc{LSTM}^{D}\) is a two-layer LSTM recurrence, \(\phi^{y}\) is a learned embedding function for query tokens, and \({\bf c}_{k}\) is an attention vector computed from the encoder states. \(y_{i,0}\) is a special start token, and \({\bf c}_{0}\) is a zero-vector. The initial hidden state and cell memory of each layer are initialized as \({\bf h}^{E}_{|{\bar{x}_{i}}|}\) and \({\bf c}^{E}_{|{\bar{x}_{i}}|}\). The attention vector \({\bf c}_{k}\) is a weighted sum of the encoder hidden states:

\[s_{k}(j) = {\bf h}^{E}_{j}{\bf W}^{\textbf{A}}{\bf h}^{D}_{k} \tag{2}\] \[\alpha_{k} = \textrm{softmax}({\bf s}_{k})\] (3) \[{\bf c}_{k} = \sum_{j=1}^{|{\bar{x}_{i}}|}{\bf h}^{E}_{j}\alpha_{k}(j)~{}~{}, \tag{4}\]

where \({\bf W}^{\textbf{A}}\) is a learned matrix. The probabilities of output query tokens are computed as:

\[{\bf m}_{k} = \tanh\left({\bf h}^{D}_{k};{\bf c}_{k}|{\bf W}^{m}\right) \tag{5}\] \[P(y_{i,k}=w\mid{\bar{x}_{i}},{\bar{y}_{i,1:k-1}}) \propto \exp({\bf m}_{k}{\bf w}^{o}_{w}+{\bf b}^{o}_{w}) \tag{6}\]

where \({\bf W}^{m}\), \({\bf W}^{o}\), and \({\bf b}^{o}\) are learned.

### Incorporating Recent History

We provide the model with the most recent interaction history by concatenating the previous \(h\) utterances \(\langle{\bar{x}_{i-h}},...,{\bar{x}_{i-1}}\rangle\) with the current utterance in order, adding a special delimiter token between each utterance. The concatenated input provides the model access to previous utterances, but not to previously generated queries, or utterances that are more than \(h\) turns in the past. The architecture remains the same, except that the encoder and attention are computed over the concatenated sequence of tokens. The probability of an output query token is computed the same, but is now conditioned on the interaction history:

\[P(y_{i,k}=w\mid{\bar{x}_{i}},{\bar{y}_{i,1:k-1}},\bar{I}[:i-1])\propto\\ \exp({\bf m}_{k}{\bf W}^{o}_{w}+{\bf b}^{o}_{w})~{}~{}. \tag{7}\]

### Turn-level Encoder

Concatenating recent utterances to provide access to recent history has computational drawbacks. The encoding of the utterance depends on its location in the concatenated string. This requires encoding all recent history for each new utterance, and does not allow re-use of computation between utterances during encoding. It also introduces a tradeoff between computation cost and expressivity: attending over the \(h\) previous utterances allows the decoder access to the information in these utterances when generating a query, but is computationally more expensive as \(h\) increases. We address this by encoding each utterance once. To account for the influence of the interaction history on utterance encoding, we maintain a discourse state encoding \({\bf h}^{I}_{i}\) computed with a turn-level recurrence, and use it during utterance encoding. The state is maintained and updated over the entire interaction. At turn \(i\), this model has access to the complete prefix of the interaction \(\bar{I}[:i-1]\) and the current request \({\bar{x}_{i}}\). In contrast, the concatenation-based encoder (Section 4.2) has access only to information from the previous \(h\) utterances. We also use positional encoding in the attention computation to account for the position of each utterance relative to the current utterance.

Formally, we modify Equation 1 to encode \({\bar{x}_{i}}\):

\[{\bf h}^{\overline{E}}_{i,j}=\textsc{LSTM}^{\overline{E}}\left(\left[\phi^{ \varphi}(x_{i,j});{\bf h}^{I}_{i-1}\right];{\bf h}^{\overline{E}}_{i,j-1} \right)~{}~{},\]

where \({\bf h}^{I}_{i-1}\) is the discourse state following utterance \({\bar{x}_{i-1}}\). \(\textsc{LSTM}^{\overline{E}}\) is modified analogously. In contrast to the concatenation-based model, the recurrence processes a single utterance. The dis 

course state \({\bf h}_{i}^{I}\) is computed as

\[{\bf h}_{i}^{I}=\texttt{lstm}^{I}\left({\bf h}_{i,|x_{i}|;}^{E},{\bf h}_{i-1}^{ I}\right)~{}.\]

Similar to the concatenation-based model, we attend over the current utterance and the \(h\) previous utterances. We add relative position embeddings \(\phi^{I}\) to each hidden state. These embeddings are learned for each possible distance \(0,\ldots,h-1\) from the current utterance. We modify Equation 2 to index over both utterances and tokens:

\[s_{k}(t,j)=\left[{\bf h}_{t,j}^{E};\phi^{I}(i-t)\right]{\bf W}^{\mathbf{A}}{ \bf h}_{k}^{D}~{}. \tag{8}\]

In contrast to the concatenation model, without position embeddings, the attention computation has no indication of the utterance position, as our ablation shows in Section 8. The attention distribution is computed as in Equation 3, and normalized across all utterances. The position embedding is also used to compute the context vector \({\bf c}_{k}\):

### Copying Query Segments

The discourse state and attention over previous utterances allow the model to consider the interaction history when generating queries. However, we observe that context-dependent reasoning often requires generating sequences that were generated in previous turns. Figure 2 shows how segments (underlined) extracted from previous utterances are predominant in later queries. To take advantage of what was previously generated, we add copying of complete segments from previous queries by expanding the set of outputs at each generation step. This mechanism explicitly models references, reduces the number of steps required to generate queries, and provides an interpretable view of what parts of a query originate in context. Figure 3 illustrates this architecture.

Extracting SegmentsGiven the interaction history \(\bar{I}[:\ i-1]\), we construct the set of segments \(S_{i-1}\) by deterministically extracting subtrees from previously generated queries.4 In our data, we extract \(13\pm 5.9\) (\(\mu\pm\sigma\)) segments for each annotated query. Each segment \(\bar{s}\in S_{i-1}\) is a tuple \(\langle a,b,l,r\rangle\), where \(a\) and \(b\) are the indices of the first and most recent queries, \(\bar{y}_{a}\) and \(\bar{y}_{b}\), in the interaction that contain the segment. \(l\) and \(r\) are the start and end indices of the segment in \(\bar{y}_{b}\).

Footnote 4: The process of extracting sub-trees is described in the supplementary material.

Encoding SegmentsWe represent a segment \(\bar{s}=\langle a,b,l,r\rangle\) using the hidden states of an RNN encoding of the query \(\bar{y}_{b}\). The hidden states \(\langle{\bf h}^{Q}{}_{1},...,{\bf h}^{Q}{}_{|\bar{y}_{b}|}\rangle\) are computed using a bi-directional LSTM RNN similar to the utterance encoder (Equation 1), except using separate LSTM parameters and \(\phi^{y}\) to embed the query tokens. The embedded representation of a segment is a concatenation of the hidden states at the segment endpoints and an embedding of the relative position of the utterance where it appears first:

\[{\bf h}^{S}=\left[{\bf h}_{r}^{Q};{\bf h}_{r}^{Q};\phi^{g}(\min(g,i-a))\right]~ {},\]

where \(\phi^{g}\) is a learned embedding function of the position of the initial query \(\bar{y}_{a}\) relative to the current turn index \(i\). We learn an embedding for each relative position that is smaller than \(g\), and use the same embedding for all other positions.

Generation with SegmentsAt each generation step, the decoder selects between a single query token or a segment. When a segment is selected, it

Figure 3: Illustration of the model architecture during the third decoding step while processing the instruction _which ones arrive at 7pm_ from the interaction in Figure 2. The current discourse state \({\bf h}_{2}^{L}\) is used to encode the current utterance \(\bar{x}_{3}\) (Section 4.3). Query segments from previous queries are encoded into vector representations (Section 4.4). In each generation step, the decoder attends over the previous and current utterances, and a probability distribution is computed over SQL tokens and query segments. Here, segment \(\bar{s}_{1}\) is selected.

 

is appended to the generated query, an embedded segment representation for the next step is computed, and generation continues. The probability of a segment \(\bar{s}=\langle a,b,l,r\rangle\) at decoding step \(k\) is:

\[P(y_{i,k}=\bar{s}\mid\bar{x}_{i},\bar{y}_{i,1:k-1},\bar{I}[:i-1])\propto \tag{9}\] \[\exp\left(\mathbf{m}_{k}\mathbf{W}^{S}\mathbf{h}^{S}\right)~{},\]

where \(\mathbf{m}_{k}\) is computed in Equation 5 and \(\mathbf{W}^{S}\) is a learned matrix. To simplify the notation, we assign the segment to a single output token. The output probabilities (Equations 7 and 9) are normalized together to a single probability distribution. When a segment is selected, the embedding used as input for the next generation step is a bag-of-words encoding of the segment. We extend the output token function \(\phi^{y}\) to take segments:

\[\phi^{y}(\bar{s}=\langle a,b,l,r\rangle)=\frac{1}{r-l}\sum_{k=l}^{r}\phi^{y} \left(y_{b,k}\right)~{}.\]

The recursion in \(\phi^{y}\) is limited to depth one because segments do not contain other segments.

### Inference with Full Model

Given an utterance \(\bar{x}_{i}\) and the history of interaction \(\bar{I}[:~{}i-1]\), we generate the query \(\bar{y}_{i}\). An interaction starts with the user providing the first utterance \(\bar{x}_{1}\). The utterance is encoded using the initial discourse state \(\mathbf{h}_{0}^{I}\), the discourse state \(\mathbf{h}_{1}^{I}\) is computed, the query \(\bar{y}_{1}\) is generated, and the set of segments \(S_{1}\) is created. The initial discourse state \(\mathbf{h}_{0}^{I}\) is learned, and the set of segments \(S_{0}\) used when generating \(\bar{y}_{1}\) is the empty set. The attention is computed only over the first utterance because no previous utterances exist. The user then provides the next utterance or concludes the interaction. At turn \(i\), the utterance \(\bar{x}_{i}\) is encoded using the discourse state \(\mathbf{h}_{i-1}^{I}\), the discourse state \(\mathbf{h}_{i}^{I}\) is computed, and the query \(\bar{y}_{i}\) is generated using the set of segments \(S_{i-1}\). The model has no access to future utterances. We use greedy inference for generation. Figure 3 illustrates a single decoding step.

## 5 Learning

We assume access to a training set of \(N\) interactions \(\{\bar{I}^{(l)}\}_{l=1}^{N}\). Given an interaction \(\bar{I}^{(l)}\), each utterance \(\bar{x}_{i}^{(l)}\) where \(1\leq i\leq|\bar{I}^{(l)}|\), is paired with an annotated query \(\bar{y}_{i}^{(l)}\). The set of segments from previous utterances is deterministically extracted from the annotated queries during learning. However, the data does not indicate what parts of each query originate in segments copied from previous utterances. We adopt a simple approach and heuristically identify context-dependent segments based on entities that appear in the utterance and the query.5 Once we identify a segment in the annotated query, we replace it with a unique placeholder token, and it appears to the learning algorithm as a single generation decision. Treating this decision as latent is an important direction for future work. Given the segment copy decisions, we minimize the token cross-entropy loss:

Footnote 5: The alignment is detailed in the supplementary material.

\[\mathcal{L}(y_{i,k}^{(l)})=-\log P\left(y_{i,k}^{(l)}\mid\bar{x}_{i}^{(l)}, \bar{g}_{i,1:k-1}^{(l)},\hat{I}^{(l)}[:i-1]\right)~{},\]

where \(k\) is the index of the output token. The base and recent-history encoders (Sections 4.1 and 4.2) can be trained by processing each utterance separately. For these models, given a mini-batch \(\mathcal{B}\) of utterances, each identified by an interaction-utterance index pair, the loss is the mean token loss

\[\mathcal{L}=\frac{1}{\sum_{(i,j)\in\mathfrak{B}}|\bar{y}_{i}^{(j)}|}\sum_{(i, j)\in\mathfrak{B}}\sum_{k=1}^{|\bar{g}_{i}^{(j)}|}\mathcal{L}(y_{i,k}^{(j)})~{}~{}.\]

The turn-level encoder (Section 4.3) requires building a computation graph for the entire interaction. We update the model parameters for each interaction. The interaction loss is

\[\mathcal{L}=\frac{n}{B}\frac{1}{\sum_{i=1}^{n}|\bar{y}_{i}^{(j)}|}\sum_{i=1}^{ n}\sum_{k=1}^{|\bar{g}_{i}^{(j)}|}\mathcal{L}(y_{i,k}^{(j)})~{}~{},\]

where \(B\) is the batch size, and \(\frac{n}{B}\) re-normalizes the loss so the gradient magnitude is not dependent on the number of utterances in the interaction. Our ablations (-batch re-weight in Table 2) shows the importance of this term. For both cases, we use teacher forcing Williams and Zipser (1989).

## 6 Reasoning with Anonymized Tokens

An important practical consideration for generation in ATIS and other database domains is reasoning about database values, such as entities, times, and dates. For example, the first utterance in Figure 2 includes two entities and a date reference. With limited data, learning to both reason about a large number of entities and to resolve dates are challenging for neural network models. Following previous work Dong and Lapata (2016); Iyer et al. (2017), we address this with anonymization, where the data is pre- and post-processed to abstract over tokens that can be heuristically resolved to tokens in the query language. In contrast to previous work, we design a special scoring function to anonymized tokens to reflect how they are used in the input utterances. Figure 4 illustrates preprocessing in ATIS. For example, we use a temporal semantic parser to resolve dates (e.g., _next 

_Monday_) and replace them with day, month, and year placeholders. To anonymize database entries, we use a dictionary compiled from the database (e.g., to map _Seattle_ to SEATTLE). The full details of the anonymization procedure are provided in the supplementary material. Following preprocessing, the model reasons about encoding and generation of anonymized tokens (e.g., CITY#1) in addition to regular output tokens and query segments from the interaction history. Anonymized tokens are typed (e.g., CITY), map to a token in the query language (e.g., 'BOSTON' ), and appear both in input utterances and generated queries.

We modify our encoder and decoder embedding functions (\(\phi^{x}\) and \(\phi^{y}\)) to map anonymized tokens to the embeddings of their types (e.g., CITY). The type embeddings in \(\phi^{x}\) and \(\phi^{y}\) are separate. Using the types only, while ignoring the indices, avoids learning biases that arise from the arbitrary ordering of the tokens in the training data. However, it does not allow distinguishing between entries with the same type for generation decisions; for example, the common case where multiple cities are mentioned in an interaction. We address this by scoring anonymized token based on the magnitude of attention assigned to them at generation step \(k\). The attention magnitude is computed from the encoder hidden states. This computation considers both the decoder state and the location of the anonymizer tokens in the input utterances to account for how they are used in the interaction. The probability of an anonymized token \(w\) at generation step \(k\) is

\[P(y_{i,k}=w\mid\bar{x}_{i},\bar{y}_{i,1:k-1},\bar{I}[:i-1])\propto\\ \sum_{t=i-h}^{i}\sum_{j=1}^{|x_{i}|}(\exp{(s_{k}\left(t,j\right))})\]

where \(s_{k}\left(t,j\right)\) is the attention score computed in Equation 8. This probability is normalized together with the probabilities in Equations 7 and 9 to form the complete output probability.

## 7 Experimental Setup

Hyperparameters, architecture details, and other experimental choices are detailed in the supplementary material.

DataWe use ATIS Hemphill et al. (1990); Dahl et al. (1994) to evaluate our approach. The data was originally collected using wizard-of-oz experiments, and annotated with SQL queries. Each interaction was based on a scenario given to a user. We observed that the original data split shares scenarios between the train, development, and test splits. This introduces biases, where travel patterns that appeared during training repeat in testing. For example, a model trained on the original data split often correctly resolves the exact referenced by _on Saturday_ with no pre-processing or access to the document date. We evaluate this overfitting empirically in the supplementary material. We re-split the data to avoid this bias. We evenly distribute scenarios across splits so that each split contains both scenarios with many and few representative interactions. The new split follows the original split sizes with \(1148/380/130\) train/dev/test interactions. Table 1 shows data statistics. The system uses a SQL database of \(27\) tables and \(162\)K entries. \(96.6\%\) of the queries require at least one join, and \(93\%\) at least two joins. The most related work on ATIS to ours is Miller et al. (1996), which we discuss in Section 3.

The most related corpora to ATIS are SCONE Long et al. (2016) and SequentialQA Iyyer et al. (2017). SCONE Long et al. (2016) contains micro-domains consisting of stack- or list-like elements. The formal representation is linguistically-motivated and the majority of queries include a single binary predicate. All interactions include five turns. SequentialQA Iyyer et al. (2017) contains sequences of questions on a single Wikipedia table. Interactions are on average \(2.9\) turns long, and were created by re-phrasing a question from a context-independent corpus Pasupat and Liang (2015). In contrast, ATIS uses a significantly larger

\begin{table}
\begin{tabular}{|l|c|} \hline Mean/max utterances per interaction & \multicolumn{2}{c|}{\(7.0\,f\,64\)} \\ \hline Mean/max tokens per utterance & \multicolumn{2}{c|}{\(10.2\,47\)} \\ \hline Mean/max token per SQL query & \multicolumn{2}{c|}{\(102.9\,f\,1286\)} \\ \hline Input vocabulary size & \multicolumn{2}{c|}{\(1582\)} \\ \hline Output vocabulary size & \multicolumn{2}{c|}{\(982\)} \\ \hline \end{tabular}
\end{table}
Table 1: ATIS data statistics.

 

database, requires generating complex queries with multiple joins, includes longer interactions, and was collected through interaction with users. The supplementary material contains analysis of the contextual phenomena observed in ATIS.

Pre-processingWe pre-process the data to identify and anonymize entities (e.g., cities), numbers, times, and dates. We use string matching heuristics to identify entities and numbers, and identify and resolve times and dates using UWTime Lee et al. (2014). When resolving dates we use the original interaction date as the document time. The supplementary material details this process.

MetricsWe evaluate using query accuracy, strict denotation accuracy, and relaxed denotation accuracy. Query accuracy is the percentage of predicted queries that match the reference query. Strict denotation accuracy is the percentage of predicted queries that execute to exactly the same table as the reference query. In contrast to strict, relaxed gives credit to a prediction query that fails to execute if the reference table is empty. In cases when the utterance is ambiguous and there are multiple gold queries, we consider the query or table correct if they match any of the gold labels.

SystemsWe evaluate four systems: (a) seq2seq-0: the baseline encoder-decoder model (Section 4.1); (b) seq2seq-h: encoder-decoder with attention on current and previous utterances (Section 4.2); (c) s2s+anon: encoder-decoder with attention on previous utterances and anonymization scoring (Section 6); and (d) Full: the complete approach including segment copying (Section 4.4). For Full, we evaluate with predicted and gold (Full-Gold) previous queries, and without attention on previous utterances (Full-0). All models except seq2seq-0 and Full-0 use \(h=3\) previous utterances. We limit segment copying to segments that appear in the most recent query only.6 Unless specifically ablated, all experiments use pre-processing.

Footnote 6: While we only use segments from the most recent query, they often appear for the first time much earlier in the interaction, which influences their absolute position value \(a\).

## 8 Results

Table 2 shows development and test results. We run each experiment five times and report mean and standard deviation. The main metric we focus on is strict denotation accuracy. The relatively low performance of seq2seq-0 demonstrates the need for context in this task. Attending on recent history significantly increases performance. Both seq2seq models score anonymized tokens as regular vocabulary tokens. Adding anonymized token scoring further increases performance (s2s+anon). Full-0 and Full add segment copying and the turn-level encoder. The relatively high performance of Full-0 shows that substituting segment copying with attention maintains and even improves the system effectiveness. However, the best performance is provided with Full, which combines both. This shows the benefit of redundancy in accessing contextual information. Unlike the other systems, both Full and Full-0 suffer from cascading errors due to selecting query segments from previously incorrect predictions. The higher Full-Gold performance illustrates the influence of error propagation. While part of this error can be mitigated by having both attention and segment copying, this behavior is unlikely to be learned from supervised learning, where errors are never observed.

Ablations show that all components contribute to the system performance. Performance drops when using a concatenation-based encoder instead of the turn-level encoder (\(-\)turn-level enc.; Section 4.3). Using batch-reweighting (\(-\)batch-reweight; Section 5) and input position embeddings (\(-\)input pos. embs.; Section 4.3) are critical to the performance of the turn-level encoder. Removing copying of query segments

\begin{table}
\begin{tabular}{|l|p{79.7pt}|p{79.7pt}|p{79.7pt}|} \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{p{79.7pt}|}{**Query**} & \multicolumn{2}{p{79.7pt}|}{**Denotation**} \\ \cline{2-4}  & & **Relaxed** & **Strict** \\ \hline \hline \multicolumn{4}{|l|}{**Development Results**} \\ \hline \hline seq2seq-0 & \(28.7_{\pm 1.7}\) & \(48.8_{\pm 1.4}\) & \(43.2_{\pm 1.8}\) \\ \hline seq2seq-H & \(35.1_{\pm 2.2}\) & \(59.4_{\pm 2.4}\) & \(56.7_{\pm 3.2}\) \\ \hline S2s+anon & \(37.6_{\pm 0.7}\) & \(61.6_{\pm 0.7}\) & \(60.6_{\pm 0.7}\) \\ \hline Full-0 & \(36.3_{\pm 0.5}\) & \(61.5_{\pm 0.8}\) & \(61.0_{\pm 0.9}\) \\ \hline Full & \(37.5_{\pm 0.9}\) & \(63.0_{\pm 0.7}\) & \(\mathbf{62.5_{\pm 0.9}}\) \\ \hline – turn-level enc. & \(37.4_{\pm 1.5}\) & \(62.1_{\pm 2.5}\) & \(61.4_{\pm 2.7}\) \\ \hline – batch re-weight & \(36.4_{\pm 0.6}\) & \(61.8_{\pm 0.3}\) & \(61.5_{\pm 0.4}\) \\ \hline – input pos. embs. & \(33.3_{\pm 0.2}\) & \(57.9_{\pm 0.8}\) & \(57.4_{\pm 0.8}\) \\ \hline – query segments & \(36.0_{\pm 0.9}\) & \(59.5_{\pm 1.3}\) & \(58.3_{\pm 1.4}\) \\ \hline – anon. scoring & \(35.7_{\pm 0.5}\) & \(60.8_{\pm 1.1}\) & \(60.0_{\pm 1.0}\) \\ \hline – pre-processing & \(26.4_{\pm 6.1}\) & \(53.3_{\pm 8.6}\) & \(53.0_{\pm 8.5}\) \\ \hline \hline \multicolumn{4}{|l|}{Full-Gold} & \multicolumn{1}{p{79.7pt}|}{\(42.1_{\pm 0.8}\)} & \(66.6_{\pm 0.7}\) & \(66.1_{\pm 0.7}\) \\ \hline \hline \multicolumn{4}{|l|}{**Test Results**} \\ \hline seq2seq-0 & \(35.7_{\pm 1.5}\) & \(56.4_{\pm 1.1}\) & \(53.8_{\pm 1.0}\) \\ \hline seq2seq-H & \(42.2_{\pm 2.0}\) & \(66.6_{\pm 3.2}\) & \(65.8_{\pm 3.4}\) \\ \hline S2s+anon & \(44.0_{\pm 1.2}\) & \(69.3_{\pm 1.0}\) & \(68.6_{\pm 1.1}\) \\ \hline Full-0 & \(43.1_{\pm 1.3}\) & \(67.8_{\pm 1.6}\) & \(67.2_{\pm 1.6}\) \\ \hline Full & \(43.6_{\pm 1.0}\) & \(69.3_{\pm 0.8}\) & \(\mathbf{69.2_{\pm 0.8}}\) \\ \hline \hline Full-Gold & \(47.4_{\pm 1.3}\) & \(72.3_{\pm 0.5}\) & \(72.0_{\pm 0.5}\) \\ \hline \end{tabular}
\end{table}
Table 2: Mean and standard deviation development and test results, including ablations on the Full model.

 

from the interaction history lowers performance (\(-\)query segments; Section 4.4). Treating indexed anonymized tokens as regular tokens, rather than using attention-based scoring and type embeddings, lowers performance (\(-\)anon. scoring; Section 6). Finally, pre-processing, which includes anonymization, is critical (\(-\)pre-processing).

Figure 5 shows the performance as interactions progress. All systems show a drop in performance after the first utterance, which is always context-independent. As expected, seq2seq-0 shows the biggest drop. The Full approach is the most stable as the interaction progresses.

Figure 5 shows the performance as we decrease the number of previous utterances used for attention \(h\). Without the turn-level encoder and segment copying (seq2seq-h and s2s+anon), performance decreases significantly as \(h\) decreases. In contrast, the Full model shows a smaller decrease (\(1.5\%\)). The supplementary material includes attention analysis demonstrating the importance of previous-utterance attention. However, attending on fewer utterances improves inference speed: Full-0 is \(30\%\) faster than Full.

Finally, while we re-split the data due to scenario sharing between train and test early in development and used this split only for development, we also evaluate on the original split (Table 3). We report mean and standard deviation over three trials. The high performance of s2s+anon potentially indicates it benefits more from the differences between the splitting procedures.

## 9 Analysis

We analyze errors made by the full model on thirty development interactions. When analyzing the output of Full, we focus on error propagation and analyze predictions that resulted in an incorrect table when using Full, but a correct table when using Full-Gold. \(56.7\%\) are due to selection of a segment that contained an incorrect constraint. \(43.4\%\) of the errors are caused by a necessary segment missing during generation. \(93.0\%\) of all predictions are valid SQL and follow the database schema. We also analyze the errors of Full-Gold. We observe that \(30.0\%\) of errors are due to generating constraints that were not mentioned by the user. Other common errors include generating relevant constraints with incorrect values (\(23.3\%\)) and missing constraints (\(23.3\%\)).

We also evaluate our model's ability to recover long-distance references while constraints are added, changed, or removed, and when target attributes change. The supplementary material includes the analysis details. In general, the model resolves references well. However, it fails to recover constraints mentioned in the past following a user focus state change Grosz and Sidner (1986).

## 10 Discussion

We study models that recover context-dependent executable representations from user utterances by reasoning about interaction history. We observe that our segment-copying models suffer from error propagation when extracting segments from previously-generated queries. This could be mitigated by training a model to ignore erroneous segments, and recover by relying on attention for generation. However, because supervised learning does not expose the model to erroneous states, a different learning approach is required. Our analysis demonstrates that our model is relatively insensitive to interaction length, and is able to recover both explicit and implicit references to previously-mentioned entities and constraints. Further study of user focus change is required, an important phenomenon that is relatively rare in ATIS.

### Acknowledgements

This research was supported by the NSF (CRII-1656998), Schmidt Sciences, a gift from Google, and cloud computing credits from Amazon. We thank Valts Blukis, Luke Zettlemoyer, and the anonymous reviewers for their helpful comments.

Figure 5: Mean development strict denotation accuracy as function of turns and \(h\).

 



## References

* Artzi and Zettlemoyer (2011) Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. [http://www.aclweb.org/anthology/D11-1039](http://www.aclweb.org/anthology/D11-1039).
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the International Conference on Learning Representations_.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. [http://www.aclweb.org/anthology/D13-1160](http://www.aclweb.org/anthology/D13-1160).
* Cheng et al. (2017) Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata. 2017. Learning structured natural language representations for semantic parsing. In _Proceedings Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P17-1005](https://doi.org/10.18653/v1/P17-1005).
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. [https://doi.org/10.3115/vl/D14-1179](https://doi.org/10.3115/vl/D14-1179).
* Clarke et al. (2010) James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In _Proceedings of the Conference on Computational Natural Language Learning_. [http://www.aclweb.org/anthology/Wl0-2903](http://www.aclweb.org/anthology/Wl0-2903).
* Dahl et al. (1994) Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In _Proceedings of the Workshop on Human Language Technology_. [http://www.aclweb.org/anthology/H94-1010](http://www.aclweb.org/anthology/H94-1010).
* Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P16-1004](https://doi.org/10.18653/v1/P16-1004).
* Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time. _Cognitive Science_ 14:179-211.
* Giordani and Moschitti (2012) Alessandra Giordani and Alessandro Moschitti. 2012. Translating questions to SQL queries with generative parsers discriminatively reranked. In _Conference on Computational Linguistics_. [http://www.aclweb.org/anthology/Cl2-2040](http://www.aclweb.org/anthology/Cl2-2040).
* Grosz and Sidner (1986) Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. _Computational Linguistics_ 12(3). [http://www.aclweb.org/anthology/J86-3001](http://www.aclweb.org/anthology/J86-3001).
* Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In _Proceedings of the DARPA Speech and Natural Language Workshop_. [http://www.aclweb.org/anthology/H90-1021](http://www.aclweb.org/anthology/H90-1021).
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_ 9.
* Iyer et al. (2017) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayat Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In _Proceedings of the the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P17-1089](https://doi.org/10.18653/v1/P17-1089).
* Iyyer et al. (2017) Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P17-1167](https://doi.org/10.18653/v1/P17-1167).
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P16-1002](https://doi.org/10.18653/v1/P16-1002).
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In _Proceedings of the International Conference on Learning Representations_.
* Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Proceedings of theConference on Empirical Methods in Natural Language Processing_. [https://www.aclweb.org/anthology/D17-1160](https://www.aclweb.org/anthology/D17-1160).
* Khshan and Barzilay (2013) Nate Kushnan and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In _Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. [http://www.aclweb.org/anthology/N13-1103](http://www.aclweb.org/anthology/N13-1103).
* Kwiatkowski et al. (2011) Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2011. Lexical generalization in CCG grammar induction for semantic parsing. In _Proceedings of Conference on Empirical Methods in Natural Language Processing_. [http://www.aclweb.org/anthology/D11-1140](http://www.aclweb.org/anthology/D11-1140)

Kenton Lee, Yoav Artzi, Jesse Dodge, and Luke Zettlemoyer. 2014. Context-dependent semantic parsing for time expressions. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.3115/v1/P14-1135](https://doi.org/10.3115/v1/P14-1135).
* Li et al. (2017) Xiujun Li, Yun-Nung Chen, Lihong Li, and Jianfeng Gao. 2017. End-to-end task-completion neural dialogue systems. _CoRR_ abs/1703.01008.
* Liang et al. (2011) Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In _Proceedings of the Association for Computational Linguistics: Human Language Technologies_. [http://www.aclweb.org/anthology/P11-1060](http://www.aclweb.org/anthology/P11-1060).
* Ling et al. (2016) Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. _CoRR_ abs/1603.06744.
* Long et al. (2016) Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P16-1138](https://doi.org/10.18653/v1/P16-1138).
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. [https://doi.org/10.18653/v1/D15-1166](https://doi.org/10.18653/v1/D15-1166).
* Miller et al. (1996) Scott Miller, David Stallard, Robert Bobrow, and Richard Schwartz. 1996. A fully statistical approach to natural language interfaces. In _Proceedings of the Association for Computational Linguistics_. [http://www.aclweb.org/anthology/P96-1008](http://www.aclweb.org/anthology/P96-1008).
* Ng (2010) Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In _Proceedings of the Association for Computational Linguistics_. [http://www.aclweb.org/anthology/P10-1142](http://www.aclweb.org/anthology/P10-1142).
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.3115/v1/P15-1142](https://doi.org/10.3115/v1/P15-1142).
* Peng et al. (2017) Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. [https://aclanthology.info/pdf/D/D17/D17-1236.pdf](https://aclanthology.info/pdf/D/D17/D17-1236.pdf).
* Poon (2013) Hoifung Poon. 2013. Grounded unsupervised semantic parsing. In _Proceedings of the Association for Computational Linguistics_. [http://www.aclweb.org/anthology/P13-1092](http://www.aclweb.org/anthology/P13-1092).
* Popescu et al. (2004) Ana-Maria Popescu, Alex Armanasu, Oren Etzioni, David Ko, and Alexander Yates. 2004. Modern natural language interfaces to databases: Composing statistical parsing with semantic tractability. In _International Conference on Computational Linguistics_. [http://www.aclweb.org/anthology/C04-1021](http://www.aclweb.org/anthology/C04-1021).
* Rabinovich et al. (2017) Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P17-1105](https://doi.org/10.18653/v1/P17-1105).
* Serban et al. (2016) Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _Association for the Advancement of Artificial Intelligence_.
* Serban et al. (2017) Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In _Association for the Advancement of Artificial Intelligence_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Neural Information Processing Systems_.
* Tur et al. (2010) Gokhan Tur, Dilek Z. Hakkani-Tur, and Larry P. Heck. 2010. What is left to be understood in ATIS? In _Proceedings of the Spoken Language Technology Workshop_.
* Wang et al. (2014) Adrienne Wang, Tom Kwiatkowski, and Luke Zettlemoyer. 2014. Morpho-syntactic lexical generalization for CCG semantic parsing. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. [https://doi.org/10.3115/v1/D14-1135](https://doi.org/10.3115/v1/D14-1135).
* Williams and Zipser (1989) Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. _Neural Computation_ 1:270-280.
* Xu et al. (2017) Xiaojun Xu, Chang Liu, and Dawn Xiaodong Song. 2017. SQLNet: Generating structured queries from natural language without reinforcement learning. _CoRR_ abs/1711.04436.
* Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In _Proceedings of the Association for Computational Linguistics_. [https://doi.org/10.18653/v1/P17-1041](https://doi.org/10.18653/v1/P17-1041).
* Zelle and Mooney (1996) John M. Zelle and Raymond J. Mooney. 1996. Comparative results on using inductive logic programming for corpus-based parser construction. In _Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing_ 

Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In _Proceedings of the Conference on Uncertainty in Artificial Intelligence_.
* Zettlemoyer and Collins (2007) Luke S. Zettlemoyer and Michael Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In _Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. [http://www.aclweb.org/anthology/D07-1071](http://www.aclweb.org/anthology/D07-1071).
* Zettlemoyer and Collins (2009) Luke S. Zettlemoyer and Michael Collins. 2009. Learning context-dependent mappings from sentences to logical form. In _Proceedings of the Joint Conference of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing of the AFNLP_. [http://www.aclweb.org/anthology/P09-1110](http://www.aclweb.org/anthology/P09-1110).
* Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. _CoRR_ abs/1709.00103.