Venue: NAACL 2018

# TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation

 Tao Yu

Yale University

tao.yu@yale.edu &Zifan Li

Yale University

zifan.li@yale.edu &Zilin Zhang

Yale University

zilin.zhang@yale.edu Rui Zhang

Yale University

r.zhang@yale.edu &Dragomir Radev

Yale University

dragomir.radev@yale.edu

###### Abstract

Interacting with relational databases through natural language helps users of any background easily query and analyze a vast amount of data. This requires a system that understands users' questions and converts them to SQL queries automatically. In this paper we present a novel approach, TypeSQL, which views this problem as a slot filling task. Additionally, TypeSQL utilizes type information to better understand rare entities and numbers in natural language questions. We test this idea on the WikiSQL dataset and outperform the prior state-of-the-art by 5.5% in much less time. We also show that accessing the content of databases can significantly improve the performance when users' queries are not well-formed. TypeSQL gets 82.6% accuracy, a 17.5% absolute improvement compared to the previous content-sensitive model.

## 1 Introduction

Building natural language interfaces to relational databases is an important and challenging problem [9, 30, 4, 28, 14]. It requires a system that is able to understand natural language questions and generate corresponding SQL queries. In this paper, we consider the WikiSQL task proposed by zhong2017using, a large scale benchmark dataset for the text-to-SQL problem. Given a natural language question for a table and the table's schema, the system needs to produce a SQL query corresponding to the question.

We introduce a knowledge-based type-aware text-to-SQL generator, TypeSQL. Based on the prior state-of-the-art SQLNet [26], TypeSQL employs a sketch-based approach and views the task as a slot filling problem (Figure 2). By grouping different slots in a reasonable way and capturing relationships between attributes, TypeSQL outperforms SQLNet by about 3.5% in half of the original training time.

Furthermore, natural language questions often contain rare entities and numbers specific to the underlying database. Some previous work [1] already shows those words are crucial to many downstream tasks, such as infering column names and condition values in the SQL query. However, most of such key words lack accurate embeddings in popular pre-trained word embedding models. In order to solve this problem, TypeSQL assigns each word a type as an entity from either the knowledge graph, a column or a number. For example, for the question in Figure 1, we label "mort drucker" as person according to our knowledge graph; "spooded title," "artist" and "issue" as column since they are column names; and "88.5" as float. Incorporating this type information, TypeSQL further improves the state-of-the-art performance by about another 2% on the WikiSQL dataset, resulting in a final 5.5% improvement in total.

Moreover, most previous work assumes that user queries contain exact column names and entries. However, it is unrealistic that users always formulate their questions with exact column names and string entries in the table. To tackle this issue, when scaleability and privacy are not of a concern, the system needs to search databases to better understand what the user is querying. Our content-sensitive model TypeSQL + TC gains roughly 9% improvement compared to the content-insensitive model, and outperforms the previous content-sensitive model by 17.5%.

## 2 Related Work

Semantic parsing maps natural language to meaningful executable programs. The programs could

be a range of representations such as logic forms (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Wong and Mooney, 2007; Das et al., 2010; Liang et al., 2011; Banarescu et al., 2013; Artzi and Zettlemoyer, 2013; Reddy et al., 2014; Berant and Liang, 2014; Pasupat and Liang, 2015). Another area close to our task is code generation. This task parses natural language descriptions into a more general-purpose programming language such as Python (Allamanis et al., 2015; Ling et al., 2016; Rabinovich et al., 2017; Yin and Neubig, 2017).

As a sub-task of semantic parsing, the text-to-SQL problem has been studied for decades (Warren and Pereira, 1982; Popescu et al., 2003, 2004; Li et al., 2006; Giordani and Moschitti, 2012; Wang et al., 2017b). The methods of the Database community (Li and Jagadish, 2014; Yaghmazadeh et al., 2017) involve more hand feature engineering and user interactions with the systems. In this work, we focus on recent neural network based approaches (Yin et al., 2016; Zhong et al., 2017; Xu et al., 2017; Wang et al., 2017a; Iyer et al., 2017). Dong and Lapata (2016) introduce a sequence-to-sequence approach to converting text to logical forms. Most of previous work focus on specific table schemas, which means they use a single database in both train and test. Thus, they don't generalize to new databases. Zhong et al. (2017) publish the WikISQL dataset and propose a sequence-to-sequence model with reinforcement learning to generate SQL queries. In the problem definition of the WikISQL task, the databases in the test set do not appear in the train and development sets. Also, the task needs to take different table schemas into account. Xu et al. (2017) further improve the results by using a SQL sketch based approach employing a sequence-to-set model.

## 3 Methodology

Like SQLNet, we employ a sketch-based approach and format the task as a slot filling problem. Figure 2 shows the SQL sketch. Our model needs to predict all slots that begin with SS in Figure 2.

Figure 1 illustrates the architecture of TypeSQL on the right and a detailed overview of one of three main models model_col on the left. We first preprocess question inputs by type recognition (Section 3.1). Then we use two bi-directional LSTMs to encode words in the question with their types and the column names separately (Section 3.2). The output hidden states of LSTMs are then used to predict the values for the slots in the SQL sketch (Section 3.3).

### Type Recognition for Input Preprocessing

In order to create one-to-one type input for each question, we, first, tokenize each question into \(n\)-grams of length 2 to 6, and use them to search over the table schema and label any column name appears in the question as column. Then, we assign numbers and dates in the question into four self

Figure 1: TypeSQL consists of three slot-filling models on the right. We only show model_col on the left for brevity. model_agg and model_opval have the similar pipelines.

Figure 2: SQL Sketch. The tokens starting with “S” are slots to fill. ‘**’ indicates zero or more **AND** clauses.

 

explanatory categories: integer, Float, date, and year. To identify named entities, we search for five types of entities: person, place, country, organization, and sport, on Freebase1 using grams as keyword queries. The five categories cover a majority of entities in the dataset. Thus, we do not use other entity types provided by Freebase. Domain-specific knowledge graphs can be used for other applications.

Footnote 1: [https://developers.google.com/freebase/](https://developers.google.com/freebase/)

In the case where the content of databases is available, we match words in the question with both the table schema and the content and labels of the columns as column and match the entry values as the corresponding column names. For example, the type in the Figure 1 would be [none, column, column, none, artist, artist, none, none, column, none, column, issue, none] in this case. Other parts in the Figure 1 keep the same as the content-insensitive approach.

### Input Encoder

As shown in the Figure 1, our input encoder consists of two bi-directional LSTMs, bi-LSTMQT and bi-LSTMCOL. To encode word and type pairs of the question, we concatenate embeddings of words and their corresponding types and input them to bi-LSTMQT. Then the output hidden states are \({\bf H}_{\textsc{gt}}\) and \({\bf H}_{\textsc{col}}\), respectively.

For encoding column names, SQLNet runs a bi-directional LSTM over each column name. We first average the embeddings of words in the column name. Then, we run a single bi-LSTMCOL between column names. This encoding method improves the result by 1.5% and cuts the training time by half. Even though the order of column names does not matter, we attribute this improvement to the fact that the LSTM can capture their occurrences and relationships.

### Slot-Filling Model

Next, we predict values for the slots in the SQL sketch. For the slots in Figure 2, SQLNet has a separate model for each of them which do not share their trainable parameters. This creates five models for the five slots and one model for SSDND# (12 bi-LSTMs in total). However, since the predict procedures of SSEECT_COL, SSCOND_COL, and SSCD# are similar, we combine them into a single model. Additionally, SSCOND_COL depends on the output of SSELECT_COL, which reduces errors of predicting the same column in these two slots SSCOND_COL Moreover, we group SOP and SSCOND_VAL together because both depend on the outputs of SSCOND_COL. Furthermore, we use one model for SAGG because we notice that the SAGG model converges much faster and suffers from overfitting when combined with other models. Finally, TypeSQL consists of three models (Figure 1 right):

* model_col for SSELECT_COL, SSCD# and SSCOND_COL
* model_agg for SAGG
* model_opval for SOP and SSCOND_VAL

where the parameters of Bi-LSTMQT and bi-LSTMCOL are shared in each model (6 bi-LSTMs in total).

Since all three models use the same way to compute the weighted question and type representation \({\bf H}_{\textsc{qf/conl}}\) using the column attention mechanism proposed in SQLNet, we first introduce the following step in all three models:

\[\alpha_{\textsc{qf/conl}}=\textbf{softmax}({\bf H}_{\textsc{ col}}{\bf W}_{ct}{\bf H}^{\top}_{\textsc{qt}})\] \[{\bf H}_{\textsc{qf/conl}}=\alpha_{\textsc{qf/conl}}{\bf H}_{ \textsc{qf}}\]

where **softmax** applies the softmax operator over each row of the input matrix, \(\alpha_{\textsc{qf/conl}}\) is a matrix of attention scores, and \({\bf H}_{\textsc{qf/conl}}\) is the weighted question and type representation. In our equations, we use \({\bf W}\) and \({\bf V}\) to represent all trainable parameter matrices and vectors, respectively.

model_col-$select_COLHOT COL is used to predict the column name in the SSEECT_COL:

\[s={\bf V}^{sel}\textbf{tanh}({\bf W}^{sel}_{c}{\bf H}^{\top}_{ \textsc{coll}}+{\bf W}^{sel}_{qt}{\bf H}^{\top}_{\textsc{qf/conl}})\] \[P_{sel\_coli}=\textbf{softmax}(s)\]

model_col-$COND# Unlike SQLNet, we compute number of conditions in the where in a simpler way:

\[P_{num}=\textbf{softmax}\left({\bf V}^{num}\textbf{tanh}({\bf W}^{ num}_{ql}\sum_{i}{\bf H}^{\top}_{\textsc{qf/col}_{i}})\right)\]

We set the maximum number of conditions to 4.

 

Model_col-$cond.colWe find that SQLNet often selects the same column name in the $COND_COL as SSSELECT_COL, which is incorrect in most cases. To avoid this problem, we pass the weighted sum of question and type hidden states conditioned on the column chosen in SSELECT_COL\({}_{\text{QT/SCOL}}\) (expended as the same shape of \({\bf H}_{\text{QT/COL}}\)) to the prediction:

\[c={\mathbf{V}^{col}}{\textbf{tanh}}({\mathbf{W}_{c}^{col}}{\mathbf{H}_{\text{ Cot}}^{\top}}+{\mathbf{W}_{qt}^{col}}{\mathbf{H}_{\text{GT/COL}}^{\top}}+{ \mathbf{W}_{qt}^{scal}}{\mathbf{H}_{\text{Qtfscol}}^{\top}})\\ P_{cond.col}=\textbf{softmax}(c)\]

model_agg-$AggGiven the weighted sum of question and type hidden states conditioned on the column chosen in SSELECT_COL\({}_{\text{QT/SCOL}}\), SAGG is chosen from {null, max, min, count, sum, avg} in the same way as SQLNet:

\[P_{agg}=\textbf{softmax}\left({\mathbf{V}^{agg}}{\textbf{ tanh}}({\mathbf{W}_{qt}^{agg}}{\mathbf{H}_{\text{QT/SCOL}}^{\top}})\right)\]

model_opval-$opFor each predicted condition column, we choose a $OP from \(\{=,>,<\}\) by:

\[P_{gp}=\textbf{softmax}\left({\mathbf{W}_{t}^{vp}}{\textbf{ tanh}}({\mathbf{W}_{c}^{op}}{\mathbf{H}_{\text{Col}}^{\top}}+{\mathbf{W}_{ qt}^{op}}{\mathbf{H}_{\text{GT/COL}}^{\top}})\right)\]

model_opval-$cond.ValThen, we need to generate a substring from the question for each predicted column. As in SQLNet, a bi-directional LSTM is used for the encoder. It employs a pointer network Vinyals et al. (2015) to compute the distribution of the next token in the decoder. In particular, the probability of selecting the \(i\)-th token \(w_{i}\) in the natural language question as the next token in the substring is computed as:

\[v={\mathbf{V}_{t}^{val}}{\textbf{tanh}}({\mathbf{W}_{qt}^{val}}{ \mathbf{H}_{\text{QT}}^{i}}+{\mathbf{W}_{c}^{val}}{\mathbf{H}_{\text{COL}}}+{ \mathbf{W}_{h}^{val}}{\mathbf{h}})\\ P_{cond.val}=\textbf{softmax}(v)\]

where \({\mathbf{h}}\) is the hidden state of the previously generated token. The generation process continues until the \(\langle\texttt{END}\rangle\) token is the most probable next token of the substring.

## 4 Experiments

DatasetWe use the WikiSQL dataset Zhong et al. (2017), a collection of 87,673 examples of questions, queries, and database tables built from 26,521 tables. It provides train/dev/test splits such that each table is only in one split. This requires model to generalize to not only new questions but new table schemas as well.

Implementation DetailsWe implement our model based on SQLNet Xu et al. (2017) in Py

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline  & \multicolumn{3}{c|}{Dev} & \multicolumn{3}{c|}{Test} \\ \cline{2-9}  & Acctr & Accgr & Accgr & Accgr & Accgr & Accgr \\ \hline \multicolumn{9}{|c|}{Content insensitive} \\ \hline Dong and Lapata (2016) & 23.3\% & - & 37.0\% & 23.4\% & - & 35.9\% \\ \hline Augmented Pointer Network Zhong et al. (2017) & 44.1\% & - & 53.8\% & 42.8\% & - & 52.8\% \\ \hline Seq2SQL Zhong et al. (2017) & 49.5\% & - & 60.8\% & 48.3\% & - & 59.4\% \\ \hline SQLNet Xu et al. (2017) & - & 63.2\% & 69.8\% & - & 61.3\% & 68.0\% \\ \hline TypeSQL w/o type-awareness (ours) & - & 66.5\% & 72.8\% & - & 64.9\% & 71.7\% \\ \hline TypeSQL (ours) & - & **68.0\%** & **74.5\%** & - & **66.7\%** & **73.5\%** \\ \hline \multicolumn{9}{|c|}{Content Sensitive} \\ \hline Wang et al. Wang et al. (2017a) & 59.6\% & - & 65.2\% & 59.5\% & - & 65.1\% \\ \hline TypeSQL+TC (ours) & - & **79.2\%** & **85.5\%** & - & **75.4\%** & **82.6\%** \\ \hline \end{tabular}
\end{table}
Table 1: Overall results on WikiSQL. Accgr, Accgr, and Accgr denote the accuracies of exact string, canonical representation, and execute result matches between the synthesized SQL with the ground truth respectively. The top six results are content-insensitive, which means only the question and table schema are used as inputs. The bottom two are content-sensitive, where the models use the question, the table schema, and the content of databases.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline  & \multicolumn{3}{c|}{Dev} & \multicolumn{3}{c|}{Test} \\ \cline{2-7}  & Accgr & Accgr & Accgr & Accgr & Accgr \\ \hline Seq2SQL Zhong et al. (2017) & 90.0\% & 89.6\% & 62.1\% & 90.1\% & 88.9\% & 60.2\% \\ \hline SQLNet Xu et al. (2017) & 90.1\% & 91.5\% & 74.1\% & 90.3\% & 90.9\% & 71.9\% \\ \hline TypeSQL (ours) & 90.3\% & **93.1\%** & **78.5\%** & 90.5\% & **92.2\%** & **77.8\%** \\ \hline \hline TypeSQL+TC (ours) & 90.3\% & **93.5\%** & **92.8\%** & 90.5\% & **92.1\%** & **87.9\%** \\ \hline \end{tabular}
\end{table}
Table 2: Breakdown results on WikiSQL. Accag, Accgr, and Accgr are the accuracies of canonical representation matches on aggregator, select column, and where clauses between the synthesized SQL and the ground truth respectively.

 

Torch (Paszke et al., 2017). We concatenate pretrained Glove Pennington et al. (2014) and paraphrase Wieting and Gimpel (2017) embeddings. The dimensions and dropout rates of all hidden layers are set to 120 and 0.3 respectively. We use Adam Kingma and Ba (2015) with the default hyperparameters for optimization. The batch size is set to 64. The same loss functions in Xu et al. (2017) are used. Our code is available at [https://github.com/taoyds/typesql](https://github.com/taoyds/typesql).

Results and DiscussionTable 1 shows the main results on the WikiSQL task. We compare our work with previous results using the three evaluation metrics used in Xu et al. (2017). Table 2 provides the breakdown results on aggregation, selection, and where clauses.

Without looking at the content of databases, our model outperforms the previous best work by 5.5% on execute accuracy. According to Table 2, TypeSQL improves the accuracy of SELECT by 1.3% and WHERE clause by 5.9%. By encoding column names and grouping model components in a simpler but reasonable way, TypeSQL achieves a much higher result on the most challenging subtask where clause. Also, the further improvement of integrating word types shows that TypeSQL could encode the rare entities and numbers in a better way.

Also, if complete access to the database is allowed, TypeSQL can achieve 82.6% on execute accuracy, and improves the performance of the previous content-aware system by 17.5%. Although Zhong et al. (2017) enforced some limitations when creating the WikiSQL dataset, there are still many questions that do not have any column name and entity indicator. This makes generating the right SQLs without searching the database content in such cases impossible. This is not a critical problem for WikiSQL but is so for most real-world tasks.

## 5 Conclusion and Future Work

We propose TypeSQL for text-to-SQL which views the problem as a slot filling task and uses type information to better understand rare entities and numbers in the input. TypeSQL can use the database content to better understand the user query if it is not well-formed. TypeSQL significantly improves upon the previous state-of-the-art on the WikiSQL dataset.

Although, unlike most of the previous work, the WikiSQL task requires model to generalize to new databases, the dataset does not cover some important SQL operators such as JOIN and GROUP BY. This limits the generalization of the task to other SQL components. In the future, we plan to advance this work by exploring other more complex datasets under the database-split setting. In this way, we can study the performance of a generalized model on a more realistic text-to-SQL task which includes many complex SQL and different databases.

## Acknowledgement

We thank Alexander Fabbri for his help in reviewing.

## References

* Agrawal and Srikant (2003) Rakesh Agrawal and Ramakrishnan Srikant. 2003. Searching with numbers. _IEEE Trans. Knowl. Data Eng._, 15(4):855-870.
* Allamanis et al. (2015) Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language. In _ICML_, volume 37 of _JMLR Workshop and Conference Proceedings_, pages 2123-2132. JMLR.org.
* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics_.
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_.
* Berant and Liang (2014) Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1415-1425, Baltimore, Maryland. Association for Computational Linguistics.
* Das et al. (2010) Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A. Smith. 2010. Probabilistic frame-semantic parsing. In _NAACL_.
* Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_.
* Giordani and Moschitti (2012) Alessandra Giordani and Alessandro Moschitti. 2012. Translating questions to sql queries with generative parserminatively reranked. In _COLING (Posters)_, pages 401-410.

 

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. _CoRR_, abs/1704.08760.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. _The 3rd International Conference for Learning Representations, San Diego_.
* Li and Jagadish (2014) Fei Li and HV Jagadish. 2014. Constructing an interactive natural language interface for relational databases. _VLDB_.
* Li et al. (2006) Yunyao Li, Huhai Yang, and HV Jagadish. 2006. Constructing a generic natural language interface for an xml database. In _EDBT_, volume 3896, pages 737-754. Springer.
* Liang et al. (2011) P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In _Association for Computational Linguistics (ACL)_, pages 590-599.
* Ling et al. (2016) Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In _ACL (1)_. The Association for Computer Linguistics.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers_, pages 1470-1480.
* Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. _NIPS 2017 Workshop_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, pages 1532-1543. ACL.
* Popescu et al. (2004) Ana-Maria Popescu, Alex Armanasu, Oren Etzioni, David Ko, and Alexander Yates. 2004. Modern natural language interfaces to databases: Composing statistical parsing with semantic tractability. In _Proceedings of the 20th international conference on Computational Linguistics_, page 141. Association for Computational Linguistics.
* Popescu et al. (2003) Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In _Proceedings of the 8th international conference on Intelligent user interfaces_, pages 149-157. ACM.
* Rabinovich et al. (2017) Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In _ACL (1)_, pages 1139-1149. Association for Computational Linguistics.
* Reddy et al. (2014) Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. _Transactions of the Association for Computational Linguistics_, 2:377-392.
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_, pages 2692-2700. Curran Associates, Inc.
* Wang et al. (2017a) Chenglong Wang, Marc Brockschmidt, and Rishabh Singh. 2017a. Pointing out sql queries from text. _Technical Report_.
* Wang et al. (2017b) Chenglong Wang, Alvin Cheung, and Rastislav Bodik. 2017b. Synthesizing highly expressive sql queries from input-output examples. In _Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation_, pages 452-466. ACM.
* Warren and Pereira (1982) David HD Warren and Fernando CN Pereira. 1982. An efficient easily adaptable system for interpreting natural language queries. _Computational Linguistics_, 8(3-4):110-122.
* Wieting and Gimpel (2017) John Wieting and Kevin Gimpel. 2017. Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. _arXiv preprint arXiv:1711.05732_.
* Wong and Mooney (2007) Yuk Wah Wong and Raymond J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-2007)_, Prague, Czech Republic.
* Xu et al. (2017) Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. _arXiv preprint arXiv:1711.04436_.
* Yaghmazadeh et al. (2017) Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Solizer: Query synthesis from natural language. _Proc. ACM Program. Lang._, 1(OOPSLA):63:1-63:26.
* Yin et al. (2016) Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2016. Neural enquirer: Learning to query tables in natural language. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016_, pages 2308-2314.
* Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In _ACL (1)_, pages 440-450. Association for Computational Linguistics 

John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In _AAAI/IAAI_, pages 1050-1055, Portland, OR. AAAI Press/MIT Press.
* Zettlemoyer and Collins (2005) Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. _UAI_.
* Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. _CoRR_, abs/1709.00103.