Venue: NAACL 2018

# Natural Language to Structured Query Generation via Meta-Learning

 Po-Sen Huang\({}^{*}\), Chenglong Wang\({}^{\dagger}\), Rishabh Singh\({}^{*}\), Wen-tau Yih\({}^{\ddagger}\), Xiaodong He\({}^{*}\)

Work performed while XH was at Microsoft Research. \({}^{*}\)Microsoft Research, \({}^{\dagger}\)University of Washington,

\({}^{\ddagger}\)Allen Institute for Artificial Intelligence, \({}^{\diamond}\)JD AI Research

pshuang@microsoft.com, clwang@cs.washington.edu, rishabb.iit@gmail.com, scottyih@allenai.org, xiaodong.he@jd.com

###### Abstract

In conventional supervised training, a model is trained to fit all the training examples. However, having a monolithic model may not always be the best strategy, as examples could vary widely. In this work, we explore a different learning protocol that treats each example as a unique _pseudo-task_, by reducing the original learning problem to a few-shot meta-learning scenario with the help of a domain-dependent _relevance function_. When evaluated on the WikiSQL dataset, our approach leads to faster convergence and achieves 1.1%-5.4% absolute accuracy gains over the non-meta-learning counterparts.

## 1 Introduction

Conventional supervised training is a pervasive paradigm for NLP problems. In this setting, a model is trained to fit all the training examples and their corresponding targets. However, while sharing the same surface form of the prediction task, examples of the same problem may vary widely. For instance, _recognizing textual entailment_ is a binary classification problem on whether the hypothesis follows a given textual statement, but the challenge datasets consist of a huge variety of inference categories and genres Dagan et al. (2013); Williams et al. (2017). Similarly, for a semantic parsing problem that maps natural language questions to SQL statements, the number of conditions in a SQL query or the length of a question can vary substantially Zhong et al. (2017).

The inherently high variety of the examples suggests an alternative training protocol: instead of learning a monolithic, one-size-fits-all model, it could be more effective to learn multiple models, where each one is designed for a specific "task" that covers a group of _similar_ examples. However, this strategy is faced with at least two difficulties. As the number of tasks increases, each task will have much fewer training examples for learning a robust model. In addition, the notion of "task", namely the group of examples, is typically not available in the dataset.

In this work, we explore this alternative learning setting and address the two difficulties by adapting the _meta-learning_ framework. Motivated by the few-shot learning scenario Andrychowicz et al. (2016); Ravi and Larochelle (2016); Vinyals et al. (2016), meta-learning aims to learn a general model that can quickly adapt to a new task given very few examples without retraining the model from scratch Finn et al. (2017). We extend this framework by effectively creating _pseudo-tasks_ with the help of a _relevance function_. During training, each example is viewed as the test example of an individual "task", where its top-\(K\) relevant instances are used as training examples for this specific task. A general model is trained for all tasks in aggregation. Similarly during testing, instead of applying the general model directly, the top-\(K\) relevant instances (in the training set) to the given test example are first selected to update the general model, which then makes the final prediction. The overview of the proposed framework is shown in Figure 1.

When empirically evaluated on a recently proposed, large semantic parsing dataset, WikiSQL Zhong et al. (2017), our approach leads to faster convergence and achieves 1.1%-5.4% absolute accuracy gain over the non-meta-learning counterparts, establishing a new state-of-the-art result. More importantly, we demonstrate how to design a relevance function to successfully reduce a regular supervised learning problem to a meta-learning problem. To the best of our knowledge, this is the first successful attempt in adapting meta-learning to a semantic task.





## 2 Background: Meta-Learning

Our work is built on the recently proposed Model-Agnostic Meta-Learning (MAML) framework Finn et al. (2017), which we describe briefly here. MAML aims to learn the _learners_ (for the tasks) and the _meta-learner_ in the few-shot meta-learning setup Vinyals et al. (2016); Andrychowicz et al. (2016); Ravi and Larochelle (2016). Formally, it considers a model that is represented by a function with parameters \(\theta\). When the model adapts to a new task \({\mathcal{T}}_{i}\), the model changes parameters \(\theta\) to \(\theta^{\prime}_{i}\), where a task contains \(K\) training examples and one or more test examples (\(K\)-shot learning). MAML updates the parameters \(\theta^{\prime}_{i}\) by one or a few rounds of gradient descent based on the training examples of task \({\mathcal{T}}_{i}\). For example, with one gradient update,

\[\theta^{\prime}_{i}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{{\mathcal{T}}_{ i}}(f_{\theta}),\]

where the step size \(\alpha\) is a hyper-parameter; \(\mathcal{L}_{{\mathcal{T}}_{i}}(f_{\theta})\) is a loss function that evaluates the error between the prediction \(f_{\theta}({\mathbf{x}}^{(j)})\) and target \({\mathbf{y}}^{(j)}\), where \({\mathbf{x}}^{(j)},{\mathbf{y}}^{(j)}\) are an input/output pair sampled from the training examples of task \({\mathcal{T}}_{i}\). Model parameters \(\theta\) are trained to optimize the performance of \(f_{\theta^{\prime}_{i}}\) on the unseen test examples from \({\mathcal{T}}_{i}\) across tasks. The meta-objective is:

The goal of MAML is to optimize the model parameters \(\theta\) such that the model can learn to adapt new tasks with parameters \(\theta^{\prime}_{i}\) via a few gradient steps on the training examples of new tasks. The model is improved by considering how the test error on unseen test data from \({\mathcal{T}}_{i}\) changes with respect to the parameters.

The meta-objective across tasks is optimized using stochastic gradient descent (SGD). The model parameters \(\theta\) are updated as follows:

\[\theta\leftarrow\theta-\beta\nabla_{\theta}\sum_{{\mathcal{T}}_{i}\sim p({ \mathcal{T}})}\mathcal{L}_{{\mathcal{T}}_{i}}(f_{\theta^{\prime}_{i}}),\]

where \(\beta\) is the meta step size.

## 3 Approach

As discussed in Section 1, to reduce traditional supervised learning to a few-shot meta-learning problem, we introduce a _relevance function_, which effectively helps group examples to form _pseudo-tasks_. Because the relevance function is problem-dependent, we first describe the semantic parsing problem below, followed by the design of our relevance function and the complete algorithm.

### The Semantic Parsing Task

The specific semantic parsing problem we study in this work is to map a natural language question to a SQL query, which can be executed against a given table to find the answer to the original question. In particular, we use the currently largest natural language questions to SQL dataset, WikiSQL Zhong et al. (2017), to develop our model and to conduct the experiments.

### Relevance Function

The intuition behind the design of a relevance function is that examples of the same type should have higher scores. For the questions to SQL problem, we design a simple relevance function that depends on (1) the predicted type of the corresponding SQL query and (2) the question length.

There are five SQL types in the WikiSQL dataset: {Count, Min, Max, Sum, Avg, Select}. We train a SQL type classifier \(f_{sql}\) using SVMs with bag-of-words features of the input question, which achieves 93.5% training accuracy and 88% test accuracy in SQL type prediction. Another soft indication on whether two questions can be viewed as belonging to the same "task" is their lengths, as they correlate to the lengths of the mapped SQL queries. The length of a question is the number of tokens in it after normal

Figure 1: Diagram of the proposed framework. (Upper) we propose using a relevant function to find a support set \(S_{K}^{(j)}\) from all training datapoints given a datapoint \(D_{j}^{\prime}\) for constructing a pseudo-task \({\mathcal{T}}_{j}\) as in the few-shot meta-learning setup. (Bottom) We optimize the model parameters \(\theta\) such that the model can learn to adapt a new task with parameters \(\theta^{\prime}_{j}\) via a few gradient steps on the training examples of the new task. The model is updated by considering the test error on the test example of the new task. See Section 2 for detail.



izing entity mentions to single tokens.1 Our relevance function only considers examples of the same predicted SQL types. If examples \({\bf x}^{(i)}\) and \({\bf x}^{(j)}\) have the same SQL type, then their relevance score is \(1-|q_{len}({\bf x}^{(i)})-q_{len}({\bf x}^{(j)})|\), where \(q_{len}\) calculates the question length. Notice that the relevance function does not need to be highly accurate as there is no formal definition on which examples should be grouped in the same pseudo-task. A heuristic-based function that encodes some domain knowledge typically works well based on our preliminary study. In principle, the relevance function can also be jointly learned with the meta-learning model, which we leave for future work.

Footnote 1: Phrases in questions that can match some table cells are treated as entities.

### Algorithm

Given a relevance function, the adaptation of the meta-learning using the MAML framework can be summarized in Algorithm 1, called Pseudo-Task MAML (PT-MAML). For each training example \({\bf x}^{(j)}\), we create a pseudo-task \(\mathcal{T}_{j}\) using the top-\(K\) relevant examples as the support set \(\mathcal{S}_{K}^{(j)}\) (Step 1). The remaining steps of the algorithm mimics the original MAML design, update task-level models (Step 8) and the meta-level, general model (Step 10) using gradient descent.

```
0: Training Datapoints \(\mathcal{D}=\{{\bf x}^{(j)},{\bf y}^{(j)}\}\)
0:\(\alpha\), \(\beta\): step size hyperparameters
0:\(K\): support set size hyperparameter
1: Construct a task \(\mathcal{T}_{j}\) with training examples using a support set \(\mathcal{S}_{K}^{(j)}\) and a test example \(\mathcal{D}_{j}^{\prime}=({\bf x}^{(j)},{\bf y}^{(j)})\).
2: Denote \(p(\mathcal{T})\) as distribution over tasks
3: Randomly initialize \(\theta\)
4:while not done do
5: Sample batch of tasks \(\mathcal{T}_{i}\sim p(\mathcal{T})\)
6:for all\(\mathcal{T}_{i}\)do
7: Evaluate \(\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta})\) using \(\mathcal{S}_{K}^{(j)}\)
8: Compute adapted parameters with gradient descent: \(\theta_{i}^{\prime}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}( f_{\theta})\)
9:endfor
10: Update \(\theta\leftarrow\theta-\beta\nabla_{\theta}\sum_{\mathcal{T}_{i}\sim p( \mathcal{T})}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta_{i}^{\prime}})\) using each \(\mathcal{D}_{i}^{\prime}\) from \(\mathcal{T}_{i}\) and \(\mathcal{L}_{\mathcal{T}_{i}}\) for the meta-update
11:endwhile
```

**Algorithm 1** Pseudo-Task MAML (PT-MAML)

## 4 Experiments

In this section, we introduce the WikiSQL dataset and preprocessing steps, the learner model in our meta-learning setup, and the experimental results.

### Dataset

We evaluate our model on the WikiSQL dataset Zhong et al. (2017). We follow the data preprocessing in Wang et al. (2017). Specifically, we first preprocess the dataset by running both tables and question-query pairs through Stanford Stanza Manning et al. (2014) using the script included with the WikiSQL dataset, which normalizes punctuations and cases of the dataset. We further normalize each question based on its corresponding table: for table entries and columns occurring in questions or queries, we normalize their format to be consistent with the table. After pre-processing, we filter the training set by removing pairs whose ground truth solution contains constants not mentioned in the question, as our model requires the constants to be copied from the question. We train and tune our model only on the filtered training and filtered development set, but we report our evaluation on the full development and test sets. We obtain 59,845 (originally 61,297) training pairs, 8,928 (originally 9,145) development pairs and 17,283 test pairs (the test set is not filtered).

### Learner Model

We use the model of Wang et al. (2017) as the _learner_ in our meta-learning setup. The model is a grammar-aware Seq2Seq encoder-decoder model with attention Cho et al. (2014); Bahdanau et al. (2014). The encoder is a bidirectional LSTM, which takes the concatenation of the table header (column names) of the queried table and the question as input to learn a joint representation. The decoder is another LSTM with attention mechanism. There are three output layers corresponding to three decoding types, which restricts the vocabulary it can sample from at each decoding step. The three decoding types are defined as follows:

* \(\tau_{V}\) (SQL operator): The output has to be a SQL operator, i.e., a terminal from \(V=\{\text{Select, From, Where, Id, Max, Min, Count, Sum, Avg, And, =, >, \(\geq\), <, \(\leq\), <END>, <GO>}.
* \(\tau_{C}\) (column name): The output has to be a column name, which will be copied from either the table header or the query section of 

the input sequence. Note that the column required for the correct SQL output may or may not be mentioned explicitly in the question.
* \(\tau_{Q}\) (constant value): The output is a constant that would be copied from the question section of the input sequence.

The grammar of SQL expressions in the the WikiSQL dataset can be described in regular expression as "Select\(f\ c\) From\(t\) Where (\(c\ op\ v\))"" (\(f\) refers to an aggregation function, \(c\) refers to a column name, \(t\) refers to the table name, \(op\) refers an comparator and \(v\) refers to a value). The form can be represented by a decoding-type sequence \(\tau_{V}\tau_{C}\tau_{U}\tau_{C}\tau_{V}(T_{C}\tau_{V}\tau_{Q})^{*}\), which will ensure only decoding-type corrected tokens can be sampled at each decoding step.

Wang et al. (2017) propose three cross-entropy based loss functions: "Pointer loss", which is the cross-entropy between target index and the chosen index, "Max loss", which computes the probability of copying a token \(v\) in the input as the maximum probability of pointers that point to token \(v\), and "Sum loss", which computes the probability of copying a token \(v\) in the input as the sum of probabilities of pointers that point to token \(v\). See Wang et al. (2017) for more detail.

### Model Hyperparameters

We use the pre-trained n-gram embeddings by Hashimoto et al. (2017) (100 dimension) and the GloVe word embedding (100 dimension) by Pennington et al. (2014); each token is embedded into a 200 dimensional vector. The encoder is a 3-layer bidirectional LSTM with hidden states of size 100, and the decoder is a 3-layer unidirectional LSTM with hidden states of size 100. The model is trained with question-query pairs with a batch size of 200 for 100 epochs. During training, we clip gradients at 5 and add gradient noise with \(\eta=0.3\), \(\gamma=0.55\) to stabilize training Neelakantan et al. (2015). We found the meta-learning model is trained stably without back-propagating to second order gradients. We select the support set size \(K\) to be 2 based on the development set. Empirically, the performance does not improve when we use a larger \(K\). We set the learning rates \(\alpha=0.001\) and \(\beta=0.1\) based on the development set. The model is implemented in Tensorflow and trained using Adagrad Duchi et al. (2011).

### Results

Table 1 shows the experimental results of our model on the WikiSQL dataset. We select the model based on the best logical form accuracy on the development set, and compare our results to augmented pointer network and the Seq2SQL model (with RL) in Zhong et al. (2017). Both logical form accuracy (denoted by \(Acc_{lf}\)) that compares the exact SQL syntax match, and the SQL execution results (denoted by \(Acc_{ex}\)) are reported. We compare our approach with its non-meta-learning counterpart using "Pointer loss", "Max loss", and "Sum loss" losses from Wang et al. (2017). Our model achieves 1.1%-5.3% and 1.2%-5.4% gains on the test set logical form and execution accuracy, respectively.

We also investigate the training and development set logical form accuracy over different epochs by "Meta + Sum loss" and "Sum loss" models. The results are shown in Figure 2. One interesting observation is that the "Meta + Sum loss" model converges much faster than the "Sum loss" model especially in the first 10 epochs. We attribute this improvement to the ability to adapt to new tasks even with a small number of training examples.

Figure 2: Logical form accuracy comparison, where “meta_train” and “meta_dev” are the train and development set accuracy using the “Meta + Sum loss” model, “train” and “dev” are the train and development set accuracy using the “Sum loss” model Wang et al. (2017).

 

We compare the logical form error on the test set between the "Sum loss" model Wang et al. (2017) and the proposed "Meta + Sum loss" model. Among the 17,283 test examples, there are 6,661 and 6,428 errors by the "Sum loss" and the "Meta + Sum loss", respectively. There are 5,190 common errors by both models. We examine the test examples where "Sum loss" is correct while "Meta + Sum loss" is not and vice versa, shown in Figure 3. We observe that the differences are mainly in ground truth SQL length = 7 and 10, where the "Meta + Sum loss" model outperforms "Sum loss" model by a large margin. We show some examples for the two cases in the supplementary material.

## 5 Related Work

**Meta Learning** One popular direction of meta-learning Thrun and Pratt (1998); Schmidhuber (1987); Naik and Mammone (1992) is to train a meta-learner that learns how to update the parameters of the learners model Bengio et al. (1992); Schmidhuber (1992). This direction has been applied to learning to optimize deep neural networks Hochreiter et al. (2001); Andrychowicz et al. (2016); Li and Malik (2017); Ha et al. (2017). Few-shot learning methods have also adapted meta-learning approaches for image recognition Koch (2015); Ravi and Larochelle (2016); Vinyals et al. (2016) and reinforcement learning Finn et al. (2017). Given that the few-shot learning setup cannot directly work in standard supervised learning problems, we explore reducing a regular supervised learning problem to the few-shot meta-learning scenario by creating pseudo-tasks with a relevance function.

**Semantic Parsing** Mapping natural language to logic forms has been actively studied in natural language processing research Zettlemoyer and Collins (2005); Giordani and Moschitti (2010); Artzi and Zettlemoyer (2011); Berant et al. (2013); Vlachos and Clark (2014); Yih et al. (2014, 2015); Wang et al. (2015); Golub and He (2016); Iyer et al. (2017); Krishnamurthy et al. (2017). However, unlike conventional approaches, which fit one model for all training examples, the proposed approach learns to adapt to new tasks. By using the support set based on the relevance function, the proposed model can adapt to a unique model for each example.

**Program Induction / Synthesis** Program induction Reed De Freitas (2016); Neelakantan et al. (2015); Graves et al. (2014); Yin et al. (2015); Devlin et al. (2017) aims to infer latent programs given input/output examples, while program synthesis models Zhong et al. (2017); Parisotto et al. (2017) aim to generate explicit programs and then execute them to get output. The learner model we used in this work follows the line of program synthesis models and trains on pairs of natural language (question) and program (SQL) directly.

## 6 Conclusion

In this paper, we propose a new learning protocol that reduces a regular supervised learning problem to the few-shot meta-learning scenario. This is done by effectively creating _pseudo-tasks_ with the help of a _relevance function_. When evaluated on the newly released, large semantic parsing dataset, WikiSQL, our approach leads to faster convergence and enjoys 1.1%-5.4% absolute accuracy gains over the non-meta-learning counterparts, achieving a new state-of-the-art result.

While the initial finding is encouraging, we believe the potential of this meta-learning framework has not yet been fully realized. In the future, we plan to explore more variations of the meta-learning setup, such as using different relevance functions, including the ones that are jointly learned. We also would like to understand this approach better by testing it on more natural language processing tasks.

## Acknowledgments

We thank Chelsea Finn and Eugene Brevdo for helpful discussions in meta-learning, and Adith Swaminathan, Asli Celikyilmaz, and anonymous reviewers for their valuable feedback.

Figure 3: Logical form accuracy comparison. “Meta + Sum loss (o), Sum loss (x)” indicates the generated SQL is incorrect by the “Sum loss” model and is correct by the “Meta + Sum loss” model. Similarly, “Meta + Sum loss (x), Sum loss (o)” indicates the generated SQL is incorrect by the “Meta + Sum loss” model and is correct by the “Sum loss” model.

 



## References

* Andrychowicz et al. (2016) Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. 2016. Learning to learn by gradient descent by gradient descent. In _Advances in Neural Information Processing Systems_. pages 3981-3989.
* Artzi and Zettlemoyer (2011) Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. pages 421-432.
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_ .
* Bengio et al. (1992) Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. 1992. On the optimization of a synaptic learning rule. In _Preprints Conf. Optimality in Artificial and Biological Neural Networks_. Univ. of Texas, pages 6-8.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_ .
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanotto. 2013. _Recognizing Textual Entailment: Models and Applications_. Morgan and Claypool.
* Devlin et al. (2017) Jacob Devlin, Rudy R. Bunel, Rishabh Singh, Matthew J. Hausknecht, and Pushmeet Kohli. 2017. Neural program meta-induction. In _NIPS_. pages 2077-2085.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of Machine Learning Research_ 12(July):2121-2159.
* Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In _ICML_.
* Giordani and Moschitti (2010) Alessandra Giordani and Alessandro Moschitti. 2010. Semantic mapping between natural language questions and SQL queries via syntactic pairing. In _Proceedings of the 14th International Conference on Applications of Natural Language to Information Systems_. pages 207-221.
* Golub and He (2016) David Golub and Xiaodong He. 2016. Character-level question answering with attention. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing machines. _arXiv preprint arXiv:1410.5401_ .
* Ha et al. (2017) David Ha, Andrew Dai, and Quoc V Le. 2017. Hypernetworks. In _International Conference on Learning Representations_.
* Hashimoto et al. (2017) Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Hochreiter et al. (2001) Sepp Hochreiter, A Steven Younger, and Peter R Conwell. 2001. Learning to learn using gradient descent. In _International Conference on Artificial Neural Networks_. pages 87-94.
* Iyer et al. (2017) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. _arXiv preprint arXiv:1704.08760_ .
* Koch (2015) Gregory Koch. 2015. Siamese neural networks for one-shot image recognition. In _ICML Deep Learning Workshop_.
* Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. pages 1517-1527.
* Li and Malik (2017) Ke Li and Jitendra Malik. 2017. Learning to optimize neural nets. In _International Conference on Learning Representations_.
* Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Association for Computational Linguistics (ACL) System Demonstrations_.
* Naik and Mammone (1992) Devang K Naik and RJ Mammone. 1992. Meta-neural networks that learn by learning. In _International Joint Conference on Neural Networks_. IEEE, volume 1, pages 437-442.
* Neelakantan et al. (2015) Arvind Neelakantan, Luke Vilnis, Quoc V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. 2015. Adding gradient noise improves learning for very deep networks. _arXiv preprint arXiv:1511.06807_ .
* Parisotto et al. (2017) Emilio Parisotto, Abdel-rahman Mohamed, Rishabb Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017. Neuro-symbolic program synthesis. In _International Conference on Learning Representations (ICLR)_

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_. pages 1532-1543.
* Ravi and Larochelle (2016) Sachin Ravi and Hugo Larochelle. 2016. Optimization as a model for few-shot learning. In _International Conference on Learning Representations (ICLR)_.
* Reed and De Freitas (2016) Scott Reed and Nando De Freitas. 2016. Neural programmer-interpreters. In _International Conference on Learning Representations (ICLR)_.
* Schmidhuber (1987) Jurgen Schmidhuber. 1987. _Evolutionary Principles in Self-Referential Learning. On Learning now to Learn: The Meta-Meta-Meta...Hook_. Diploma thesis, Technische Universitat Munchen, Germany.
* Schmidhuber (1992) Jurgen Schmidhuber. 1992. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. _Neural Computation_ 4(1):131-139.
* Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt, editors. 1998. _Learning to Learn_. Kluwer Academic Publishers.
* Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In _Advances in Neural Information Processing Systems_. pages 3630-3638.
* Vlachos and Clark (2014) Andreas Vlachos and Stephen Clark. 2014. A new corpus and imitation learning framework for context-dependent semantic parsing. _Transactions of the Association for Computational Linguistics_ 2:547-559.
* Wang et al. (2017) Chenglong Wang, Marc Brockschmidt, and Rishabh Singh. 2017. Pointing out SQL queries from text. Technical Report MSRTR-2017-45. [https://www.microsoft.com/en-us/research/publication/pointing-sql-queries-text/](https://www.microsoft.com/en-us/research/publication/pointing-sql-queries-text/).
* Wang et al. (2015) Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In _Association for Computational Linguistics (ACL)_.
* Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. _CoRR_ abs/1704.05426.
* Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _ACL-IJCNLP_.
* Yih et al. (2014) Wen-tau Yih, Xiaodong He, and Chris Meek. 2014. Semantic parsing for single-relation question answering. In _ACL_.
* Yin et al. (2015) Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2015. Neural enquirer: Learning to query tables with natural language. _arXiv preprint arXiv:1512.00965_ .
* Zettlemoyer and Collins (2005) Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In _Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence_. pages 658-666.
* Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. _CoRR_ abs/1709.00103.