Venue: NAACL 2021

# Learning to Synthesize Data for Semantic Parsing

 Bailin Wang\({}^{1}\)  Wenpeng Yin\({}^{\ddagger}\) Xi Victoria Lin\({}^{\lx@sectionsign*}\) and Caiming Xiong\({}^{\ddagger}\)

\({}^{\dagger}\) University of Edinburgh, \({}^{\ddagger}\) Salesforce Research, \({}^{\lx@sectionsign}\) Facebook AI

bailin.wang@ed.ac.uk, {wyin,cxiong}@salesforce.com, victorialin@fb.com

Work done at Salesforce Research. Bailin was doing a research internship.

###### Abstract

Synthesizing data for semantic parsing has gained increasing attention recently. However, most methods require handcrafted (high-precision) rules in their generative process, hindering the exploration of diverse unseen data. In this work, we propose a generative model which features a (non-neural) PCFG that models the composition of programs (e.g., SQL), and a BART-based translation model that maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. Moreover, explicitly modeling compositions using PCFG leads to a better exploration of unseen programs, thus generate more diverse data. We evaluate our method in both in-domain and out-of-domain settings of text-to-SQL parsing on the standard benchmarks of GeoQuery and Spider, respectively. Our empirical results show that the synthesized data generated from our model can substantially help a semantic parser achieve better compositional and domain generalization.

## 1 Introduction

Recently, synthesizing data for semantic parsing has gained increasing attention Yu et al. (2018a, 2020); Zhong et al. (2020). However, these models require handcrafted rules (or templates) to synthesize new programs or utterance-program pairs. This can be sub-optimal as fixed rules cannot capture the underlying distribution of programs which usually vary across different domains Herzig and Berant (2019). Meanwhile, designing such rules also requires human involvement with expert knowledge. To alleviate this, we propose to learn a generative model from the existing data at hand. Our key observation is that programs (e.g., SQL) are formal languages that are intrinsically compositional. That is, the underlying grammar of programs is usually known and can be used to model the space of all possible programs effectively. Typically, grammars are used to constrain the program space during decoding of neural parsers Yin and Neubig (2018); Krishnamurthy et al. (2017). In this work, we utilize grammars to generate (unseen) programs, which are then used to synthesize more parallel data for semantic parsing.

Concretely, we use text-to-SQL as an example task, and propose a generative model to synthesize utterance-SQL pairs. As illustrated in Figure 1, we first employ a probabilistic context-free grammar (PCFG) to model the distribution of SQL queries. Then with the help of a SQL-to-text translation model, the corresponding utterances of SQL queries are generated subsequently. Our approach is in the same spirit as back-translation Sennrich et al. (2016). The major difference is that the 'target language', in our case, is a formal language with known underlying grammar. Just like the training of a semantic parser, the training of the data synthesizer requires a set of utterance-SQL pairs. Hence, our generative model is unlikely to be useful if it is as data-hungry as a semantic parser. Our two-stage data synthesis approach, i.e. the PCFG and the translation model, is designed to be more sample-efficient compared to a neural semantic parser. To achieve better sample efficiency, we use the non

Figure 1: A two-stage generative process for synthesizing utterance-SQL pairs.



neural parameterization of PCFG Manning and Schutze (1999) and estimate it via simple counting. For the translation model, we use the pre-trained text generation model BART Lewis et al. (2020). We sample synthetic data from the generative model to pre-train a semantic parser. The resulting parameters can presumably provide a strong compositional inductive bias in the form of initializations.

We conduct experiments on two text-to-SQL parsing datasets, namely GeoQueryZelle and Mooney (1996) and SpiderYu et al. (2018b). In the query split of GeoQuery, where training and test sets do not share SQL patterns, synthesized data helps boost the performance of a base parser by a large margin of 12.6%, leading to better _compositional generalization_ of a parser. In the cross-domain 1 setting of Spider, synthesized data also boosts the performance by 3.1% in terms of execution accuracy, resulting in better _domain generalization_ of a parser. Our work can be summarized as follows:

Footnote 1: We use the terms domain and database interchangeably.

* We propose to efficiently learn a generative model that can synthesize parallel data for semantic parsing.
* We empirically show that the synthesized data can help a neural parser achieve better compositional and domain generalization. Our code and data are available at [https://github.com/berlino/tensor2struct-public](https://github.com/berlino/tensor2struct-public).

## 2 Related Work

Data AugmentationData augmentation for semantic parsing has gained increasing attention in recent years. Dong et al. (2017) use backtranslation Sennrich et al. (2016) to obtain paraphrase of questions. Jia and Liang (2016) induce a high-precision SCFG from training data to generate more new "recombinant" examples. Yu et al. (2018a, 2020) follow the same spirit and use a handcrafted SCFG rule to generate new parallel data. However, the production rules of these approaches usually have low coverage of meaning representations. In this work, instead of using SCFG that accounts for rigid alignments between utterance and programs, we use a two-stage approach that implicitly models the alignments by taking advantage of powerful conditional text generators such as BART. In this way, our approach can generate more diverse data. The most related work to ours is GAZP Zhong et al. (2020) which synthesizes parallel data directly on test databases in the context of cross-database semantic parsing. Our work complements GAZP and shows that synthesizing data indirectly in training databases can also be beneficial for cross-database semantic parsing. Crucially, we learn the distribution of SQL programs instead of relying on handcrafted templates as in GAZP. The induced distribution helps a model explore unseen programs, leading to better compositional generalization of a parser.

Generative ModelsIn the history of semantic parsing, grammar-based generative models Wong and Mooney (2006, 2007); Zettlemoyer and Collins (2005); Lu et al. (2008) have played an important role. However, learning and inference of such models are usually expensive as they typically require grammar induction (from text to logical forms). Moreover, their grammars are designed specifically for linguistically faithful languages, e.g., logical forms, thus not suitable for programming languages such as SQL. In contrast, our generative model is more flexible and efficient to train due to the two-stage decomposition.

## 3 Method

In this section, we explain how our method can be applied to text-to-SQL parsing.

### Problem Definition

Formally, the labeled data for text-to-SQL parsing is given as a set of triples \((x,d,y)\), and each triple represents an utterance \(x\), the corresponding SQL query \(y\) and relational database \(d\). A probabilistic semantic parser is trained to maximize \(p(y|x,d)\). The goal of this work is to learn a generative model of \(q(x,y|d)\) given databases such that it can synthesize more data (i.e., triplets) for training a semantic parser \(p(y|x,d)\). Note that we use different notations \(q\) and \(p\) to represent the _generative model_ and the _discriminative parser_, respectively, where \(p(y|x,d)\) is not a posterior distribution of \(q\). Instead, \(p\) is a separate model trained with different parameterization with \(q\). This is primarily due to the intractability of posterior inference of \(q(y|x,d)\). Specifically, we use a two-stage process to model the generation of utterance-SQL pairs as follows:

\[q(x,y|d)=q(y|d)q(x|y,d)\] 

sql = (select select, cond? where) select = (agg* agg) agg = (agg_type egg_id, column col_id) agg_type = NoneAggOp | Max | Min cond = And(cond left, cond right)  | Or(cond left, cond right)  | Not(cond c)

Figure 2: A simplified ASDL grammar for SQL, where “sql, select, cond, agg” stands for variable types, “where, agg_id” for variable names, and “And, Or, Not” for constructor names.

 

_databases_ but evaluated in unseen _target databases_. Concretely, we apply our method to the Spider(Yu et al., 2018) dataset where the training contains utterance-SQL pairs from 146 source databases and the test set contains data from a disjoint set of target databases. In this out-of-domain setting, we synthesize data in the source databases in the hope that it can promote its domain generalization to unseen target databases.

TrainingAs mentioned in Section 3.4, we use pre-training to augment a semantic parser with synthesized data. Specifically, we use the following four-step training procedure: 1) train a two-stage generative model, namely \(q(x,y|d)\), 2) sample new data from it, 3) pre-train a semantic parser \(p(y|x,d)\) using the synthesized data, 4) fine-tune the parser with the target training data. In the in-domain setting, one PCFG and translation model is trained. In the out-of-domain setting, a separate PCFG is trained on each source database assuming that each database has a different distribution of SQL queries. In contrast, a single translation model is trained and shared across source databases. We use RAT-SQL Wang et al. (2020) as our base parser.

The size of the synthesized data is always proportional to the size of the original data. We tune the ratio in \(\{1,3,6,12\}\), and find that \(3\), \(6\) works best for GeoQuery and Spider respectively. We use the RAT-SQL implementation from Wang et al. (2020) which supports value prediction and evaluation by execution. We train it with the default hyper-parameters. For the SQL-to-utterance translation model, we reuse all the default hyperparameters from BART Lewis et al. (2020). Both models are trained using NVIDIA V100.

### Main Results

For GeoQuery, we report execution accuracy on the test sets of the question and query split; for Spider, we report exact set match Yu et al. (2018) along with execution accuracy on the dev set. The main results are shown in Table 1 and 2. First, we can see that compared with previous work, our base parser achieves the best performance, confirming that we are using a strong base parser to test our synthesized data.

With the pre-training using synthesized data, the performance of the base parsers is boosted in both GeoQuery and Spider. In GeoQuery, the pre-training results in the margin of 12.6% in the query split. This is somewhat expected as our generative model, especially \(q(y|d)\) directly models the composition underlying SQL queries, which helps a parser generalize better to unseen queries. Moreover, our sampled SQL queries cover around 15% test SQL queries of the query split, partially explaining why it is so beneficial for the query split. In Spider, the pre-training boosts the performance by 3.1% in terms of execution accuracy. Although our model does not synthesize data directly for target databases (which are unseen), it still helps a parser achieve better domain generalization. This contradicts the observation by Zhong et al. (2020) that synthesizing data in source databases is useless, even harmful without careful consistency calibration. We attribute this to the pre-training strategy we use, as in our preliminary experiments we found that directly mixing the synthesized data with the original training data is indeed harmful.

### Ablation Study

We try to answer two questions: a) _whether it is necessary to learn a PCFG_; b) _whether pre-trained translation model, namely BART, is required for success_. To answer the first question, we use a randomized version of \(q(y|d)\) where the probability of production rules are uniformly distributed, instead of being estimated from data in Equation (2). As

\begin{table}
\begin{tabular}{l|c c} \hline Model & Question Split & Query Split \\ \hline seq2tree Dong and Lapata (2016) & 62 \({}^{\dagger}\) & 31 \({}^{\dagger}\) \\ GECA Andreas (2020) & 68 \({}^{\dagger}\) & 49 \({}^{\dagger}\) \\ template-based Liu et al. (2018) & 55.2 & - \\ seq2seqTyler et al. (2017) & 72.5 & - \\ Base Parse\({}^{\clubsuit}\) & 70.9 & 49.5 \\ Base Parse\({}^{\clubsuit}\) + Syn Pre-Train & **74.6** & **62.1** \\ \(w..o.\) trained PCFG & 72.4 & 54.8 \\ \(w..o.\) pre-trained BART & 71.5 & 53.9 \\ \hline \end{tabular}
\end{table}
Table 1: Execution accuracies on GeoQuery. Methods with \({}^{\dagger}\) measure exact match accuracy. \(w.o.\) stands for ablating a certain component.

 

shown in Table 1 and 2, this variant (_w.o._ trained PCFG) still improves the base parsers, but with a smaller margin. This shows that a trained PCFG model is better at synthesizing useful SQL queries. To answer the second question, we use a randomly initialized SQL-to-utterance translation model instead of BART. As shown in Table 1 and 2, this variant (_w.o._ pre-trained BART) results in a drop in performance as well, indicating that pre-trained BART is crucial for synthesizing useful utterances.

### Qualitative Analysis

Table 3 shows examples of synthesized paired data for GeoQuery. In the positive examples, the sampled SQLs can be viewed as recombinations of SQLs fragments observed in the training data. For example, SELECT Sum(length) and traverse = "colorado" are SQL fragments from separate training examples. Our PCFG combines them together to form a new SQL, and the SQL-to-utterance model successfully maps it to a reasonable translation. The negative examples consist of two kinds of errors. First, the PCFG generated semantically invalid SQLs which cannot be mapped to reasonable utterances. This error is due to the independence assumption made by the PCFG. For instance, when a column and its corresponding entity is separately sampled, there is no guarantee that they form a meaningful clause, as shown in population = "mississippi". To address this, future work might consider more powerful generative models to model the dependencies within and across clauses in a SQL. Second, the SQL-to-utterances model failed to translate the sampled SQLs, as shown in the last example.

## 5 Conclusion

In this work, we propose to efficiently learn a generative model that can synthesize parallel data for semantic parsing. The synthesized data is used to pre-train a semantic parser and provide a strong inductive bias of compositionality. Empirical results on GeoQuery and Spider show that the pre-training can help a parser achieve better compositional and domain generalization.

## Acknowledgements

We would like to thank the anonymous reviewers for their valuable comments. We thank Naihao Deng for providing the preprocessed database for GeoQuery.

## References

* J. Andreas (2020)Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7556-7566. External Links: Link Cited by: SS4.
* D. Choi, M. Cheol Shin, E. Kim, and D. Ryeol Shin (2020)Ryansql: recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. arXiv preprint arXiv:2004.03125. Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1, SS2.
* Li Dong and M. Lapata (2016)Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 33-43. External Links: ISBN 978-3-319-3861-3 Cited by: SS1, SS2.
* L. Dong, J. Mallinson, S. Reddy, and M. Lapata (2017)Learning to paraphrase for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 875-886. External Links: Link Cited by: SS1, SS2.

\begin{table}
\begin{tabular}{p{142.3pt} p{142.3pt}} \hline \hline
**Sampled SQLs (\(y\))** & **Generated Utterances (\(x\))** \\ \hline SELECT length FROM river WHERE traverse = “new york” & What is the length of the river whose traverse is in New York city? \\ SELECT Sum(length) FROM river WHERE traverse = “colorado” & What is the total length of the rivers that traverse the state of Colorado? \\ SELECT state\_name FROM border\_info WHERE border = ”woyming” & What are the names of the states that have a border with Wyoming? \\ \hline SELECT state\_name FROM city WHERE population = ”mississippi” & What are the names of all cities in the state of Mississippi? \\ SELECT Min(state\_name) FROM state WHERE state\_name = ”misissippi” & What is the minimum state name of the state with the name Mississippi? \\ SELECT capital FROM state WHERE population = 15000 & What are the capitals of states with population of 150000 or more? \\ \hline \hline \end{tabular}
\end{table}
Table 3: Positive (top) and negative (bottom) examples of synthesized paired data for GeoQuery.

 

Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
* Guo et al. (2019) Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
* Herzig and Berant (2019) Jonathan Herzig and Jonathan Berant. 2019. Don't paraphrase, detect! rapid and effective data collection for semantic parsing. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3810-3820, Hong Kong, China. Association for Computational Linguistics.
* Iyer et al. (2017) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12-22, Berlin, Germany. Association for Computational Linguistics.
* Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1516-1526, Copenhagen, Denmark. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Lin et al. (2020) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-sql semantic parsing. _CoRR_, abs/2012.12627.
* Lu et al. (2008) Wei Lu, Hwee Tou Ng, Wee Sun Lee, and Luke S. Zettlemoyer. 2008. A generative model for parsing natural language to meaning representations. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, pages 783-792, Honolulu, Hawaii. Association for Computational Linguistics.
* Manning and Schutze (1999) Christopher Manning and Hinrich Schutze. 1999. _Foundations of statistical natural language processing_. MIT press.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Sennrich et al. (2016) Rico Sennrich, Barry Hadlow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Wang et al. (2020a) Bailin Wang, Mirella Lapata, and Ivan Titov. 2020a. Meta-learning for domain generalization in semantic parsing. _arXiv preprint arXiv:2010.11988_.
* Wang et al. (2020b) Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020b. RATSQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7567-7578, Online. Association for Computational Linguistics.
* Wang et al. (1997) Daniel C Wang, Andrew W Appel, Jeffrey L Korn, and Christopher S Serra. 1997. The zephyr abstract syntax description language.
* Wong and Mooney (2006) Yuk Wah Wong and Raymond Mooney. 2006. Learning for semantic parsing with statistical machine translation. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_, pages 439-446, New York City, USA. Association for Computational Linguistics.
* Wong and Mooney (2007) Yuk Wah Wong and Raymond Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_, pages 960-967, Prague, Czech Republic. Association for Computational Linguistics.
* Yin and Neubig (2018) Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In _Proceedings of the 2018 Conference on Empirical Methods in 

Natural Language Processing: System Demonstrations_, pages 7-12, Brussels, Belgium. Association for Computational Linguistics.
* Yu et al. (2020) Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-training for table semantic parsing.
* Yu et al. (2018a) Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1653-1663, Brussels, Belgium. Association for Computational Linguistics.
* Yu et al. (2018b) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
* Zelle and Mooney (1996) John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In _Proceedings of the national conference on artificial intelligence_, pages 1050-1055.
* Zettlemoyer and Collins (2005) Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In _Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence_, UAI'05, page 658-666, Arlington, Virginia, USA. AUAI Press.
* Zhong et al. (2020) Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero-shot executable semantic parsing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6869-6882, Online. Association for Computational Linguistics.

## Appendix A CFG Rules

Following Yin and Neubig (2018), we represent ASDL grammar of SQLs using a set of production rules, as illustrated in Figure 3.

Formally, a production rule \(T\) is denoted as \(N\rightarrow\zeta\), where \(N\) represents a non-terminal variable type, \(\zeta\) represents a sequence of terminal or non-terminals. We can derive a set of production rules from our pre-defined ASDL grammar by instantiating original ASDL statements. For example, "sql = (select select, cond? where)" is instantiated into two rules: "sql \(\rightarrow\) select" and "sql \(\rightarrow\) select, cond". With pre-defined production rules, a SQL can be transformed into a sequence of production rules. For example, the SQL query "select max(age)" can be represented by the sequence:

1. sql \(\rightarrow\) select
2. select \(\rightarrow\) agg
3. agg \(\rightarrow\) agg_type, column
4. agg_type \(\rightarrow\) Max
5. column \(\rightarrow\) age

Figure 3: Context-free grammars that represent the ASDL grammar in Figure 2 of the main paper. Only variable types are used in the production rules.