SlowGuess/NAACL_2022_OCR huggingface dataset:
tating new in-domain examples. Our first step to address this issue is to evaluate the coverage of SQL synthesis on predicted QDMRs, for out-of-domain questions. As question domains in SPIDER dev differ from those in its training set, it serves as our initial testbed. We further explore this setting in §4.3.4. Our QDMR parser (§4.1) is fine-tuned on BREAK for 10 epochs and we select the model with highest exact string match (EM) on BREAK dev. Evaluating on the hidden test set reveals our model scores 42.3 normalized EM, setting the state-of-the-art on BREAK. The predicted QDMRs, are then used in SQL synthesis together with examples  $\langle x_i, a_i, D_i \rangle$ . In Table 3, the last row shows that coverage on SPIDER dev is nearly identical to that of manually annotated QDMRs (77.6% to 77.2%).
4.3 Training Text-to-SQL Models
Next, we compare text-to-SQL models trained on our synthesized data to training on expert annotated SQL. Given examples  $\langle x_i, D_i \rangle$  we test the following settings: (1) A fully supervised training set, that uses gold SQL annotations  $\{\langle x_i, Q_i, D_i \rangle\}_{i=1}^n$ . (2) A weakly supervised training set, where given answer  $a_i$  and QDMR  $s_i$ , we synthesize queries  $\hat{Q}_i$ . As SQL synthesis coverage is less than  $100\%$ , the process returns a subset of  $m < n$  examples  $\{\langle x_i, \hat{Q}_i, D_i \rangle\}_{i=1}^m$  on which the model is trained.
4.3.1 Training Data
We train models on two text-to-SQL datasets: SPIDER (Yu et al., 2018) and GEO880 (Zelle and Mooney, 1996). As our weakly supervised training sets, we use the synthesized examples  $\langle x_i,\hat{Q}_i,D_i\rangle$  described in §4.2, (using annotated QDMRs). We successfully synthesized 5,349 training examples for SPIDER and 547 examples for GEO880 train.
4.3.2 Models and Evaluation
Models We fine-tune T5-large for text-to-SQL, using the hyperparameters from §4.1. We choose T5 as it is agnostic to the structure of its input sequences. Namely, it has been shown to perform competitively on different text-to-SQL datasets, regardless of their SQL conventions (Shaw et al., 2021; Herzig et al., 2021). This property is particularly desirable in our cross-database evaluation (§4.3.3), where we test on multiple datasets.
3The metric is a strict lower bound on performance.
4https://leaderboard.allenai.org/break
In practice, we do not train directly on  $\hat{Q}_i$  but on  $s_i$  following its phrase-column linking. This representation is then automatically mapped to SQL to evaluate its execution.
Model	Supervision	Training set	Exec. %
T5-SQL-G	{x_i, Q_i, D_i}	7,000	68.0 ± 0.3
T5-SQL-Gpart	{x_i, Q_i, D_i}	5,349	66.4 ± 0.8
T5-QDMR-G	{x_i, a_i, s_i, D_i}	5,349	65.8 ± 0.3
T5-QDMR-P	{x_i, a_i, D_i}^*	5,075	62.9 ± 0.8
Table 5: SPIDER trained model results on the dev set. *Supervision for T5-QDMR-P also includes 700 annotated QDMRs of SPIDER train questions.
Model	ACADEMIC	GEO880	IMDB	YELP
T5-SQL-G	8.2 ± 1.3	33.6 ± 2.5	19.8 ± 3.6	22.7 ± 1.2
T5-SQL-Gpart	4.9 ± 1.5	32.4 ± 1.3	20.9 ± 0.8	20.7 ± 1.4
T5-QDMR-G	10.7 ± 0.7	40.4 ± 1.8	27.1 ± 3.6	16.2 ± 4.7
T5-QDMR-P	8.2 ± 0.4	39.7 ± 1.4	23.6 ± 5.5	16.7 ± 3.7
We train and evaluate the following models:
- T5-SQL-G trained on  $\{\langle x_i, Q_i, D_i \rangle\}_{i=1}^n$ , using gold SQL, annotated by experts
- T5-QDMR-G trained on  $\{\langle x_i, \hat{Q}_i, D_i \rangle\}_{i=1}^m$  with  $\hat{Q}_i$  that were synthesized using weak supervision
- T5-SQL-Gpart trained on  $\{\langle x_i, Q_i, D_i \rangle\}_{i=1}^m$ , using gold SQL. This models helps us measure the degree to which the smaller size of the synthesized training data and its different query structure (compared to gold SQL) affects performance
Evaluation Metric Due to our SQL being automatically synthesized, its syntax is often different from that of the gold SQL (see Appendix E.2). As a result, the ESM metric of Yu et al. (2018) does not fit our evaluation setup. Instead, we follow Suhr et al. (2020) and evaluate text-to-SQL models using the execution accuracy of output queries. We define execution accuracy as the percentage of output queries which, when executed on the database, result in the same set of tuples (rows) as  $a_{i}$ .
4.3.3 Training on Annotated QDMRs
We begin by comparing the models trained using annotated QDMRs to those trained on gold SQL. Meanwhile, the discussion of T5-QDMR-P, trained using predicted QDMRs, is left for §4.3.4. The results in Tables 5-7 list the average accuracy and standard deviation of three model instances, trained using separate random seeds.
SPIDER & XSP Evaluation Tables 5-6 list the results of the SPIDER trained models. All models were trained for 150 epochs and evaluated on the dev set of 1,034 examples. When comparing T5-QDMR-G to the model trained on gold SQL, it achieves  $96.8\%$  of its performance (65.8 to 68.0). The T5-SQL- $\mathbf{G}_{\mathrm{part}}$  model, trained on the
Table 6: SPIDER trained models zero-shot performance on cross-database (XSP) examples.
Model	Supervision	Train. set	Exec. %
T5-SQL-G	(xi, Qi, Di)	547	82.1 ± 1.9
T5-SQL-Gpart	(xi, Qi, Di)	454	79.4 ± 0.4
T5-QDMR-G	(xi, ai, si, Di)	454	74.5 ± 0.2
T5-QDMR-P	(xi, ai, Di)	432	70.4 ± 0.2
Table 7: GEO880 trained models results on the test set. Supervision for T5-QDMR-P does not include any indomain annotated QDMRs.
same 5,349 examples as T5-QDMR-G, performs roughly on par, scoring  $+0.6$  points (66.4 to 65.8).
As SPIDER is used to train cross-database models, we further evaluate our models performance on cross-database semantic parsing (XSP) (Suhr et al., 2020). In Table 6 we test on four additional text-to-SQL datasets (sizes in parenthesis): ACADEMIC (183), GEO880 (877), IMDB (113) and YELP (66). For ACADEMIC, IMDB and YELP we removed examples whose execution result in an empty set. Otherwise, the significant percentage of such examples would result in false positives of predictions which incidentally execute to an empty set. In practice, evaluation on the full datasets remains mostly unchanged and is provided in Appendix E. Similarly to Suhr et al. (2020), the results in Table 6 show that SPIDER trained models struggle to generalize to XSP examples. However, T5-QDMR-G performance is generally better on XSP examples, which further indicates that QDMR and answer supervision is effective, compared to gold SQL. Example predictions are shown in Appendix E.2.
GEO880 Table 7 lists the execution accuracy of models trained on GEO880. Models were trained for 300 epochs, fine-tuned on the dev set and then evaluated on the 280 test examples. We note that T5-QDMR-G achieves  $90.7\%$  of the performance of T5-SQL-G (74.5 to 82.1). The larger performance gap, compared to SPIDER models, may be partly due to the dataset size. As GEO880 has 547 training examples, fewer synthesized SQL to train T5-QDMR-G on (454) may have had a greater effect on its accuracy.
4.3.4 Training on Predicted QDMRs
We extend our approach by replacing the annotated QDMRs with the predictions of a trained QDMR parser (a T5-large model, see §4.1). In this setting, we now have two sets of questions: (1) questions used to train the QDMR parser; (2) questions used to synthesize NL-SQL data. We want these two sets to be as separate as possible, so that training the QDMR parser would not require new in-domain
annotations. Namely, the parser must generalize to questions in the NL-SQL domains while being trained on as few of these questions as possible.
SPIDER Unfortunately, SPIDER questions make up a large portion of the BREAK training set, used to train the QDMR parser. We therefore experiment with two alternatives to minimize the in-domain QDMR annotations of NL-SQL questions. First, is to train the parser using few-shot QDMR annotations for SPIDER. Second, is to split SPIDER to questions used as the NL-SQL data, while the rest are used to train the QDMR parser.
In Table 5, T5-QDMR-P is trained on 5,075 queries, synthesized using predicted QDMRs (and answer supervision) for SPIDER train questions. The predictions were generated by a QDMR parser trained on a subset of BREAK, excluding all SPIDER questions save 700 (10% of SPIDER train). Keeping few in-domain examples minimizes additional QDMR annotation while preserving the predictions quality. Training on the predicted QDMRs, instead of the annotated ones, resulted in accuracy being down 2.9 points (65.8 to 62.9) while the model achieves 92.5% of T5-SQL-G performance on SPIDER dev. On XSP examples, T5-QDMR-P is competitive with T5-QDMR-G (Table 6).
In Table 8, we experiment with training T5-QDMR-P without in-domain QDMR annotations. We avoid any overlap between the questions and domains used to train the QDMR parser and those used for SQL synthesis. We randomly sample 30-40 databases from SPIDER and use their corresponding questions exclusively as our NL-SQL data. For training the QDMR parser, we use BREAK while discarding the sampled questions. We experiment with 3 random samples of SPIDER train, numbering 1,348, 2,028 and 2,076 examples, with synthesized training data of 1,129, 1,440 and 1,552 examples respectively. Results in Table 8 show that, on average, T5-QDMR-P achieves  $95.5\%$  of the performance of T5-SQL-G. This indicates that even without any in-domain QDMR annotations, data induced from answer supervision and out-of-domain QDMRs is effective in training text-to-SQL models, compared to gold SQL.
GEO880 For predicted QDMRs on GEO880, we train the QDMR parser on BREAK while discarding all of its 547 questions. Therefore, the parser was trained without any in-domain QDMR annotations for GEO880. SQL synthesis using the pre
Model	Supervision	Train. set	DB #	Exec. %
T5-SQL-G	(xi, Qi, Di)	1,348	30	48.4
T5-SQL-Gpart	(xi, Qi, Di)	1,129	30	47.4
T5-QDMR-P	(xi, ai, Di)	1,129	30	46.2
T5-SQL-G	(xi, Qi, Di)	2,028	40	54.7
T5-SQL-Gpart	(xi, Qi, Di)	1,440	40	51.3
T5-QDMR-P	(xi, ai, Di)	1,440	40	52.1
T5-SQL-G	(xi, Qi, Di)	2,076	40	56.2
T5-SQL-Gpart	(xi, Qi, Di)	1,552	40	53.7
T5-QDMR-P	(xi, ai, Di)	1,552	40	53.8
Table 8: SPIDER models results on the dev set. T5-QDMR-P is trained without using any QDMR annotations for training set questions. We train separate models on the three randomly sampled training sets.
dicted QDMRs resulted in 432 queries. In Table 7, T5-QDMR-P reaches  $85.7\%$  of T5-SQL-G performance while being trained using question-answer supervision and no in-domain QDMR annotations.
Our approach uses question decompositions and answers as supervision for text-to-SQL parsing. As annotating SQL requires expertise, our solution serves as a potentially cheaper alternative. Past work has shown that non-experts can provide the answers for questions on knowledge graphs (Berant et al., 2013) and tables (Pasupat and Liang, 2015). However, manually annotating question-answer pairs on large-scale databases may present new challenges which we leave for future work.
During SQL synthesis we assume that literal values (strings or dates) appear verbatim in the database as they do in the question. We observe that, for multiple datasets, this assumption generally holds true (\$4.2). Still, for questions with domain-specific jargon (Lee et al., 2021) our approach might require an initial step of named-entity-recognition. Failure to map a QDMR to SQL may be due to a mismatch between a QDMR and its corresponding SQL structure (\$3.2). We account for such mismatches by using heuristics to modify the structure of a candidate query (Appendix B). A complementary approach could train a model, mapping QDMR to SQL, to account for cases where our heuristic rules fail. Nevertheless, our SQL synthesis covers a diverse set of databases and query patterns, as shown in our experiments.
For a thorough review of NL interfaces to databases see Affolter et al. (2019); Kim et al. (2020). Research on parsing text-to-SQL gained significant
traction in recent years with the introduction of large supervised datasets for training models and evaluating their performance (Zhong et al., 2017; Yu et al., 2018). Recent approaches relied on specialized architectures combined with pre-trained language models (Guo et al., 2019; Wang et al., 2020; Lin et al., 2020; Yu et al., 2021; Deng et al., 2021; Scholak et al., 2021). As our solution synthesizes NL-SQL pairs (using weak supervision) it can be used to train supervised text-to-SQL models.
Also related is the use of intermediate meaning representations (MRs) in mapping text-to-SQL. In past work MRs were either annotated by experts (Yaghmazadeh et al., 2017; Kapanipathi et al., 2020), or were directly induced from such annotations as a way to simplify the target MR (Dong and Lapata, 2018; Guo et al., 2019; Herzig et al., 2021). Instead, QDMR representations are expressed as NL utterances and can therefore be annotated by non-experts. Similarly to us, Saparina and Osokin (2021) map QDMR to SPARQL. However, our SQL synthesis does not rely on the annotated linking of question phrases to DB elements (Lei et al., 2020). In addition, we train models without gold QDMR annotations and test our models on four datasets in addition to SPIDER.
This work presents a weakly supervised approach for generating NL-SQL training data, using answer and QDMR supervision. We implemented an automatic SQL synthesis procedure, capable of generating effective training data for dozens of target databases. Experiments on multiple text-to-SQL benchmarks demonstrate the efficacy of our synthesized training data. Namely, our weakly-supervised models achieve  $91\% - 97\%$  of the performance of fully supervised models trained on annotated SQL. Further constraining our models supervision to few or zero in-domain QDMRs still reaches  $86\% - 93\%$  of the fully supervised models performance. Overall, we provide an effective solution to train text-to-SQL parsers while requiring zero SQL annotations.
We would like to thank Mor Geva, Ori Yoran and Jonathan Herzig for their insightful comments. This research was partially supported by The Israel Science Foundation (grant 978/17), and the Yandex Initiative for Machine Learning and the European Research Council (ERC) under the Euro
pean Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800). This work was completed in partial fulfillment of the PhD of Tomer Wolfson.
Katrin Affolter, Kurt Stockinger, and A. Bernstein. 2019. A comparative survey of recent natural language interfaces for databases. The VLDB Journal, 28:793 - 819.
J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP).
James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 18-27, Uppsala, Sweden. Association for Computational Linguistics.
Edgar F Codd. 1970. A relational model of data for large shared data banks. Communications of the ACM, 13(6):377-387.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure-grounded pretraining for text-to-sql. In NAACL.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742, Melbourne, Australia. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Mor Geva, Tomer Wolfson, and Jonathan Berant. 2021. Break, perturb, build: Automatic perturbation of reasoning paths through question decomposition. *ArXiv*, abs/2107.13935.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Association for Computational Linguistics (ACL).
Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Unlocking compositional generalization in pre-trained models using intermediate representations. *ArXiv*, abs/2104.07478.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 541-550, Portland, Oregon, USA. Association for Computational Linguistics.
Pavan Kapanipathi, I. Abdelaziz, Srinivas Ravishankar, S. Roukos, Alexander G. Gray, Ramón Fernández Astudillo, Maria Chang, Cristina Cornelio, S. Dana, Achille Fokoue, Dinesh Garg, A. Gliozzo, Sairam Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Youngsuk Lee, Yunyao Li, Francois Luus, Ndivhuwo Makondo, Nandana Mihindukula-sooriya, Tahira Naseem, Sumit Neelam, L. Popa, Revanth Reddy, R. Riegel, G. Rossiello, Udit Sharma, G P Shrivatsa Bhargav, and M. Yu. 2020. Question answering over knowledge bases by leveraging semantic parsing and neuro-symbolic reasoning. ArXiv, abs/2012.01707.
Hyeonji Kim, Byeong-Hoon So, Wook-Shin Han, and Hongrae Lee. 2020. Natural language to sql: Where are we today? Proc. VLDB Endow., 13:1737-1750.
D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic evaluation of text-to-SQL parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2261-2273, Online. Association for Computational Linguistics.
Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. 2020. Re-examining the role of schema linking in text-tosql. In EMNLP.
Fei Li and Hosagrahar Visvesvaraya Jagadish. 2014. Nalir: an interactive natural language interface for querying relational databases. In International Conference on Management of Data, SIGMOD.
P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL), pages 590-599.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4870-4888, Online. Association for Computational Linguistics.
P. Pasupat and P. Liang. 2015. Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics (ACL).
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
A. Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason Alan Fries, Sen Wu, and C. Ré. 2017. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases.
Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive bottom-up semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324, Online. Association for Computational Linguistics.
Irina Saparina and Anton Osokin. 2021. Sparqling database queries from intermediate question decompositions. In EMNLP.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah-danau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In ACL/IJCNLP.
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020. Obtaining faithful interpretations from compositional neural networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5594-5608, Online. Association for Computational Linguistics.
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In ACL.
James Thorne, Majid Yazdani, Marzieh Saeidi, F. Silvestri, S. Riedel, and A. Halevy. 2021. From natural language processing to neural databases. Proc. VLDB Endow., 14:1033-1039.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and M. Richardson. 2020. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. ArXiv, abs/1911.04942.
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In EMNLP.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1:1-26.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir R. Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmented pre-training for table semantic parsing. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Empirical Methods in Natural Language Processing (EMNLP).
M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1050-1055.
L. S. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI), pages 658-666.
Zhen-Yu Zhang, Peng Zhao, Yuan Jiang, and Zhi-Hua Zhou. 2019. Learning from incomplete and inaccurate supervision. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
A QDMR to SQL Mapping Rules
B SQL Candidate Search Heuristics