SlowGuess/ACL_2020_OCR huggingface dataset:
Our model takes as input an utterance  $\overline{x}$  and a database schema  $S$ . Similar to Guo et al. (2019), we serialize  $S$  into a sequence of wordpieces  $\overline{s} = \bar{t}_0 + \bar{t}_1 + \dots + \bar{t}_{|\mathcal{S}|}$ . Each  $\bar{t}_i$  is a serialization of table  $\mathcal{T}_i$ , where  $\bar{t}_i = \langle \mathrm{TAB} \rangle + \overline{T}_i + \bar{c}_{i,0} + \bar{c}_{i,1} + \dots + \bar{c}_{i,|\mathcal{T}_i|}$ . TAB is a token noting the beginning of a table schema serialization.  $\overline{T}_i$  is the tokenization of table  $\mathcal{T}_i$ 's name. Each  $\bar{c}_{i,j}$  is a serialization of a column  $\mathcal{C}_{i,j}$ , where  $\bar{c}_{i,j} = \mathrm{CT}_{i,j} + \overline{C}_{i,j}$ .  $\mathrm{CT}_{i,j}$  is a token denoting the type of the column's contents as provided by the database schema, for example numerical or text.  $\overline{C}_{i,j}$  is the tokenization of the column's
name. The ordering of table schemas in  $\overline{s}$  and table columns in each  $\bar{t}_i$  is arbitrary. $^7$  The input to the encoder is the concatenation of the query wordpieces and the serialized schema, represented as the sequence of tokens  $\overline{x} = \langle \mathrm{CLS} \rangle + \overline{u} + \langle \mathrm{SEP} \rangle + \overline{s}$ . The inputs to the encoder are embedded and passed to a pretrained Transformer encoder such as BERT (Devlin et al., 2019). The decoder is an autoregressive Transformer decoder (Vaswani et al., 2017) that attends over the outputs of the encoder and the generated prefix.
We use a training set  $\{\overline{x}^{(l)},\overline{y}^{(l)},S^{(l)}\}_{l = 1}^{N}$  consisting of pairs of natural language utterances, gold SQL queries, and database schemas. We train the encoder and decoder end-to-end, minimizing the token-level cross-entropy loss of the gold query  $\overline{y}^{(l)}$ . We update the parameters of the pre-trained encoder during training. For training data we use training sets developed for XSP. Importantly, to ensure we are evaluating the cross-database setting, our training data does not include examples from the evaluation databases. During inference, we use beam search and execute the highest-probability, syntactically correct prediction. We impose a maximum execution time of 45 seconds for predictions. More details on the model, learning, and evaluation setup are available in Appendix B.
5.1 Generalization Strategies
While using pre-trained language models can help encode natural language text, we need other strategies to reason jointly about the language and the database schema in completely unseen domains. We focus on generalizing to domain-specific language and novel database structures.
Value Copying Similar to previous work (Jia and Liang, 2016; Gu et al., 2016; Gulcehre et al., 2016; See et al., 2017), we use a copy mechanism in the decoder. At each output step, the decoder generates a distribution over possible actions, including selecting a symbol from the output vocabulary, and copying a token from the input  $\overline{x}$ . We only allow copying of certain token types, and mask out invalid copying actions, including independent wordpieces from  $\overline{u}$  and TAB and column-type tokens. For table and column tokens, the name of
To discourage over-fitting to an arbitrary ordering of schema elements, we duplicate each Spider training example seven times with randomly permuted orderings. Duplicating seven times results in the number of Spider training examples roughly matching the number of WikiSQL training examples (Section 5.1).
the corresponding table or column is recovered by post-processing the predicted sequence  $\overline{y}$ .
Previous approaches on Spider do not evaluate execution accuracy over the databases. Because the main metric does not require values in the predicted and gold queries to be the same, many approaches simplify the problem by using a placeholder token for all values during training. However, correctly generating values is critical for correctly executing predicted queries. To the best of our knowledge, our approach is the first to evaluate on Spider with execution accuracy and to generate SQL queries without placeholder values.
Multiple Data Sources We train with training data from Spider (Yu et al., 2018) and WikiSQL (Zhong et al., 2017). Spider includes examples of complex SQL queries grounded in multtable databases, while queries in WikiSQL are compositionally simple and grounded in single web tables. We use WikiSQL to improve generalization to domain-specific data, as it covers a large variety of domains. WikiSQL contains many more tables than Spider, and prior work estimates that roughly half of WikiSQL examples require using domain knowledge to map from entity mentions to column names (Yavuz et al., 2018).
Different Output Space Guo et al. (2019) demonstrated improvements on Spider by deterministically mapping SQL to an intermediate representation, SemQL, and learning to predict outputs in this space. SemQL does not require predicting all of the tables in the FROM clause of the SQL query, or explicitly predicting the columns on which tables are joined. Instead of reasoning about foreign keys, the model predicts queries in the SemQL space, which are deterministically transformed to a final SQL query. In most cases, SemQL queries can be mapped back to SQL using database foreign key relations. We implement this aspect of SemQL as a mapping from SQL to a representation with an under-specified FROM clause, which we call  $\text{SQL}^{\text{UF}}$ . Conversion from SQL to  $\text{SQL}^{\text{UF}}$  removes tables from the FROM clause(s) of the SQL query implicitly referenced via a column elsewhere in the query, and removes JOIN clauses. Conversion from  $\text{SQL}^{\text{UF}}$  to SQL restores these tables, and joins between tables are inferred by greedily identifying a path that connects all tables in the FROM clause,
given foreign key relations. $^{8}$  Examples of  $\mathbf{SQL}^{\mathsf{UF}}$  are shown in Appendix C.
Comparison to Existing XSP Systems Our best model performs well on the Spider development set. Table 3 compares our system with top systems on the Spider leaderboard<sup>9</sup>. On the development set, our model performs competitively with contemporaneous systems. Table 3 shows that Spider performance correlated to the choice of the pre-trained models. Public BERT<sub>LARGE</sub> is better than BERT<sub>BASE</sub>. To further improve performance, we experiment with an enhanced pre-trained model BERT<sub>LARGE+</sub> following the recipe proposed by Liu et al. (2019). The BERT<sub>LARGE+</sub> model is trained with 8K batch size and 100k training steps, and in contrast to RoBERTa, is only trained on the Wiki+Books Corpus used in Devlin et al. (2019). Training our model to predict value placeholders (- Value Copying) instead of copying values from the input results in a performance drop, showing a benefit of modeling values even when ignored by the metric.<sup>10</sup>
XSP on Unseen Datasets Table 2 shows results on all evaluation data, including datasets originally studied in the SSP setting. We report results on the filtered set (Section 3) as well as the full set of these datasets. A large portion of the examples in datasets such as Restaurants and Advising yield empty execution results. This shows the need to also evaluate on the filtered set, where incorrect spurious predictions are much less likely to result in the same table as a gold query with an empty table result. Second, while execution accuracy on Spider is relatively high, performance on the other evaluation datasets is much lower.
We find that all three techniques for addressing generalization challenges are effective. First, including WikiSQL in the training data results in better performance than only using Spider training data. We hypothesize that this is due to the addi
Like SemQL, this conversion is not possible if foreign key relations between predicted tables are not provided or if a given table is referenced more than once in a FROM clause. This can also result in a lossy or ambiguous conversion if there are multiple foreign key relations between a pair of tables.
$^{9}$ https://yale-lily.github.io/spider. In Table 3, we include non-anonymized leaderboard submissions, and for anonymous systems, the most recent submission for duplicate systems.
10 About  $55\%$  of examples in Spider do not require copying values from the input utterance to the gold query.
8378
Dataset	Metric	# Examples	Our best	-WikiSQL	-SQLUF	- Value copying	Empty Prior
ATIS	289 (486)	0.8 (11.9)	0.5 (11.9)	0.8 (11.9)	0.1 (10.8)	0.0 (11.9)
GeoQuery	532 (598)	41.6 (40.0)	35.6 (35.0)	34.7 (33.4)	2.2 (5.6)	0.0 (4.0)
Restaurants	27 (378)	3.7 (45.2)	3.7 (46.3)	0.0 (46.6)	0.0 (51.1)	0.0 (51.6)
Academic	180 (196)	8.2 (12.1)	6.1 (9.4)	5.7 (9.0)	2.8 (7.7)	0.0 (4.1)
IMDB	Execution	107 (131)	24.6 (33.3)	24.3 (32.3)	23.1 (32.3)	0.0 (14.3)	0.0 (13.0)
Yelp	54 (128)	19.8 (49.2)	16.7 (47.9)	14.8 (47.9)	4.9 (53.1)	0.0 (41.4)
Scholar	394 (599)	0.5 (6.8)	0.4 (7.4)	0.5 (8.6)	0.2 (7.8)	0.0 (9.3)
Advising	309 (2858)	2.3 (35.2)	1.2 (35.7)	1.4 (37.3)	0.0 (38.0)	0.0 (38.3)
Spider	Execution	1034	69.0	68.4	65.1	33.9	4.7
Exact Set Match	65.0	65.1	60.5	54.1	-
Table 2: Execution accuracy on the XSP task for the eight evaluation datasets and Spider, comparing our best system with baselines and independent ablations. For Spider, we also report performance using Exact Set Match, the official Spider metric. Results are averaged over three trials. The full set results are reported in parentheses. The empty prior represents the baseline accuracy of returning empty set for all queries. The accuracies on the re-purposed datasets are much lower than the Spider performance.
System	Exact Set Match (Dev.)
Top Leaderboard Systems (As of May 1, 2020)
RYANSQL v2 + BERT (Choi et al., 2020)	70.6
RYANSQL + BERT (Choi et al., 2020)	66.6
RATSQL v2 + BERT (Anonymous)	65.8
IRNet++ + XLNET (Anonymous)	65.5
IRNet + BERT (Guo et al., 2019)	61.9
RASQL + BERT (Anonymous)	60.8
GIRN + BERT (Anonymous)	60.2
CNSQL (Anonymous)	58.0
EditSQL + LSL + BERT (Anonymous)	57.9
GNN + Bertrand-DR (Kelkar et al., 2020)	57.9
Ours with BERTLARGE+	65.8
Ours with BERTLARGE	63.2
Ours with BERTBASE	60.4
Ours with BERTLARGE+ - Value Copying	55.1
Table 3: Performance on the Spider development set using Spider's official evaluation metric (Exact Set Match), ordered by the development set performance. For our systems, we report the Exact Set Match of the best of three trials. While the focus of our paper is not on Spider performance, our system still performs well.
tional domains in WikiSQL, as well as the larger proportion of examples that require mapping from entities in the utterance to column names (Yavuz et al., 2018). Using  $\mathrm{SQL}^{\mathrm{UF}}$  also improves performance, as it produces queries coherent with respect to the schema, for example only selecting columns from tables where the column exists. Finally, using value placeholders significantly reduces execution accuracy in all datasets. While masking values decreases Exact Set Match on Spider by  $10.9\%$ , its effect on execution accuracy can be devastating both for Spider and the eight evaluation datasets. This demonstrates the need to consider execution
results when evaluating semantic parsing systems.
Error Analysis For each evaluation dataset, we analyze twenty random predictions from the filtered subset. Examples of the most common error types are shown in Figure 2, along with the proportion of analyzed predictions in the eight target datasets that contain the error type. Appendix D discusses the complete results of error analysis.
$40\%$  of errors are caused by comparing an entity to the wrong column, for example searching for 'James Bond' in the director.name column when it actually refers to a movie.title. This usually requires using domain knowledge identify to columns that are likely to contain the mentioned entity (Section 4.1).  $31.1\%$  of errors are caused by missing constraints specified in the utterance, for example by failing to use a relevant entity in the predicted query.  $28.8\%$  of errors are also caused by incorrectly identifying entity spans, for example by treating FIN 340 as a single entity rather than two separate entities in the database (Section 4.1). Another common error is predicting the wrong final column. While choosing what to return is difficult for the model due to understanding domain-specific phrases such as how many (20.0% of errors; Section 4.1), sometimes the errors are due to dataset conventions (26.9% of errors; Section 4.3). For example, the paperid column should be selected instead of the title column in Scholar. Such dataset conventions could be learned through few-shot learning, where a small number of in-domain training examples are available.
Our system is required to generalize to larger databases than it was trained on, including more complex compositions of tables (Section 4.2). For
8379
Figure 2: The most common error types made by our best system, including an example. The subscript indicates the results are in the XSP setting. Each prediction may be annotated with more than one error type.
example, while  $\mathrm{SQL}^{\mathrm{UF}}$  can be used to represent most gold queries in most evaluation datasets (shown in Appendix C), in ATIS, only  $17.3\%$  of gold queries are covered by  $\mathrm{SQL}^{\mathrm{UF}}$ . Most of the uncovered examples require mapping two columns, to_airport and from_airport, in the same table flight to the same foreign key airport服务能力.airport_code. This compositional structure is not covered by  $\mathrm{SQL}^{\mathrm{UF}}$ , but is critical to perform well on ATIS.
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen at training time. While this task has been studied through datasets developed specifically for XSP, we propose a more holistic evaluation for XSP, where we also evaluate on datasets originally studied in a setting where in-domain training data is available. We identify several new generalization challenges that arise when evaluating in our proposed setup, including identifying entities, mapping entities and domain-specific phrases to a database schema, and generalizing to more complex database schemas. Using a model that performs well on evaluation data designed for XSP, we are able to move towards addressing some of the generalization challenges on these additional evaluation sets without any indomain training data. Our results and analysis demonstrate the need for developing more holistic evaluation of cross-database semantic parsing using a more diverse set of language and databases.
Several significant generalization challenges remaining, including improving commonsense and in-domain reasoning and table schema understanding capabilities. Some examples in our filtered evaluation set still require reasoning about dataset conventions that are difficult to acquire without in-domain training examples. Future work could also make the stronger assumption that a small number of in-domain training examples are available, and train and evaluate in a few-shot setting.
The first author is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1650441. We thank the Google Research Language and Cornell NLP groups for their comments and feedback during the project's development. We also thank Jing Li for pre-training the  $\mathrm{BERT}_{\mathrm{LARGE+}}$  model, and Philip Massey, Zuyao Li, Angelica Chen, Karl Pichotta, and Francesco Piccinno for their contributions to the codebase. Finally, we thank the anonymous reviewers for their comments and suggestions.
8380
Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 421-432, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Ben Bogin, Jonathan Berant, and Matt Gardner. 2019a. Representing schema structure with graph neural networks for text-to-SQL parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560-4565, Florence, Italy. Association for Computational Linguistics.
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b. Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3650-3655, Hong Kong, China. Association for Computational Linguistics.
David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. pages 859-865.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. RYANSQL: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases.
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the atist task: The atis-3 corpus. In HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33-43, Berlin, Germany. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Ofer Givoli and Roi Reichart. 2019. Zero-shot semantic parsing for instructions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4454-4464, Florence, Italy. Association for Computational Linguistics.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 140-149, Berlin, Germany. Association for Computational Linguistics.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Jonathan Herzig and Jonathan Berant. 2018. Decoupling structure and lexicon for zero-shot semantic parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1619-1629, Brussels, Belgium. Association for Computational Linguistics.
Jonathan Herzig and Jonathan Berant. 2019. Don't paraphrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3801-3811, Hong Kong, China. Association for Computational Linguistics.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643-1652, Brussels, Belgium. Association for Computational Linguistics.
Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821-1831, Vancouver, Canada. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, and Peter Relan. 2020. Bertrand DR: Improving text-to-SQL using a discriminative re-ranker.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516-1526, Copenhagen, Denmark. Association for Computational Linguistics.
Igor Labutov, Bishan Yang, and Tom Mitchell. 2018. Learning to learn semantic parsers from natural language supervision. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1676-1690, Brussels, Belgium. Association for Computational Linguistics.
Fei Li and H. V. Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.