SlowGuess/ACL_2020_OCR huggingface dataset:
Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based neural text-to-SQL generation. arXiv preprint arXiv:1905.13326.
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kocisky, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 599-609, Berlin, Germany. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456-1465, Berlin, Germany. Association for Computational Linguistics.
Scott Miller, David Stallard, Robert Bobrow, and Richard Schwartz. 1996. A fully statistical approach to natural language interfaces. In 34th Annual Meeting of the Association for Computational Linguistics, pages 55-61, Santa Cruz, California, USA. Association for Computational Linguistics.
Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for Chinese SQL semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3643-3649, Hong Kong, China. Association for Computational Linguistics.
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 574-584. IEEE.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the
8382
7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470-1480, Beijing, China. Association for Computational Linguistics.
Hoifung Poon. 2013. Grounded unsupervised semantic parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 933-943, Sofia, Bulgaria. Association for Computational Linguistics.
Ana-Maria Popescu, Alex Armanasu, Oren Etzioni, David Ko, and Alexander Yates. 2004. Modern natural language interfaces to databases: Composing statistical parsing with semantic tractability. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 141-147, Geneva, Switzerland. COLING.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. 2019. Generating logical forms from graph representations of text and entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 95-106, Florence, Italy. Association for Computational Linguistics.
Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2238-2249, New Orleans, Louisiana. Association for Computational Linguistics.
Lappoon R. Tang and Raymond J. Mooney. 2000. Automated construction of database interfaces: Integrating statistical and relational learning for semantic parsing. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 133-141, Hong Kong, China. Association for Computational Linguistics.
Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In International Joint Conference on Artificial Intelligence (IJCAI).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Yuk Wah Wong and Raymond Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 439-446, New York City, USA. Association for Computational Linguistics.
Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1341-1350, Berlin, Germany. Association for Computational Linguistics.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: Query synthesis from natural language. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ACM, pages 63:1-63:26.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and a text-to-SQL case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5450-5461, Hong Kong, China. Association for Computational Linguistics.
Semin Yavuz, Izzeddin Gur, Yu Su, and Xifeng Yan. 2018. What it takes to achieve  $100\%$  condition accuracy on WikiSQL. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1702-1711, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), pages 476-486. IEEE.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada. Association for Computational Linguistics.
8383
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019a. CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1962-1979, Hong Kong, China. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A largescale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. SParC: Cross-domain semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4511-4523, Florence, Italy. Association for Computational Linguistics.
John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, pages 1050-1055.
Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 678-687, Prague, Czech Republic. Association for Computational Linguistics.
Luke Zettlemoyer and Michael Collins. 2009. Learning context-dependent mappings from sentences to logical form. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 976-984, Suntec, Singapore. Association for Computational Linguistics.
Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars.
In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, UAI'05, pages 658-666, Arlington, Virginia, United States. AUAI Press.
Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based SQL query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5341-5352, Hong Kong, China. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
8384
A Data Details
Measuring Exact Column Match in Utterances For each example, we identify columns used for direct comparison with values in the correct SQL query (not considering columns used to link two tables, order or group results, or in the top-level SELECT statement). We then heuristically identify whether any of the used column names appear in the utterance by canonicalizing the column name (e.g., replacing underscores with spaces) and performing a basic substring match. Because slight variants of column names may appear in the utterance, our reported results show an lower bound.
Heuristically Filtering Datasets We use several heuristics to filter evaluation data. Although we cannot automatically filter out all examples where database conventions are required to select the correct final column, we found that a good heuristic is filtering out examples that require selecting more than one final column. For example, in Advising, when an utterance asks for a list of classes, the labeled query always selects four columns from the course table: department, name, number, and semester. We remove all examples where a numerical or text value is not copiable from the input utterance, except for the numbers 0 and 1, which are often not copied from the input (for example, limiting table results in LIMIT 1). We also remove all examples that result in an empty table, and examples where the gold query returns a count, and the resulting table is [0].
To choose model and learning hyperparameters, we began with the hyperparameters of Shaw et al. (2019), and performed a small number of experiments to improve performance on Spider.
Model For our encoder, we use a pre-trained BERT model (Devlin et al., 2019). All input tokens use the same segment ID. We use absolute positional embeddings. The word embedding size for all tokens is 128. We use wordpiece tokenization for the input utterance. To tokenize column and table names, we replace underscores with spaces, and then apply wordpiece tokenization.
The outputs of the encoder are transformed using a linear layer before being used by the decoder. We use a two-layer Transformer decoder (Vaswani et al., 2017) with eight attention heads. We use
a gated copying mechanism, supervising gating decisions during training.
Learning During training, we use a maximum input size of 512 tokens. During training and inference, we use a maximum decoding length of 100. For training, we create seven examples per original Spider training example, with randomized permutations of table ordering, and column ordering within the table spans. We use a batch size of 32 and train for 30,000 update steps. We apply teacher-forcing during training. During training, we apply a dropout at a rate of 0.3 on the embeddings of decoder tokens and within the decoder Transformer. We use Adam optimizer (Kingma and Ba, 2014). We increase learning rate linearly from 0 to 0.00008 until 5,625 steps, then decrease it linearly to 0.0 by the end of training. The encoder begins as a pre-trained BERT model, whose parameters we freeze for the first 2,100 updates.
Inference For each evaluation example, we perform beam search against our trained model with a beam size of 100. Among the 10 most probable predictions from the beam search, we choose the highest-probability prediction that is a syntactically valid SQL query. To check syntactic validity, we test each prediction's execution against an emptied copy of the database (emptied to ensure we are not using database contents) and pass over queries which are infeasible. We test the top 10 items in the beam to limit evaluation time, and find that in nearly all cases, if a syntactically correct prediction exists in the beam, it appears as one of the top 10 items. To correctly resolve predicted  $\text{SQL}^{\text{UF}}$  queries, we use gold-standard foreign keys for all target databases.
Some queries are highly inefficient, and can take minutes to execute due to the sizes of the databases. To reduce the execution time of gold queries, we add database indices where possible. Even with these indices, some predicted queries still take a long time to execute. To make evaluation tractable, we use a timeout of 45 seconds per predicted query. If the query has not executed after 45 seconds, we return the empty table. Figure 3 shows the influence of the timeout threshold on the model performance for our best model on each evaluation dataset. By 45 seconds, the vast majority of predictions can execute, and execution accuracy has stabilized.
8385
(a) Exec. Acc. (Full)
(b) Prop. Exec. (Full)
(c) Exec. Acc. (Filtered)
Figure 3: Influence of the timeout threshold on model performance for each dataset evaluation dataset. The  $x$ -axis shows the maximum execution time (in seconds) we allow before terminating query execution. In our experiments, we use a timeout of 45 seconds. We show how the execution accuracy is influenced by the cutoff time for the full (a) and filtered (c) evaluation sets. We also show the proportion of queries which finished execution for the full (b) and filtered (d) evaluation sets. Results are generated using the model which achieved the highest execution accuracy on the Spider development set. By 45 seconds, the majority of queries return an execution result, and execution accuracy has stabilized.
(d) Prop. Exec. (Filtered)
Dataset	% of Gold Queries Covered by SQLUF
ATIS	17.3
GeoQuery	97.5
Restaurants	100.0
Academic	85.7
IMDB	94.7
Yelp	81.3
Scholar	92.2
Advising	87.4
Spider	97.4
Table 4: Estimates of coverage of  ${\mathrm{{SQL}}}^{\mathrm{{UF}}}$  across each evaluation dataset.
C SQL $^{\text{UF}}$  Examples
Examples of our intermediate representation  $\mathbf{SQL}^{\mathrm{UF}}$  are shown in Figure 4. Table 4 shows an estimate of the proportion of all gold queries that can be generated by our system.
D Supplementary Error Analysis
Table 5 shows the rate of occurrence of eight types of errors in each evaluation dataset, out of twenty random incorrect predictions. Below, we give more detailed descriptions of these error types, as well
as examples from our model predictions.
Entity Understanding We consider two types of errors: entity identification errors, and column matching errors. Entity identification errors include copying the wrong span of tokens that comprise an entity into the output. Column matching errors include comparing an entity to the wrong database column type. Examples for these errors are included in Figure 2.
Final Column We consider two types of final column errors: incorrect predictions, and ambiguous predictions. In incorrect predictions, the final column type is obviously incorrect with respect to the utterance. In ambiguous predictions, the final column is a reasonable prediction with respect to the utterance, but due to the dataset conventions, is incorrect. Examples for these errors are included in Figure 2.
Database Understanding We consider two types of database understanding errors: syntactically incorrect predictions, and predictions which do not compose the tables correctly. Although the second category execute successfully, their result
8386
SQL:	SELECT people.name FROM people JOIN films ON people.id = film.person_id WHERE films.id = 5
\( \mathbf{SQL}^{\mathbf{UF}} \):	SELECT people.name UF WHERE films.id = 5
SQL:	SELECT people.name FROM people JOIN films ON people.id = film.person_id
\( \mathbf{SQL}^{\mathbf{UF}} \):	SELECT people.name UF films
SQL:	SELECT cities.state, count(*) FROM cities GROUP BY cities.state
\( \mathbf{SQL}^{\mathbf{UF}} \):	SELECT cities.state, count(*) UF GROUP BY cities.state
SQL:	SELECT count(*) FROM cities
\( \mathbf{SQL}^{\mathbf{UF}} \):	SELECT count(*) UF cities
SQL:	SELECT student_id FROM student-course registrations UNION SELECT student_id FROM student-course Attendance
\( \mathbf{SQL}^{\mathbf{UF}} \):	SELECT student-course registrations student_id UF UNION SELECT student-course Attendance/student_id UF
SQL:	SELECT table_1.id, table_3.id FROM table_1 JOIN table_2 ON table_1.table_2_id = table_2.id JOIN table_3 ON table_2.table_3_id
\( \mathbf{SQL}^{\mathbf{UF}} \):	SELECT table_1.id, table_3.id UF table_2
Figure 4: Examples of  $\mathbf{SQL}^{\mathrm{UF}}$ , which uses under-specified FROM clauses. Tables are omitted from the FROM clause unless a column belonging to the given table is not mentioned elsewhere in the query. Under certain assumptions, reconstruction of the original SQL is possible given schema information.
ing tables are incorrect due to how the database is structured. For example, in Restaurants, although the restaurant table has a city_name column, the correct way to construct a query corresponding to an utterance like how many places for french food are there in palo alto? is to traverse the location table instead.
Query Implementation We consider two types of errors related to incorrectly implementing the utterance's intent in SQL: missing and incorrect constraints. Missing constraints involve ignoring a constraint mentioned in the utterance, such as the paper title constraint in the Academic example in Figure 2. Incorrect constraints are incorrect for reasons besides those described above, such as entity-column matching. For example (from GeoQuery):
NL:	what is the smallest state in the usa
Pred.:	select state.state_name from state where state.cOUNTRY_name = 'usa' order by state/population limit 1;
Gold:	select T1.state_name from state as T1 where T1.area = (select min(T2.area) from state as T2);
Our model's prediction incorrectly orders by population rather than the state's area.
E Additional Results on Spider
Table 6 shows the performance on the Spider (Yu et al., 2018) development set for our single best model split by hardness level. Table 7 shows the F1
over query components of our best model on the Spider development set.
8387
Error Type	Spider	Restaurants	IMDB	ATIS	Academic	Scholar	Yelp	GeoQuery	Advising
Entity understanding
Identification	4	9	3	2	5	9	5	5	8
Column match	6	9	6	20	4	5	8	4	8
Final column
Incorrect	5	0	3	3	8	4	1	5	8
Ambiguous	0	0	0	16	3	9	11	0	4
Database understanding
Syntax error	0	2	15	0	0	1	1	0	1
Table composition	6	6	0	1	2	0	1	2	1
Query implementation
Missing constraint	2	6	2	9	10	4	4	3	12
Incorrect constraint	2	0	0	0	0	0	0	6	0
Table 5: For each evaluation dataset, we analyzed twenty random incorrect predictions of our best model. We categorized each into one or more error categories, including errors of understanding entities mentioned in the utterance, generating the correct top-level column selection, understanding the database structure, and correctly implementing the constraints of the utterance in SQL. We report the number of predictions, out of the twenty analyzed per dataset, which had each error type.
System	Easy	Medium	Hard	Extra Hard
Ours with BERTLARGE+	83.2	71.1	57.5	34.7
Table 6: Spider's official evaluation metric results for our best model split by hardness level on the Spider development set.
Component	F1
SELECT	89.2
SELECT (no AGG)	90.4
WHERE	71.7
WHERE (no OP)	76.3
GROUP (no HAVING)	82.0
GROUP	79.4
ORDER	82.9
AND/OR	98.1
IEUN	46.5
KEYWORDS	89.5
Table 7: Per-component F1 of our best model on the Spider development set.
8388