SlowGuess/NAACL_2021_OCR huggingface dataset:
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372-8388, Online. Association for Computational Linguistics.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Daniel C. Wang, Andrew W. Appel, Jeff L. Korn, and Christopher S. Serra. 1997. The zephyr abstract syntax description language. In Proceedings of the Conference on Domain-Specific Languages on Conference on Domain-Specific Languages (DSL), 1997, DSL'97, page 17, USA. USENIX Association.
Pengcheng Yin and Graham Neubig. 2018. TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation. arXiv:1810.02720 [cs]. ArXiv:1810.02720.
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018a. TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation. arXiv:1804.09769 [cs]. ArXiv: 1804.09769.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. arXiv:1809.08887 [cs]. ArXiv:1809.08887.
John M. Zelle and Raymond J. Mooney. 1996. Learning to Parse Database Queries Using Inductive Logic Programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, AAAI'96, pages 1050-1055. AAAI Press. Event place: Portland, Oregon.
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. arXiv:2010.02840 [cs]. ArXiv: 2010.02840.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. arXiv:1709.00103 [cs]. ArXiv:1709.00103.
A Table of Relations
Table 8 lists all relations in the DuoRAT encoder while Table 9 lists all relations used in the decoder.
B Grammar-Constrained Sequence Prediction
The SQL grammar is defined in the Zephyr Abstract Syntax Description Language (ASDL) (Wang et al., 1997). Given the grammar, the ground-truth SQL query is parsed to an abstract syntax tree (AST). This tree is then serialized into a sequence of production actions, choosing the depth-first, left-to-right order as reference. The decoder is trained to either generate actions from a vocabulary of actions or to copy literals from the question or the schema. During inference, the predicted action sequence is deserialization back to an AST, from which the final SQL query string is assembled. Actions that are invalid at any decoding step are masked such that the decoder is constrained to only produce grammatically valid SQL.
Each input element to the DuoRAT decoder consists of the concatenation of an action embedding, a field embedding, and a field type embedding which are derived from the ASDL grammar.
C Training Procedure
The BERT stage of the encoder is initialized with BERT-large weights, whereas the RAT stage of the encoder is initialized randomly. BERT's input sequence length is limited to 512 tokens. Inputs that exceed this limit are truncated. We use 8 RAT layers with each  $H^{(\mathrm{enc})} = 8$  heads, an embedding dimension of  $d_x^{(\mathrm{enc})} = d_z^{(\mathrm{enc})} = 1024$ , a feed-forward network with 1024 dimensions, and a dropout probability of 0.1.
In the decoder, we use 2 randomly initialized RAT layers with  $H^{(\mathrm{dec})} = 8$  heads each, embedding dimensions for actions, fields, and field types of 64 each for a total of  $d_x^{(\mathrm{dec})} = d_z^{(\mathrm{dec})} = 192$ , a feed-forward network with 256 dimensions, and a dropout probability of 0.1. The memory pointer has a projection size of 50, and the minimum required number of times a literal action token must occur in the input of the training set to be added to the vocabulary is 5.
We train using Adam (Kingma and Ba, 2015) with default parameters for 100,000 steps in PyTorch's automatic mixed precision mode (AMP). We use a step-wise linear learning-rate schedule with a warm-up period from 0 up to 0.0001 in 2,000 steps followed by a 98,000 steps long cool-down period from 0.0001 down to 0. All model weights are trainable. The learning rate for BERT is always 8 times lower than for the rest of the model. Each training batch has 9 items, we use gradient accumulation to bring the batch size up to effectively 27. We report results obtained with the beam size of 1. Training and evaluating DuoRAT on a single V100 with 32 GB HBM2 takes 2 days. For each training run, we log the performance of the model checkpoint with the best validation accuracy during training measured in intervals of 5,000 steps. In the tables, we report the peak performance averaged over 5 training runs.
D Additional Encoder Ablations
This section reports some additional results regarding the encoding of the schema structure from Section 3.2. When encoding the schema structure through the order of its elements, Suhr et al. (2020) proposed to shuffle the order of the tables and columns to regularize training. We test this regularization and report the results in Table 10. This brings a slight improvement (fourth row of Table 10), but still gives results below a model that encodes this structure through encoder relations. Additional experiments not reported in Table 10 showed that using different ordering and shuffling did not bring an advantage when using the table-column relations.
We have also experimented with removing the foreign key relations from the model's input. The results reported in Table 6 confirm that conditioning the model on these relations is indeed necessary for the best performance.
E Details on Literal Substitution For Experiments on Single-Database Datasets
When following the protocol of Suhr et al. (2020), we observed that the model often made errors when copying literals, especially long ones. Often the literal copying mistake was the only one in an otherwise correctly predicted query, see Table 11 for an example. This issue limited our ability to rigorously assess
Relative Positioning Relations
Q→Q Distance d	Question token positions are separated by the distance d
C→C Distance d	Column positions are separated by the distance d
T→T Distance d	Table positions are separated by the distance d
Table And Column Relations
C→C Table Match	Columns are in the same table
C→T Any Table	Column is wildcard (*) in table
C→T Table Match	Column is part of table
T→C Any Table	Table contains wildcard (*) column
T→C Table Match	Table contains column
Primary-Key And Foreign-Key Relations
C→C Foreign-Key Forward	Column is foreign key to other column
C→C Foreign-Key Backward	Other column is foreign key to this column
C→T Foreign-Key	Column is foreign key of table
C→T Primary-Key	Column is primary key of table
T→C Foreign-Key	Table has column as foreign key
T→C Primary-Key	Table has column as primary key
T→T Foreign-Key Forward	Table has foreign key to other table
T→T Foreign-Key Backward	Other table has foreign key to this table
T→T Foreign-Key Bidirectional	Tables have foreign keys in both directions
Name-Based Schema Linking Relations
Q→C Name-based Match c	Question token matches column name with confidence c
Q→T Name-based Match c	Question token matches table name with confidence c
C→Q Name-based Match c	Column name matches question token with confidence c
T→Q Name-based Match c	Table name matches question token with confidence c
Content-Based Schema Linking Relations
Q→C Content-based match c	Question token matches column cell content with confidence c
C→Q Content-based match c	Column cell content matches question token with confidence c
Padding Relations
Q→Q Default	No Q→Q relation exists
Q→C Default	No Q→C relation exists
Q→T Default	No Q→T relation exists
C→Q Default	No C→Q relation exists
C→C Default	No C→C relation exists
C→T Default	No C→T relation exists
T→Q Default	No T→Q relation exists
T→C Default	No T→C relation exists
T→T Default	No T→T relation exists
Default	At least one token position is not populated
Table 8: List of encoder relations and their purposes. A  $\mathrm{Q} \rightarrow \mathrm{Q}$  relation points from one question token positions to another, a  $\mathrm{Q} \rightarrow \mathrm{C}$  relation points from a question token position to a column, and so on. The distances  $d$  count from  $-D$  to  $D$  in increments of 1, where the horizon  $D$  is configurable. The confidence  $c$  are binary and either high or low.
Self-Attention
Parent-Child	Node is parent of a node in the decoded AST
Child-Parent	Node is child of a node in the decoded AST
Identity	Self-loop, connects a node in the decoded AST to itself
Sibling-Distance d	Relative distance d of nodes that have the same parent
Default	Padding, at least one token position is not populated
Memory Attention
Copied-From	Input tokens that were copied
Default	Padding, at least one token position is not populated
Table 9: List of decoder relations and their purposes.
Model Variant	EM (dev.)
[column] [table] + relations	69.6 ± 0.8
[columns] [table]	60.0 ± 0.7
[table[columns]]	65.8 ± 1.3
[table[columns]] + shuffling	66.5 ± 1.3
Table 10: Complimentary results on the encoding of schema structure. Ordering of schema elements can be: (i) [column] [table]: first all the column types and names, then all the table names. (ii) [table[columns]]: each table is followed by the columns of that table.
Table 11: An example from the Academic dataset on which DuoRAT makes a literal copying error.
performance difference between different DuoRAT versions. To be able to evaluate improvements in predicting query structure, we modified their experimental protocol by replacing predicted literals with most similar ground-truth ones (we used the Levenshtein distance to assess similarity). Note that ignoring literals during evaluation is a standard practice in text-to-SQL literature (see e.g. exact match metric for Spider (Yu et al., 2018b), logical form accuracy for WikiSQL (Zhong et al., 2017) or the recently proposed test-suite accuracy by Zhong et al. (2020)). In future work, the literal copying issue can be addressed by generating copies of Spider queries with longer literals and adding them to the training set.