SlowGuess/NAACL_2021_OCR huggingface dataset:
As one of the first datasets to study cross-database text-to-SQL, Spider has been a widely used benchmark in assessing a model's ability to generalize to unseen programs and databases. However, as pointed out by Suhr et al. (2020), Spider eases the task by using utterances that closely match their paired SQL queries, for example by explicitly mentioning the column names in the question, while in practice NL references to columns usually differ from the original column name. To alleviate this problem, Suhr et al. (2020) propose to train the model with cross-domain dataset like Spider, and add another eight single-domain datasets like ATIS (Hemphill et al., 1990; Dahl et al., 1994) and GeoQuery (Zelle and Mooney, 1996) for evaluation. However, some of the datasets differ a lot from Spider, introducing many novel query structures and dataset conventions. As we can see from Table 1, their model (Suhr et al., 2020) has very poor performance in some datasets. In light of this, we present a new realistic and challenging evaluation set based on Spider. We first select a complex subset from
Example	Type
Show name, country, age for all singers ordered by age from the oldest to the youngest.	Remove
Find the number of concerts happened in the stadium with the highest capacity that can accommodate the most people.	paraphrase
How many pets have a greater weight than 10 are over 10 lbs?
Table 2: Examples of how we create Spider-Realistic from Spider. Phrases shown in italic exactly match with column names.
the Spider dev set where there are columns compared against values or used in clauses like ORDER BY. We then manually modify the NL questions in the subset ourselves to remove or paraphrase explicit mentions of columns names, except for the columns in SELECT clauses, while keeping the SQL queries unchanged. Some examples are shown in Table 2. This way we do not introduce extra challenges like adapting to new query structures but make it possible to fairly assess the model's capability in aligning text and tables. To make a more comprehensive comparison, we will also report results on the original Suhr et al. (2020) datasets.
5.1 Benchmarks and Base Models
Spider and the realistic evaluation sets. Spider (Yu et al., 2018b) is a complex cross-database text-to-SQL dataset. It contains 10k complex question-query pairs grounded on 200 databases where multiple tables are joined via foreign keys. In addition, we create a new realistic evaluation set Spider-Realistic as described in Section 4. We also include the original Suhr et al. (2020) datasets, for a more comprehensive comparison. For the base model, we use RAT-SQL (Wang et al., 2020) which is the state-of-the-art model according to the official leaderboard as of the submission time. To generate executable SQL queries, we modify the pointer generator in RAT-SQL to enable it to copy values from the question. We use the same trained model for evaluation on the Spider dev set and the realistic evaluation sets. Yu et al. (2018b) includes some single-domain text-to-SQL datasets like GeoQuery as extra training data for Spider. Following Suhr et al. (2020), we train the model with only the original Spider data, and discard additional training data used by some previous works like Yu et al. (2018b). We use both the set match accuracy (exact match)
<sup>2</sup>Some of the datasets contain operators that are not covered by Spider grammar or novel query structure like self join that does not exist in the training corpus.
Models	Spider-Realistic	ATIS	GeoQuery	Restaurants	Academic	IMDB	Yelp	Scholar	Advising
# Examples	508	289	532	27	180	107	54	394	309
Schema Only	Suhr et al. (2020)	-	0.8 (0.5)	41.6 (35.6)	3.7 (3.7)	8.2 (6.1)	24.6 (24.3)	19.8 (16.7)	0.5 (0.4)	2.3 (1.2)
RAT-SQL w/o value linking
w. BERTLARGE	52.4 ± 0.7 (46.9)	2.1 ± 0.6	41.2 ± 11.6	0.0 ± 0.0	5.9 ± 2.1	26.5 ± 5.0	12.3 ± 1.7	0.8 ± 0.4	1.6 ± 0.4
w. STRUG (Human Assisted)	57.8 ± 0.6 (53.3)	2.2 ± 0.2	45.5 ± 1.8	11.1 ± 9.1	14.8 ± 5.0	37.1 ± 1.8	15.4 ± 0.9	4.3 ± 1.7	2.2 ± 0.4
w. STRUG (Automatic)	60.3 ± 0.7 (54.9)	2.2 ± 0.2	50.9 ± 4.0	40.7 ± 5.2	12.4 ± 1.9	35.5 ± 2.0	13.0 ± 2.6	5.4 ± 0.7	1.0 ± 0.3
Content Used	RAT-SQL
w. BERTLARGE	62.1 ± 1.3 (58.1)	2.3 ± 0.2	47.3 ± 3.7	37.0 ± 18.9	15.6 ± 2.0	21.8 ± 1.6	16.0 ± 3.1	3.4 ± 1.4	6.4 ± 2.3
w. GRAPPA	- (59.3)
w. STRUG (Human Assisted)	65.7 ± 0.7 (62.2)	5.5 ± 1.1	59.5 ± 3.2	40.7 ± 13.9	18.7 ± 2.1	26.8 ± 2.9	21.6 ± 2.3	6.3 ± 1.8	6.9 ± 0.6
w. STRUG (Automatic)	65.3 ± 0.7 (62.2)	2.8 ± 0.7	57.5 ± 0.2	44.4 ± 32.7	20.2 ± 1.6	30.2 ± 5.8	18.5 ± 1.5	6.1 ± 0.5	5.2 ± 0.5
Table 3: Execution accuracy on the more realistic evaluation sets including Spider-Realistic and the Suhr et al. (2020) evaluation sets. For Spider-Realistic, we also show exact match accuracy in parentheses. For the Suhr et al. (2020) evaluation sets, we show results for the filtered set where examples with query returning empty set are excluded. Suhr et al. (2020) uses the WikiSQL dataset as additional training data, and we also show their results with only the Spider training data in parentheses.
Models	Exact	Exec	Exact (Test)
Schema Only	EditSQL (Zhang et al., 2019) w. BERT	57.6	-	53.4
IRNET (Guo et al., 2019) w. BERT	61.9	-	54.7
RYANSQL (Choi et al., 2020) w. BERT	70.6	-	60.6
Suhr et al. (2020) w. BERTLARGE+	65.0	69.0	-
RAT-SQL (Wang et al., 2020) w/o value linking
w. BERTLARGE	67.0 ± 0.6	69.8 ± 0.3	-
w. STRUG (Human Assisted)	70.5 ± 0.6	73.3 ± 0.4	67.4
w. STRUG (Automatic)	69.8 ± 0.3	74.2 ± 0.8	-
Content Used	Global-GNN (Boin et al., 2019)	52.7	-	47.4
TranX w. TaBERT (Yin et al., 2020)	64.5	-	-
RAT-SQL
w. BERTLARGE	69.8 ± 0.8	72.3 ± 0.6	-
w. GRAPPA (Yu et al., 2020)	73.4	-	69.6
w. STRUG (Human Assisted)	72.7 ± 0.7	75.5 ± 0.8	68.4
w. STRUG (Automatic)	72.6 ± 0.1	74.9 ± 0.1	-
from the official Spider evaluation script and execution accuracy $^3$  for evaluation on Spider and SpiderRealistic. On the Suhr et al. (2020) datasets, we use the official evaluation script $^4$  released by the authors and report execution accuracy.
WikiSQL. WikiSQL (Zhong et al., 2017) is a large-scale text-to-SQL dataset consists of over 80k question-query pairs grounded on over 30k Wikipedia tables. Although existing models are already reaching the upper-bound performance on this dataset (Hwang et al., 2019; Yavuz et al., 2018), mainly because of the simplicity of the SQL queries and large amount of data available for training, previous works have also used this dataset to demonstrate the model's generalization ability with limited training data (Yu et al., 2020; Yao et al., 2020). For the base model, we use SQLova (Hwang et al., 2019) without execution-guided decoding. Follow-
Table 4: Results on Spider. The top half shows models using only database schema, the bottom half shows models using the database content. We train our model three times with different random seeds and report the mean and standard deviation here.
Models	ACCIf	ACCex
HydraNet (Lyu et al., 2020)	83.8	89.2
X-SQL (He et al., 2019)	83.3	88.7
SQLova (Hwang et al., 2019)
w. BERTLarge	82.1	87.3
w. TaBERT	82.5	87.9
w. STRUG (Human Assisted)	82.1	87.5
w. STRUG (Automatic)	82.4	87.8
SQLova (5%)
w. BERTLarge	70.7	77.0
w. TaBERT	71.5	78.0
w. STRUG (Human Assisted)	75.6	81.6
w. STRUG (Automatic)	75.6	81.4
Table 5: Performance on WikiSQL. Here we show logical form accuracy and execution accuracy on the test set. (5%) means random sampling 5% of original training data for training.
ing the official leaderboard, we report both logical form accuracy and execution accuracy.
For all experiments, we use the BERT implementation from Huggingface (Wolf et al., 2020) and the pretrained  $\mathrm{BERT}_{\mathrm{LARGE}}$  model from Google  $^{5}$ . For pretraining, we use Adam optimizer (Kingma and Ba, 2015) with a initial learning rate of 2e-5 and batch size of 48. In both settings, we use  $K_{\text{neg}} = 1$  and pretrains our model for 5 epochs. We use 4 V100 GPUs for pretraining, which takes less than 4 hours.
For Spider and the realistic evaluation sets, we use the official implementation of RAT-SQL  $^{6}$  and modify it to generate executable SQL queries. We follow the original settings and do hyperparameter search for learning rate (3e-4, 7.44e-4) and
3We execute case insensitive SQL queries, and compare the returned table.
4https://github.com/google-research/ language/tree/master/language/xsp
5We use the BERT-Large, Uncased (Whole Word Masking) model from https://storage.googleapis.com/bert_models/2019_05_30/www_uncased_L-24_H-1024_A-16.zip
$^{6}$ https://github.com/microsoft/rat-sql
warmup step (5k, 10k). We use the same polynomial learning rate scheduler with warmup and train for 40,000 steps with batch size of 24. The learning rate for the pretrained encoder (e.g. BERT) is 3e-6 and is frozen during warmup.
For WikiSQL, we use the official SQLova implementation<sup>7</sup>. We use the default setting with learning rate of 1e-3 for the main model and learning rate of 1e-5 for the pretrained encoder. We train the model for up to 50 epochs and select the best model using the dev set.
Spider. We first show results on Spider dev set in Table 4. The original Spider setting assumes only the schema information about the target database is known in both training and evaluation phase, as the content of the database may not be accessible to the system due to privacy concern. More recently, some works have tried to using the database content to help understand the columns and link with the NL utterance. Here we show results for both settings. In the first setting where only schema information is known, we disable the value-based linking module in RAT-SQL. As we can see from Table 4, replacing  $\mathrm{BERT}_{\mathrm{LARGE}}$  with STRUG consistently improves the model performance in both settings. Under the setting where content is available, using STRUG achieves similar performance as GRAPPA and outperforms all other models. GRAPPA uses both synthetic data and larger text-table corpus for pretraining. However, it mainly learns inductive bias from the synthetic data while our model focuses on learning text-table association knowledge from the parallel text-table data. In error analysis on the Spider dev set, we notice that our best model<sup>8</sup> corrects 76 out of 270 wrong predictions made by GRAPPA while GRAPPA corrects 80 out of 274 wrong predictions made by our model. This demonstrates that the two pretraining techniques are complementary and we expect combining them can lead to further performance improvement. For results on different difficulty levels and components, please see Appendix B.1.
More realistic evaluation sets. Results on the realistic evaluation sets are summarized in Table 3. Firstly, we notice the performance of all models drops significantly on Spider-Realistic, demonstrating that inferring columns without explicit hint is
Figure 4: Execution Accuracy on the WikiSQL test set with different fractions of training data.
Figure 5: Execution Accuracy on the WikiSQL dev set during training with  $5\%$  of training data.
a challenging task and there is much room for improvement. Secondly, using STRUG brings consistent improvement over  $\mathrm{BERT}_{\mathrm{LARGE}}$  in all realistic evaluation sets. In the Spider-Realistic set, using STRUG also outperforms GRAPPA by  $2.9\%$ . Under the original Suhr et al. (2020) setting, combining RAT-SQL with STRUG significantly outperforms Suhr et al. (2020) in all datasets, despite that we do not include WikiSQL as additional training data as they did. Thirdly, comparing results in Table 4 with Table 3, using STRUG brings larger improvement over  $\mathrm{BERT}_{\mathrm{LARGE}}$  in the more realistic evaluation sets. As shown in Table 1, the original Spider dataset has a high column mention ratio, so the models can use exact match for column grounding without really understanding the utterance and database schema. The more realistic evaluation sets better simulate the real world scenario and contain much less such explicit clues, making the text-table alignment knowledge learned by STRUG more valuable. For case studies on Spider-Realistic, please check Section 5.4.
WikiSQL. Results on WikiSQL are summarized in Table 5. When using the full training corpus, we notice that using STRUG achieves similar performance as BERTLarge. This is probably because of
<sup>9</sup>We use the checkpoint provided by the author, which achieves  $73.8\%$  exact match accuracy on the Spider dev set. Here we only evaluate on Spider-Realistic with exact match accuracy because their model does not generate values and includes IMDB and Geo as extra training data.
<sup>7</sup>https://github.com/naver/sqlova
$^{8}$ RAT-SQL w. STRUG (Human Assisted)
Table 6: Case study.
the large size of training data and the simple SQL structure of WikiSQL. To better demonstrate that the knowledge learned in pretraining can be effectively transferred to text-to-SQL task and reduce the need for supervised training data, we also conduct experiments with randomly sampled training examples. From Fig. 4 we can see that with only  $1\%$  of training data (around 500 examples), models using STRUG can achieve over 0.70 accuracy, outperforming both  $\mathrm{BERT}_{\mathrm{LARGE}}$  and TaBERT by a large margin. STRUG brings consistent improvement over  $\mathrm{BERT}_{\mathrm{LARGE}}$  until we use half of the training data, where all models reach nearly the same performance as using the full training data. We also show the training progress using  $5\%$  of training data in Fig. 5. We can see that STRUG also helps speed up the training progress. For more break-down results on several subtasks, please see Appendix B.2.
Comparison of human assisted and automatic setting. In all benchmarks, we notice that STRUG pretrained using the automatic setting actually performs similarly as the setting where cell annotations are used. This indicates the effectiveness of our heuristic for cell annotation and the potential to pretrain STRUG with more unannotated parallel text-table data.
We compare the predictions made by RAT-SQL  $w$ . BERTLarge and  $w$ . STRUG (Automatic). Some examples are shown in Table 6. In the first example from Spider-Realistic, we can see that the model  $w$ . BERTLarge fails to align tournaments with the tourney_name column, because of string mismatch.
In the second example from IMDB, although the model correctly recognizes James Bond as value reference, it fails to ground it to the correct column which is movie_title. This supports our hypothesis that using STRUG helps to improve the structure grounding ability of the model.
In this paper, we propose a novel and effective structure-grounded pretraining technique for text-to-SQL. Our approach to pretraining leverages a set of novel prediction tasks using a parallel texttable corpus to help solve the text-table alignment challenge in text-to-SQL. We design two settings to obtain pretraining labels without requiring complex SQL query annotation: using human labeled cell association, or leveraging the table contents. In both settings, STRUG significantly outperforms BERTLarge in all the evaluation sets. Meanwhile, although STRUG is surprisingly effective (using only 120k text-table pairs for pretraining) and performs on par with models like TaBERT (using 26m tables and their English contexts) and GRAPPA (using 475k synthetic examples and 391.5k examples from existing text-table datasets) on Spider, we believe it is complementary with these existing text-table pretraining methods. In the future, we plan to further increase the size of the pretraining corpus, and explore how to incorporate MLM and synthetic data.
Dataset. In this work, we re-purpose an existing table-to-text generation dataset ToTTo (Parikh et al., 2020) for our pretraining. We obtain labels for our three pretraining tasks via weak supervision, which uses only the raw sentence-table pairs, or the cell annotations and revised descriptions that are already included in ToTTo dataset. As a result, no extra human effort is required for collecting our pretraining corpus. We also curate a more realistic evaluation dataset for text-to-SQL based on Spider dev set. In particular, we first select a complex subset from the Spider dev set and manually revise the NL questions to remove the explicit mention of column names. The detailed description of the process can be found in Section 4. The first author manually revised all the questions himself, which results in 508 examples in total.
Application. We focus on the task of text-to-SQL, which is a fundamental task for building natural language interfaces for databases. Such interface
can enable non-expert users to effortlessly query databases. In particular, here we focus on improving the structure grounding ability of text-to-SQL models, which is critical in real-world use cases. We evaluate our model with the widely used Spider benchmark and several more realistic datasets. Experimental results show that our method brings significant improvement over existing baselines, especially on more realistic settings.
Computing cost. We use 4 V100 GPUs for pretraining, and 1 V100 GPU for finetuning the model for text-to-SQL on Spider and WikiSQL. One advantage of our method is its efficiency. In our experiments, we pretrain the model for only 5 epochs, which can finish within 4 hours. For comparison, the largest TaBERT model (Yin et al., 2020) takes 6 days to train for 10 epochs on 128 Tesla V100 GPUs using mixed precision training.
We thank Bo Pang, Tao Yu for their help with the official Spider evaluation. We also thank anonymous reviewers for their constructive feedback.
Ion Androutsopoulos, G. Ritchie, and P. Thanisch. 1995. Natural language interfaces to databases - an introduction. Nat. Lang. Eng., 1:29-81.
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019. Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3659-3664, Hong Kong, China. Association for Computational Linguistics.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929-7942, Online. Association for Computational Linguistics.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020b. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026-1036, Online. Association for Computational Linguistics.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. Ryansql: Recursively
applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. arXiv preprint arXiv:2004.03125.
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-sql: reinforce schema representation with context. arXiv preprint arXiv:1908.08113.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer. 2016. A large public corpus of web tables containing time and context metadata. In Proceedings of the 25th International Conference Companion on World Wide Web, pages 75-76.
Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. 2020. Re-examining the role of schema linking in text-to-SQL. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943-6954, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
F. Li and H. V. Jagadish. 2014a. Nalir: an interactive natural language interface for querying relational databases. In SIGMOD Conference.
Fei Li and HV Jagadish. 2014b. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020. Hybrid ranking network for text-to-sql. Technical Report MSR-TR-2020-7, Microsoft Dynamics 365 AI.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. Association for Computational Linguistics.