SlowGuess/NAACL_2021_OCR huggingface dataset:
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.
Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. 2020. On the potential of lexico-logical alignments for semantic parsing to SQL queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1849-1864, Online. Association for Computational Linguistics.
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372-8388, Online. Association for Computational Linguistics.
Lappoon R. Tang and Raymond J. Mooney. 2000. Automated construction of database interfaces: Integrating statistical and relational learning for semantic parsing. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 133-141, Hong Kong, China. Association for Computational Linguistics.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-sql generation with execution-guided decoding. arXiv preprint arXiv:1807.03100.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Sclizer: query synthesis from natural language. Proc. ACM Program. Lang., 1(OOPSLA):63:1-63:26.
Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. 2020. An imitation game for learning semantic parsers from user interaction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6883-6902, Online. Association for Computational Linguistics.
Semin Yavuz, Izzeddin Gur, Yu Su, and Xifeng Yan. 2018. What it takes to achieve  $100\%$  condition accuracy on WikiSQL. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1702-1711, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online. Association for Computational Linguistics.
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018a. TypeSQL: Knowledge-based type-aware neural text-to-SQL generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 588-594, New Orleans, Louisiana. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based SQL query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pages 5338-5349, Hong Kong, China. Association for Computational Linguistics.
Models	Easy	Medium	Hard	Extra Hard	All
# Examples	248	446	174	166	1034
RAT-SQL w/o value linking
w. BERTLarge	82.9	72.7	65.7	46.6	69.8
w. STRUG (Human Assisted)	84.9	76.2	67.0	55.0	73.3
w. STRUG (Automatic)	87.8	75.6	69.5	55.0	74.2
RAT-SQL
w. BERTLarge	84.1	74.9	67.4	52.8	72.3
w. STRUG (Human Assisted)	87.1	77.7	70.9	57.0	75.5
w. STRUG (Automatic)	88.7	77.4	69.2	53.6	74.9
Table 7: Execution accuracy on Spider dev set with different hardness levels.
Models	SELECT	WHERE	GROUP BY	ORDER BY
RAT-SQL w/o value linking
w. BERTLARGE	89.2	71.7	78.7	81.5
w. STRUG (Human Assisted)	91.2	74.8	79.0	84.0
w. STRUG (Automatic)	90.9	75.6	77.5	84.0
RAT-SQL
w. BERTLARGE	89.4	79.2	78.5	81.3
w. STRUG (Human Assisted)	91.3	80.8	80.6	85.7
w. STRUG (Automatic)	91.2	80.1	78.6	84.5
A.1 Filtering on the Suhr et al. (2020) Datasets
We use the filtering scripts $^{10}$  released by the authors of Suhr et al. (2020). More specifically, they remove examples that fall into the following categories: (1) a numeric or text value in the query is not copiable from the utterance (except for the numbers 0 and 1, which are often not copied from the input), (2) the result of the query is a empty table, or a query for count returns [1], (3) the query requires selecting more than one final column.
B.1 Detailed Results on Spider and Spider-Realistic
We show more detailed results on the Spider dev set and Spider-Realistic in Table 7, Table 8 and Table 9. From Table 7 we can see that STRUG brings significant improvements in all difficulty levels, and is not biased towards certain subset. Since STRUG mostly improves the structure grounding ability of the model, from Table 8 and Table 9, we can see that STRUG mainly increase the accuracy for WHERE and ORDER BY clauses, especially when database content is not available to the model. On the Spider-Realistic set, as the model cannot rely on simple string matching for structure grounding,
Table 8: F1 scores of Component Matching on Spider dev set.
Models	SELECT	WHERE	GROUP BY	ORDER BY
RAT-SQL w/o value linking
w. BERTLARGE	86.2	55.6	65.9	64.3
w. STRUG (Human Assisted)	88.9	61.9	70.4	64.1
w. STRUG (Automatic)	90.1	64.5	73.0	67.4
RAT-SQL
w. BERTLARGE	86.9	74.2	59.6	61.9
w. STRUG (Human Assisted)	89.0	76.8	69.9	63.5
w. STRUG (Automatic)	89.2	76.4	64.7	64.9
Table 9: F1 scores of Component Matching on SpiderRealistic set.
Models	ACC5-COL	ACC5-AGG	ACCw-COL	ACCw-VAL
SQLova (5%)
w. BERTLARGE	95.2	88.4	89.6	88.3
w. TaBERT	95.4	88.4	90.8	88.0
w. STRUG (Human Assisted)	95.5	88.9	92.6	91.5
w. STRUG (Automatic)	95.8	88.9	92.3	91.7
Table 10: Subtask performance on WikiSQL. S-COL, S-AGG, W-COL and W-VAL stands for tasks of predicting SELECT column, aggregation operator, WHERE columns and WHERE values, respectively.
we notice greater improvement using STRUG, especially for GROUP BY clauses.
B.2 Detailed Results on MongoDB
We show subtask performance for WikiSQL in Table 10, Fig. 7 and Fig. 4. Again, we can see that STRUG mainly improves WHERE column and WHERE value accuracy. From Fig. 6 we can see that with only  $1\%$  of training data, model with STRUG already has over 0.87 WHERE column accuracy and nearly 0.85 WHERE value accuracy.
$^{10}$ https://github.com/google-research/language/tree/master/language/xsp
(a) Where Column Accuracy
(b) Where Value Accuracy
Figure 6: Model performance on the test set with different fractions of training data.
(b) Where Value Accuracy
Figure 7: Model performance on the dev set during training with  $5\%$  of training data.