SlowGuess/NAACL_2022_OCR huggingface dataset:
We generate the Spider-CG based on the combination of Spider-SS sub-sentences split by the algorithm introduced in Section 2.2. We can reuse this algorithm to split the sentence in Spider-CG and then compare the splitting results with the Spider-SS sub-sentences to evaluate the stability of the splitting algorithm. We consider that a deviation of one or two tokens in the splitting result is acceptable. For example, in Figure 1, we consider that putting the comma of the third sub-sentence into the second sub-sentence does not change the meaning of sub-sentences, same for moving both the comma and the word 'and'.
Table 4 presents the similarity between subsentences in Spider-SS and Spider-CG, which are generated by the same split algorithm under the deviation of one or two words. The similarity exceeds  $90\%$  in all evaluation set when two deviation words are allowed. Considering that the model trained on the Spider-SS does not require consistent split results, as discussed in Section 2.2, the similarity results of the splitting algorithm are good enough. The similarity of CG-SUB is higher than that of CG-APP, which means the more complex the sentence, the greater the challenge to the algorithm. Although the algorithm has been refined on the training set, the similarity between training and development in CG-SUB and CG-APP is close,
Dataset	Deviation <= 1	Deviation <= 2
CG-SUBT	93.2%	94.4%
CG-SUBD	92.9%	94.1%
CG-APPT	86.0%	90.4%
CG-APPD	88.9%	92.6%
Table 4: The similarity between sub-sentences in Spider-SS and Spider-CG generated by the same split algorithm under the deviation of one or two tokens.
showing that the algorithm performs consistently for sentences in unseen domains. In summary, we consider that as long as the sentences are not more complex than CG-APP, the algorithm can be used stably in other text-to-SQL datasets.
5.4 Model Results
Table 5 presents the exact match accuracy on the five different evaluation sets. In the two OOD datasets,  $\mathrm{CG - APP_T}$  and  $\mathrm{CG - APP_D}$ , the performance of all models has dropped by about  $10\%$  to  $30\%$ . However, the models trained on Spider-SS significantly outperform those trained on Spider when evaluated on the OOD datasets. We use the sentence split algorithm to split every sentence before inputting the models with subscript (S). Although the split sub-sentences are not completely consistent with those seen during training, it did not prevent the models with subscript (S) from getting good performance, i.e., the  $\mathrm{RATSQL_{G(S)}}$  consistently outperforms all other models on all evaluation sets. These results demonstrate that the subsentence-based method can improve the generalization performance. The limitation is that the method may not be compatible with the original model, e.g., original hyperparameters in  $\mathrm{RATSQL_{B(S)}}$  are not workable, and the performance of GNN on the  $\mathrm{Spider_D}$  and  $\mathrm{CG-SUB_D}$  is degraded.
Each model has a close result between the unseen  $\mathrm{Spider}_{\mathrm{D}}$  and  $\mathrm{CG-SUB}_{\mathrm{D}}$ , indicating that from the perspective of the model, the synthetic sentences are pretty similar to NL. Therefore, we believe the performance on  $\mathrm{CG-SUB}_{\mathrm{D}}$  can be generalized to the real world. Moreover, considering that the algorithms for generating  $\mathrm{CG-SUB}_{\mathrm{D}}$  and  $\mathrm{CG-APP}_{\mathrm{D}}$  are close (see Appendix A), we can further speculate that the synthetic sentences of  $\mathrm{CG-APP}_{\mathrm{D}}$  are also close to natural language.
The models with NatSQL is significantly better than that without NatSQL when evaluated on Spider-CG. One of the reasons is that the training
data of Spider and Spider-SS are about  $10\%$  different, which leads to the performance degradation in the model trained on Spider when evaluated on the SQL generated by the NatSQL of Spider-SS, and vice versa. On the other hand, experiments in (Gan et al., 2021b) show that NatSQL improve the model performance in extra hard SQL. Therefore,  $\mathrm{RATSQL}_{\mathrm{G(N)}}$  and  $\mathrm{RATSQL}_{\mathrm{B(N)}}$  suffer less performance degradation in CG-APP $_{\mathrm{T}}$  and CG-APP $_{\mathrm{D}}$  than  $\mathrm{RATSQL}_{\mathrm{G}}$  and  $\mathrm{RATSQL}_{\mathrm{B}}$ .
6 Limitation of this Work
The Spider-SS and Spider-CG are based on Spider, an English large-scale text-to-SQL dataset, and we did not extend the experiment to other language and text-to-SQL datasets. Therefore, we did not verify whether these methods work well in other languages and datasets. Besides, since this work is based on NatSQL, there will be around  $5\%$  of NatSQL that can not be converted to the correct SQL.
Data augmentation for text-to-SQL models. Data augmentation has been commonly used for improving performance (Xiong and Sun, 2019; Li et al., 2019). In the context of text-to-SQL generation, Yu et al. (2018a) generate synthetic training samples from some pre-defined SQL and NL question templates. Parikh et al. (2020) introduces an table-to-text dataset with over 120,000 examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. Yu et al. (2021) sample from the given examples and then give a large number of tables to generate new synthetic examples. Shi et al. (2021) present a model pre-training framework that jointly learns representations of NL utterances and table schemas by leveraging generation models to generate pre-train data. Our proposed Spider-CG dataset can be used for data augmentation.
Compositional generalization for semantic parsing. Compositional generalization for semantic parsing has captured wide attention recently (Finegan-Dollak et al., 2018; Oren et al., 2020; Furrer et al., 2020; Conklin et al., 2021). Most prior works on text-to-SQL tasks focus on the cross-domain generalization, which mainly assess how the models generalize the domain knowledge to
Approach	SpiderD	CG-SUBT	CG-SUBD	CG-APPT	CG-APPD
RATSQLG	72.7%	80.9%	70.3%	45.2%	44.2%
RATSQLG(N)	73.9%	90.2%	75.0%	67.8%	60.5%
RATSQLG(S)	74.5%	91.4%	76.7%	82.5%	68.3%
RATSQLB	72.0%	79.5%	72.0%	45.1%	47.2%
RATSQLB(N)	72.1%	83.2%	69.4%	54.6%	53.1%
RATSQLB(S)	71.9%	91.0%	72.6%	79.8%	61.5%
RATSQL(N)	63.2%	79.1%	60.7%	40.6%	34.5%
RATSQL(S)	64.7%	88.8%	63.3%	72.1%	44.1%
GNN(N)	54.4%	67.3%	57.5%	30.4%	25.1%
GNN(S)	49.3%	71.9%	51.8%	52.1%	34.6%
Table 5: Exact match accuracy on evaluation sets.
new database schemas (Suhr et al., 2020; Gan et al., 2021a). On the other hand, Shaw et al. (2021) introduces TMCD splits for studying compositional generalization in semantic parsing, where they aim to maximize the divergence of SQL compounds between the training and test sets.
Although both the TMCD split and our SpiderCG can be used to evaluate the text-to-SQL compositional generalization ability, their problem setting is different. TMCD split is based on SQL syntax structure, while Spider-CG is based on the natural language syntax, which leads to different requirements for compositional generalization ability. For example, TMCD splits require model learning "Give me the name of students who is the oldest" can predict the "Give me the name of the oldest student" since their SQL is the same. Spider-CG does not expect the model to do so because the syntax of questions is different, i.e., "Give me the name of students who is the oldest" contains two sub-sentences, and none of them is close to the "Give me the name of the oldest student". In other words, Spider-CG requires the model learning "List the id of the oldest dog" can predict the "Give me the name of the oldest student".
Our model is inspired by prior works on neural parsers constructed to capture granular information from a whole. Yin et al. (2021) describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Herzig and Berant (2021) propose SpanBasedSP, a parser that predicts a span tree over an input utterance, and dramatically improves performance on splits that require compositional generalization. Chen et al. (2020) propose the Neural-Symbolic Stack machine (NeSS), which integrates a symbolic
stack machine into a seq2seq generation framework, and learns a neural network as the controller to operate the machine. However, these works are based on datasets where component alignment is relatively easy to achieve; but for more complex text-to-SQL, their methods cannot be used directly. Our proposed Spider-SS can be used to replace or evaluate the alignment algorithm.
We introduce Spider-SS and Spider-CG for measuring compositional generalization of text-to-SQL models. Specifically, Spider-SS is a human-curated sub-sentence-based text-to-SQL dataset built upon the Spider benchmark. Spider-CG is a synthetic text-to-SQL dataset constructed by substituting and appending sub-sentences of different samples, so that the training and test sets consist of different compositions of sub-sentences. We found that the performance of previous text-to-SQL models drop dramatically on the Spider-CG OOD subset, while modifying the models to fit the segmented data of Spider-SS improves compositional generalization performance.
We thank the anonymous reviewers for their helpful comments. Matthew Purver acknowledges financial support from the UK EPSRC under grant EP/S033564/1, and from the Slovenian Research Agency for research core funding (No. P2-0103 and No. P5-0161). Xinyun Chen is supported by the Facebook Fellowship.
Ben Bogin, Jonathan Berant, and Matt Gardner. 2019. Representing schema structure with graph neural networks for text-to-SQL parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560-4565, Florence, Italy. Association for Computational Linguistics.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020. Compositional generalization via neural-symbolic stack machines. In Advances in Neural Information Processing Systems, volume 33, pages 1690-1701. Curran Associates, Inc.
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally generalize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3322-3335, Online. Association for Computational Linguistics.
Marie-Catherine de Marnee and Christopher D. Manning. 2016. Stanford typed dependencies manual.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. pages 351-360.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Scharli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. CoRR, abs/2007.08970.
Yujuan Gan, Xinyun Chen, and Matthew Purver. 2021a. Exploring underexplored limitations of cross-domain text-to-sql generalization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021b. Natural sql: Making sql easier to infer from natural language specifications.
Jonathan Herzig and Jonathan Berant. 2021. Span-based semantic parsing for compositional generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pages 908-921, Online. Association for Computational Linguistics.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2873-2882. PMLR.
Jingjing Li, Wenlu Wang, Wei Shinn Ku, Yingtao Tian, and Haixun Wang. 2019. SpatialNLI: A spatial domain natural language interface to databases using spatial comprehension. In GIS: Proceedings of the ACM International Symposium on Advances in Geographic Information Systems, pages 339-348, New York, NY, USA. Association for Computing Machinery.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. 2020. Compositional generalization by learning analytical expressions. In Advances in Neural Information Processing Systems, volume 33, pages 11416-11427. Curran Associates, Inc.
Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2482-2495, Online. Association for Computational Linguistics.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.
Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 922-938, Online. Association for Computational Linguistics.
Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos Santos, and Bing Xiang. 2021. Learning contextual representations for semantic parsing with generation-augmented pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13806-13814.
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372-8388, Online. Association for Computational Linguistics.
Bailin Wang, Mirella Lapata, and Ivan Titov. 2021. Structured reordering for modeling latent alignments in sequence transduction. In Thirty-Fifth Conference on Neural Information Processing Systems.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Hongvu Xiong and Ruixiao Sun. 2019. Transferable Natural Language Interface to Structured Queries Aided by Adversarial Generation. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 255-262. IEEE.
Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span-level supervised attention. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2810-2823, Online. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmented pre-training for table semantic parsing.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1653-1663, Brussels, Belgium. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A largescale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
A Further Discussion of Algorithm 1
As discussed in Section 3.3, we need to ensure that the Spider-CG examples meet the criteria of containing required information and being reasonable. To ensure that the generated Spider-CG sentence contains the required information, the compositional element needs to contain all the information needed to derive the target NATSQL clause. Thus some sub-sentence can not be a compositional element, such as the last sub-sentence of examples 1 and 2 in Figure 4. Among them, example 1 misses ORDER BY information; example 2 misses Total_Horses column information. In contrast, the sub-sentence of the two Spider-SS examples in Figure 2 contains the required information and can be compositional elements. So, we can filter out the sub-sentences containing the "NO MENTIONED" and "extra" label, and collect the rest as compositional elements.
The 'can_be_substituted_by' and 'can.append' function in Algorithm 1 are used to ensure that the generated sentences are reasonable. For the convenience of discussion, we refer to them as 'sub' and 'app' functions for short. These two functions examine the generated sentences from complexity, logic and coherence.
Complexity checks are used to limit the complexity of the generated examples to no more complex than the upper bound of the Spider dataset. On the NATSQL side, both functions do not allow the generated NATSQL containing: 1) more than one subqueries; 2) more than one HAVING condition; 3) more than three WHERE conditions; 4) more than one ORDER BY clause; 5) new conditions for a subquery. On the NL side, since the substitution did not clearly increase the sentence complexity, only the 'app' function performs the NL complexity checks to restrict the number of sub-sentence to less than 4.
Logic checks are used to prevent generating contradictory examples. First, logic checks filter out
examples with repeated WHERE conditions. Then, it filters out examples whose WHERE condition negates the query content, e.g., what is name of student that do not have any student. Finally, since the GROUP BY clause is often expressed implicitly, substituting or appending elements containing the GROUP BY clause may introduce logical errors. Thus, logic checks require the GROUP BY clauses to be the same if they exist.
Coherence checks are used to ensure that the expression of the generated sentence is coherent. As discussed in Section 2.2, we separate a sentence into main clause, subordinate clauses, and modifiers. The main clause expresses what you want to query, i.e., corresponding to the SELECT clause. Subordinate clauses and modifiers are restrictions on the query, i.e., corresponding to WHERE and ORDER BY clauses. Therefore, compositional elements only contain subordinate clauses and modifiers. The way to ensure the coherence of sentences by sub function is to require the substitution sub-sentences modify the same noun. Suppose the schema table of the NatSQL in a compositional element appears in advance. In that case, we consider its sub-sentence modifies the table noun because repeating a known object  ${}^{4}$  can only be a further modification. However, if the schema table has not appeared before, we consider that the sub-sentence modifies its previous word since a subordinate clause usually comes immediately after the noun it describes.
There is a high similarity between the app and sub function, but the inspection between the substituted elements is changed to the inspection between the new element and the last element in the original sentence. Therefore, the appended subsentence must modify the same noun as the last sub-sentence. If a compositional element passes the app function, we use the word 'and' or 'or' to connect it where the word 'or' can only connect a WHERE condition. Table 6 discuss some examples for ease of understanding.
B Unseen SQL Structure Template in Spider-CG
Although we limit the complexity of the generated examples lower than the upper bound of the Spider dataset, Spider-CG still contains unseen SQL structure templates. For example, the NatSQL
template 'SELECT COL WHERE COL > VAL or count(Table.*) >= VAL GROUP BY COL' and corresponding SQL can not be found in the original Spider. The new templates may degrade the performance of models.
C Spider-SS Annotation Steps
We build an annotation tool to show the sub-sentence and sub-SQL split from a question-NatSQL pair. During annotation, the annotators select the corresponding sub-SQL for sub-sentences. In rare cases, if there is no suitable sub-SQL, the annotators would write a new one, such as the example-1 in Figure4. We recruit two graduate students major in computer science to annotate the dataset manually. They are trained with a detailed annotation guideline and some samples. One is allowed to start after his trial samples are approved by the whole team. Each example is annotated twice. If the annotations are different, the final annotation will be decided by a discussion. If two annotators discuss and conclude that one of the annotations is wrong and the other is correct, the correct annotation is retained. Otherwise, the authors will annotate this example if no such conclusion can be drawn.
D Execution Match
The execution match metric measures whether the query results from the predicted query are the same as the gold query results. The original RATSQL can not generate the executable SQL until extending the NatSQL. The NatSQL2SQL conversion would analyze the utterance and generate executable SQL, irrelevant to the RATSQL model. Thus we only report the results of models with NatSQL. Since the execution match is similar to the exact match, we only report the top models in Table 7. Similar to the exact match,  $\mathrm{RATSQL}_{\mathrm{G(S)}}$  outperform other models in most evaluation set except on the CG-APP  $T$ .
${}^{4}$  A table is usually an object whose attributes are its columns in relational databases.