SlowGuess/NAACL_2022_OCR huggingface dataset:
Yujian Gan $^{1}$  Xinyun Chen $^{2}$  Qiuping Huang $^{4}$  Matthew Purver $^{1,3}$
$^{1}$ Queen Mary University of London
$^{2}$ UC Berkeley  $^{3}$ Jožef Stefan Institute
Nanning Central Sub-branch of the People's Bank of China
{y.gan,m.purver}@qmul.ac.uk xinyun.chen@berkeley.edu
qiuping_h@foxmail.com
In text-to-SQL tasks — as in much of NLP — compositional generalization is a major challenge: neural networks struggle with compositional generalization where training and test distributions differ. However, most recent attempts to improve this are based on word-level synthetic data or specific dataset splits to generate compositional biases. In this work, we propose a clause-level compositional example generation method. We first split the sentences in the Spider text-to-SQL dataset into sub-sentences, annotating each sub-sentence with its corresponding SQL clause, resulting in a new dataset Spider-SS. We then construct a further dataset, Spider-CG, by composing Spider-SS sub-sentences in different combinations, to test the ability of models to generalize compositionally. Experiments show that existing models suffer significant performance degradation when evaluated on Spider-CG, even though every sub-sentence is seen during training. To deal with this problem, we modify a number of state-of-the-art models to train on the segmented data of Spider-SS, and we show that this method improves the generalization performance.<sup>1</sup>
Neural models in supervised learning settings show good performance on data drawn from the training distribution. However, generalization performance can be poor on out-of-distribution (OOD) samples (Finegan-Dollak et al., 2018; Suhr et al., 2020; Kaushik et al., 2020; Sagawa et al., 2020). This might be the case even when the new samples are composed of known constituents; e.g., on the SCAN dataset (Lake and Baroni, 2018), many models give incorrect predictions for the input "jump twice and walk", even when "jump twice", "walk", and "walk twice" are seen during training. This
(often lacking) ability to generalize to novel combinations of elements observed during training is referred to as compositional generalization.
Previous work on compositional generalization in text-to-SQL focuses on query split. For example, Shaw et al. (2021) propose TMCD split based on SQL atoms and compounds analysis and question split based on length. Finegan-Dollak et al. (2018) proposes a query template-based split with word substitution that was much more challenging than the question split. However, these splits are limited by the dataset content, making it difficult to construct a challenging benchmark while ensuring that every question phrase (sub-sentence) appears in the training set.
Previous works (Chen et al., 2020; Wang et al., 2021; Liu et al., 2020) improve generalization by enhancing the model's component awareness. Similarly, Yin et al. (2021) and Herzig and Berant (2021) propose span-based semantic parsers that predict a sub-program over an utterance span. However, these works are based on datasets where component alignment is relatively easy to achieve; but for more complex text-to-SQL, their methods cannot be used directly. For example, as shown in the lower part of Figure 1, to align the sub-sentence with the sub-SQL, the algorithm needs to know that 'youngest' corresponds to 'age', and 'weigh' corresponds to 'weight'. For small or single-domain settings, such an alignment algorithm can be built by establishing rules; however, there is currently no simple and feasible alignment method for large complex cross-domain text-to-SQL, as in e.g. the Spider benchmark (Yu et al., 2018b).
In this work, we first introduce a new dataset, Spider-SS (SS stands for sub-sentence), derived from Spider (Yu et al., 2018b); Figure 1 compares the two. To build Spider-SS, we first design a sentence split algorithm to split every Spider sentence into several sub-sentences until indivisible. Next, we annotate every sub-sentence with its cor
$^{1}$ Our code and dataset are available at https://github.com/ygan/SpiderSS-SpiderCG
Findings of the Association for Computational Linguistics: NAACL 2022, pages 831 - 843 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: A natural language sentence in the original Spider benchmark is split into three sub-sentences in Spider-SS, where each sub-sentence has a corresponding NatSQL clause.
responding SQL clause, reducing the difficulty of this task by using the intermediate representation language NatSQL (Gan et al., 2021b), which is simpler and syntactically aligns better with natural language (NL). Spider-SS thus provides a new resource for designing models with better generalization capabilities without designing a complex alignment algorithm. Furthermore, it can also be used as a benchmark for evaluating future alignment algorithms. To our knowledge, this is the first sub-sentence-based text-to-SQL dataset.
Our annotated Spider-SS provides us with sub-sentences paired with NatSQL clauses, which serve as our elements. Based on Spider-SS, we then construct a further dataset Spider-CG (CG stands for compositional generalization), by substituting sub-sentences with those from other samples, or composing two sub-sentences to form a more complicated sample. Spider-CG contains two subsets; Figure 2 shows one example for each. The first subset contains 23,569 examples generated by substituting sub-sentences; we consider most data in this subset as in-distribution. The second subset contains 22,030 examples generated by appending sub-sentences, increasing the length and complexity of the sentence and the SQL query compared to the original samples; we consider this subset as OOD. We demonstrate that when models are trained only on the original Spider dataset, they suffer a significant performance drop on the second OOD subset of Spider-CG, even though the domain appears in the training set.
To improve the generalization performance of text-to-SQL models, we modify several previous state-of-the-art models so that they can be applied
Figure 2: Two Spider-CG samples generated by: (1) substituting the sub-sentence with one from another example; or (2) composing sub-sentences from 2 examples in Spider-SS.
to the Spider-SS dataset, with the model trained sub-sentence by sub-sentence. This modification obtains more than  $7.8\%$  accuracy improvement on the OOD subset of Spider-CG.
In short, we make the following contributions:
- Besides the sentence split algorithm, we introduce Spider-SS, a human-curated subsentence-based text-to-SQL dataset built upon the Spider benchmark, by splitting its NL questions into sub-sentences.
- We introduce the Spider-CG benchmark for measuring the compositional generalization performance of text-to-SQL models.
- We show that text-to-SQL models can be adapted to sub-sentence-based training, improving their generalization performance.
2 Spider-SS
Figure 1 presents a comparison between Spider and Spider-SS. Unlike Spider, which annotates a whole SQL query to an entire sentence, Spider-SS annotates the SQL clauses to sub-sentences. Spider-SS uses NatSQL (Gan et al., 2021b) instead of SQL for annotation, because it is sometimes difficult to annotate the sub-sentences with corresponding
Figure 3: Dependency structure of a sentence and how to split this sentence into three sub-sentences.
SQL clauses due to the SQL language design. The Spider-SS provides a combination algorithm that collects all NatSQL clauses and then generates the NatSQL query, where the NatSQL query can be converted into an SQL query.
The purpose of building Spider-SS is to attain clause-level text-to-SQL data avoiding the need for an alignment algorithm that is hard to build based on the complex large cross-domain text-to-SQL dataset, e.g., Spider benchmark. Besides, we can generate more complex examples through different combination of clauses from Spider-SS. Consistent with Spider, Spider-SS contains 7000 training and 1034 development examples, but Spider-SS does not contain a test set since the Spider test set is not public. There are two steps to build Spider-SS. First, design a sentence split algorithm to cut the sentence into sub-sentences, and then manually annotate the NatSQL clause corresponding to each sub-sentence.
2.2 Sentence Split Algorithm
We build our sentence split algorithm upon the NL dependency parser spaCy  $^{2}$ , which provides the grammatical structure of a sentence. Basically, we split the sentence with the following dependencies: prep, relcl, advcl, acl, nsubj, npadvmod, csubj, nsubjpass and conj. According to (de Marnee and Manning, 2016), these dependencies help us separate the main clause, subordinate clauses, and modifiers. Figure 3 shows the dependency structure of a sentence and how to split this sentence into three sub-sentences. However, not every sentence would be split since there are some non-splittable sentences, such as the third example in Figure 4, with the same annotation as the Spider dataset. Although this method can separate sentences well in most cases, due to the variability of natural language, some examples cannot be perfectly split.
To address the remaining issues in sentence split, we design some refinement steps tailored to text-to-SQL applications. For example, when the phase of
Figure 4: Spider-SS examples in three special cases.
a schema column or table is accidentally divided into two sub-sentences, these two sub-sentences are automatically concatenated. Besides, when there is only one word in a sub-sentence, the corresponding split should also be undone.
We sampled 500 examples from the Spider-SS development set to evaluate the acceptability of splitting results manually, and only  $< 3\%$  of the splitting results are unsatisfactory. For example, in the splitting results of the first example in Figure 4, the last two sub-sentence should be combined to correspond to "ORDER BY Customer.Email_Address, Customer.Phone_Number ASC". In this example, we did not simply give an "ORDER BY Customer.Phone_Number ASC" to the last sub-sentence, because it does not mention anything related to "ORDER BY". Here, we introduce "extra", a new NATSQL keyword designed for the Spider-SS dataset, indicating that this sub-sentence mentions a column that temporarily does not fit in any other NATSQL clauses. When combining NATSQL clauses into the final NATSQL query, the combining algorithm determines the final position for the "extra" column based on the clauses before and after. Note that even if there is a small proportion of unsatisfactory splitting results, as long as the model trained on Spider-SS can give the correct output according to the input sub-sentence, the quality of the sub-sentences itself does not strongly affect the model utility.
$^{2}$ https://github.com/explosion/spaCy
When we get the split results from the last step, we can start data annotation. We give precise annotations based on the sub-sentence content, such as the "extra" column annotation discussed in the last subsection. Besides, if the description of the schema column is missing in the sub-sentence, we will give the schema column an additional "NO MENTIONED" mark. For example, in the second example of Figure 4, the "in ascending order" subsentence does not mention the "Farm.Total_Horses" column. Therefore, we add a "NO MENTIONED" mark for it. For those sub-sentences that do not mention anything related to the query, we give a "NONE" mark, representing there are no NATSQL clauses.
Since the annotation is carried out according to the sub-sentence content, the equivalent SQL that is more consistent with the sub-sentence will be preferred to the original SQL. Similarly, if the original SQL annotation is wrong, we correct it according to the content.
We annotate the sub-sentence using NatSQL instead of SQL, where NatSQL is an intermediate representation of SQL, only keeping the SELECT, WHERE, and ORDER BY clauses from SQL. Since some sub-sentences need to be annotated with GROUP BY clause, we choose the version of NatSQL augmented with GROUP BY. We did not use SQL directly because it is difficult to annotate in some cases, such as the SQL example in Figure 5. The difficulty is that there are two SELECT clauses in this SQL query, but none of the sub-sentences seem to correspond to two SELECT clauses. In addition, considering that the two WHERE conditions correspond to different SELECT clauses, the annotation work based on SQL is far more difficult to complete. As shown in Figure 5, we can use NatSQL to complete the annotation quickly, while the NatSQL can be converted back to the target SQL. The detail of the annotation steps can be found in Appendix C.
3 Spider-CG
Spider-CG is a synthetic dataset, which is generated by recombining the sub-sentences of SpiderSS. There are two recombination methods. The first is sub-sentence substitution between different examples, and the other is to append a sub-sentence into another sentence. To facilitate the follow-up
We can think about how to correctly annotate the INTERSECT clause if using the SQL query
Figure 5: It is difficult to annotate if using the SQL instead of NatSQL.
discussion, we named the Spider-CG subset generated by the sub-sentence substitution method CGSUB, and the other named CG-APP.
In CG-SUB, there are 20,686 examples generated from the Spider-SS training set, while 2,883 examples are generated from the development set. In CG-APP, examples generated from training and development sets are 18,793 and 3,237, respectively. Therefore, the Spider-CG contains 45,599 examples, around six times the Spider dataset. We can further append sub-sentences to the CG-SUB examples if more data is needed.
3.2 Generation Algorithm
According to Algorithm 1, we can generate the CG-SUB and CG-APP based on compositional elements. Each element contains one or more sub-sentences with corresponding NatSQL clauses from Spider-SS, where these NatSQL can only be WHERE or ORDER BY clauses. Thus, Algorithm 1 only substitute and append the WHERE and ORDER BY clauses, and does not modify the SELECT clause. We collect the sub-sentences for compositional elements by scanning all sub-sentence from start to end or from end to start and stopping when encountering clauses except WHERE and ORDER BY. For example, we generate a compositional element containing the last two sub-sentences of the Spider-SS example in Figure 5. In contrast, no element is extracted from the example in Figure 1. It should be noted that elements in a do
Algorithm 1 Generate CG-SUB and CG-APP dataset in a certain domain
Input: e_list	▷ All compositional elements in a domain
Output: cg_sub and cg_app	▷ CG-SUB and CG-APP dataset in a certain domain
1: for Every element1 in e_list do
2: for Every element2 in e_list do
3: if element1 != element2 then
4: if element1.can_be_substituted_by( element2 ) then
5: cg_sub.append( element1.create_substitution_example( element2 ))
6: if element1.can.append( element2 ) then
7: cg_app.append( element1.create_appending_example( element2 ))
8: return cg_sub, cg_app
Ques	Show the name of employees
named Mark Young?
SQL	SELECT name FROM employee
WHERE name = 'Mark Young'
Table 1: One acceptable but not perfect examples in the Spider-CG.
main cannot be used in another because the schema items are different. So as many domains as there are, it needs to run Algorithm 1 as many times. We recommend reading Appendix A for details of can_be_substituted_by and can.append functions.
3.3 Quality Evaluation
We consider that the quality of a text-to-SQL sentence is determined by two criteria: containing the required information and being reasonable. The 'information' criterion requires a sentence that contains all the information needed to derive the target NatSQL. The 'reasonable' criterion requires a sentence that is logically correct and whose representation is fluent and easy to understand. We randomly sampled 2000 examples from the SpiderCG dataset, around  $99\%$  of which are acceptable, i.e., they meet the two criteria. The evaluation is conducted manually by a computer science graduate with good knowledge of text-to-SQL. However, these acceptable examples do not mean that there are no grammatical errors and they may be meaningless. We give one acceptable but not perfect examples in Table 1, where the sentence is meaningless because the content it wants to query is the condition it gave. Besides, there are around  $5\%$  NatSQL queries in these acceptable examples that can not be converted to the correct SQL. This problem can be solved by a well-designed database schema or updating the NatSQL conversion function in the
Figure 6: A example of encoding the whole sentence but decoding only the sub-sentence.
Existing text-to-SQL models input a sentence and output the corresponding SQL query. So the easiest way to think of using the Spider-SS dataset is to train the model where inputting sub-sentence and outputting the corresponding NatSQL clauses. However, this method is not workable because it will lose some essential schema information. For example, if you only look at the third sub-sentence in Figure 1, you do not know whether it enquires about the weight of pets or people.
In order to take into account the context and the sub-sentence data of Spider-SS, we propose that a seq2seq model can encode the whole sentence but decode only the sub-sentence. Figure 6 presents the workflow of encoding the whole sentence but only decoding the sub-sentence of 'who is older than ten' and outputting the corresponding NatSQL clause. Based on this modification, a seq2seq text-to-SQL
Figure 7: A Spider-SS example is split into two examples for training and evaluation.
model can be adapted to the Spider-SS. Although previous span-based semantic parsers (Yin et al., 2021; Herzig and Berant, 2021) can work with aligned annotations based on the Spider-SS dataset, none of them are designed for complex text-to-SQL problems. Our modification idea is similar in principle to the span-based semantic parsers, but we did not change the existing model according to the span-based because our modification idea has a smaller workload.
In general, we can make the seq2seq-based text-to-SQL models adapt to the Spider-SS in three steps. (1) Data preprocess. Split the Spider-SS examples by sub-sentence. For example, the example in Figure 6 is split to two examples shown in Figure 7. (2) Model modification. After data preprocessing, there are two input data for a model. The first input is an entire question that directly goes to the encoder. The second input is the sub-sentence indexes, which are used to select the encoder output, as shown in Figure 6. (3) Output combination. Since the model output may be only a clause, not a complete NatSQL query, we generate the final NatSQL query after the model outputting all NatSQL clauses.
Dataset. We evaluate the previous state-of-the-art models on the Spider-CG and Spider (Yu et al., 2018b) datasets. Since the Spider test set is not
publicly accessible, Spider-CG does not contain a test set. As discussed in Section 3.1, we divide the Spider-CG into two subsets: CG-SUB and CG-APP. Therefore, there are five evaluation sets:
- SpiderD: the original Spider development set with 1,034 examples for cross-domain in-distribution text-to-SQL evaluation.
- CG-SUBT: the CG-SUB training set, containing 20,686 examples generated from Spider-SS training set by substituting sub-sentences. CG-SUBT can be used for in-domain in-distribution text-to-SQL evaluation.
- CG-SUBD: the CG-SUB development set containing 2,883 examples for cross-domain in-distribution text-to-SQL evaluation.
- CG-APP $_T$ : the CG-APP training set, containing 18,793 examples generated from Spider-SS training set by appending sub-sentences. CG-APP $_T$  can be used for in-domain out-of-distribution  $^3$  text-to-SQL evaluation.
- CG-APP $_D$ : the CG-APP development set containing 3,237 examples for cross-domain out-of-distribution text-to-SQL evaluation.
Our evaluation is based on the exact match metric defined in the original Spider benchmark. The exact match metric measures whether the syntax tree of the predicted query without condition values is the same as that of the gold query. All models are only trained on 7000 Spider or Spider-SS training examples.
Models. We evaluate the following open-source models that reach competitive performance on Spider:
- GNN: The GNN (Bogin et al., 2019) model using the GLOVE (Pennington et al., 2014) embeddings.
- RATSQL: The RATSQL (Wang et al., 2020) model using the GLOVE embeddings.
- RATSQLB: The RATSOL model using the BERT (Devlin et al., 2019) embeddings.
- RATSQLG: The RATSOL model using the GAP (Shi et al., 2021) embeddings.
-  $(\mathbf{N})$ : This subscript indicates that the model use NatSQL instead of SQL.
- (s): This subscript indicates that the model is modified according to Section 4 and trained on Spider-SS. Besides, since Spider-SS is annotated
3Out-of-distribution means that the difficulty distribution is different from the Spider; see Table 3. Appendix A discusses the removal of overly complex examples to ensure that Spider-CG's SQL does not exceed the complexity upper bound of the Spider.
Dataset	Exact Match	Execution Match
Training Set	90.7%	93.3%
Development Set	94.8%	95.2%
Table 2: Use exact match and execution match metrics to evaluate the difference between the SQL in Spider and the SQL generated by NatSQL in Spider-SS.
Dataset	easy	medium	hard	extra
SpiderD	24.1%	43.1%	16.8%	16.1%
CG-SUBT	28.6%	38.0%	21.1%	12.3%
CG-SUBD	37.6%	38.4%	12.0%	12.0%
CG-APPT	3.3%	31.4%	26.0%	39.3%
CG-APPD	2.4%	44.3%	22.9%	30.4%
Table 3: The difficulty distribution of five different evaluation sets.
by NatSQL, this subscript also indicates that the model uses NatSQL instead of SQL.
Implementations. All experiments were performed on a machine with an Intel i5 9600 3.1GHz processor and a 24GB RTX3090 GPU. All models keep their original hyperparameters except the RATSQL $_{\mathrm{B(S)}}$ . RATSQL $_{\mathrm{B(S)}}$  cannot converge on the original parameters until we reduce the learning rate of model from 7.444e-04 to 1e-04 and raise the learning rate of BERT from 3e-06 to 1e-05. We did not conduct a hyperparameter search, so the model trained on Spider-SS may improve performance through other parameters.
5.2 Dataset Analysis
Spider-SS. Table 2 presents the difference between the SQL in Spider and the SQL generated by NatSQL in Spider-SS. Our evaluation results are lower than the original NatSQL dataset (Gan et al., 2021b) because the Spider-SS uses equivalent SQL and corrects some errors, as discussed in Section 2.3. Some equivalent and corrected SQL cannot get positive results in exact match metric and execution match. Therefore, the model trained on Spider-SS may not be ideal for chasing the Spider benchmark, especially based on the exact match metric. Similarly, the RATSQL $_G$  extending NatSQL had achieved a previous SOTA result in the execution match of the Spider test set but get a worse result than the original in the exact match (Gan et al., 2021b). Thus, we recommend using NatSQL-based datasets to evaluate models trained on NatSQL.
Spider-CG. Table 3 presents the difficulty distribution of five different evaluation sets. The difficulty criteria are defined by Spider benchmark, including easy, medium, hard and extra hard. Experiments show that the more difficult the SQL is, the more difficult it is to predict correctly (Wang et al., 2020; Shi et al., 2021; Gan et al., 2021b). It can be found from Table 3 that the difficulty distribution of CG-SUBT and CG-SUBD is similar to that of  $\mathsf{S}\mathsf{p}\mathsf{d}\mathsf{e}\mathsf{r}_{\mathsf{D}}$  . The similar distributions among CG-SUBT, CG-SUBD, and  $\mathsf{S}\mathsf{p}\mathsf{d}\mathsf{e}\mathsf{r}_{\mathsf{D}}$  support the view discussed in Section 1 that the examples generated by the substitution method are in-distribution.
On the other hand, the difficulty distributions of CG-APP $_{\mathrm{T}}$  and CG-APP $_{\mathrm{D}}$  are obviously different from that of  $\mathrm{Spider}_{\mathrm{D}}$ . Due to appending the subsentence, the NL and SQL in CG-APP become more complex, where the proportion of SQL in extra hard increased significantly, while easy was the opposite.
5.3 Sentence Split Algorithm Evaluation