Re-appraising the Schema Linking for Text-to-SQL
Year: 2023

Abstract:
Most text-to-SQL models, even though based on the same grammar decoder 1, generate the SQL structure first and then fill in the SQL slots with the correct schema items. This second step depends on schema linking: aligning the entity references in the question with the schema columns or tables. This is generally approached via Exact Match based Schema Linking (EMSL) within a neural networkbased schema linking module. EMSL has become standard in text-to-SQL: many stateof-the-art models employ EMSL, with performance dropping significantly when the EMSL component is removed. In this work, however, we show that EMSL reduces robustness, rendering models vulnerable to synonym substitution and typos. Instead of relying on EMSL to make up for deficiencies in question-schema encoding, we show that using a pre-trained language model as an encoder can improve performance without using EMSL, giving a more robust model. We also study the design choice of the schema linking module, finding that a suitable design benefits performance and interpretability. Finally, based on the above study of schema linking, we introduce the grammar linking to help model align grammar references in the question with the SQL keywords.2

heading: 1 Introduction
text: Recent years have seen great progress on the textto-SQL problem, i.e. translating a natural language (NL) question into a SQL query (Dong and Lapata, 2018; Yu et al., 2018b; Zhong et al., 2017; Gan et al., 2021a; Guo et al., 2019; Bogin et al., 2019; Wang et al., 2020), with neural networks the de facto approach. To achieve good performance on text-to-SQL tasks, a neural model needs to correlate natural language queries with the given
1The decoder will repeat the following two steps until a complete SQL is generated: (1) generates a SQL clause keyword; (2) fills in corresponding schema items.
2Our code and data are available at Github.
database schema, a process called schema linking. Previous work often explicitly designs a module to perform the schema linking, which we term Exact Match based Schema Linking (EMSL) (Guo et al., 2019; Bogin et al., 2019; Wang et al., 2020). Specifically: • Schema linking is the alignment between the
entity references in the question and the schema columns or tables. • A schema linking module is a trainable component that learns to perform schema linking, based on features that relate word tokens in the question to schema items. • A schema linking feature encodes this relational information; e.g., it can represent the similarity between words in the question and schema items. • Exact match based schema linking (EMSL) is a type of schema linking feature obtained by the exact lexical match between the words in the question and words in schema items. Figure 1 presents an example of schema linking and the EMSL feature matrix. Most previous work relies on this exact lexical matching to obtain schema linking features. Following the work of (Krishnamurthy et al., 2017; Guo et al., 2019; Bogin et al., 2019), EMSL is used in many subsequent works (Wang et al., 2020; Cai et al., 2021; Xu et al., 2021; Lei et al., 2020; Yu et al., 2021; Shi et al., 2021) and has been shown to be effective. For example, the ablation study in Guo et al. (2019) shows that removing the schema linking module incurs the most significant performance decrease.
Although EMSL has been widely used and helps models obtain the state-of-the-art performance on some text-to-SQL benchmarks (Yu et al., 2018b; Zhong et al., 2017), in this work, we show that EMSL renders models vulnerable to noise in the input, particularly synonym substitution and typos. We then investigate whether text-to-SQL models can preserve good prediction performance without EMSL. Previous ablation studies (Guo et al., 2019;
835
Wang et al., 2020) claiming the necessity of the schema linking module were conducted without pretrained language models (PLMs) such as BERT. In fact, we find that when a pretrained language model is used as a model encoder, removing EMSL has very little impact on the performance of the model. This observation is consistent for different model architectures and training schemes, such as RATSQL (Wang et al., 2020), GNN (Bogin et al., 2019), and GAP (Shi et al., 2021). Based on this finding, we introduce a more reasonable text-toSQL encoder design.
We evaluate the models in three settings: the original Spider benchmark without input noise (Yu et al., 2018b), the Spider-Syn version with synonym substitution (Gan et al., 2021a), and a new typo injection setting. Results show that the use of a PLM can provide the same performance benefit as EMSL, while achieving better robustness against synonym substitution and typos. Removing EMSL also allows the model to obtain better results when training with synonym substitution samples. We also show that MAS (Multi-Annotation Selection, Gan et al., 2021a), a method designed to improve model robustness with EMSL, can also improve models without EMSL. In conclusion, we demonstrate EMSL is no longer a necessary building block of text-to-SQL models.
The EMSL and pretrained language models are part of the schema linking module that learns a score to decide which schema item to select. There are two design choices to compute this score: the first sees it as a direct relation between the question and schema items (Bogin et al., 2019); while the second considers the question and schema items together (Wang et al., 2020). We show that our best model employing the first design choice achieves the state-of-the-art performance, while its schema linking scores are also more interpretable than models with the second design.
Despite the shortcomings of EMSL, it still significantly improves previous models: without a clear idea of how to align the entity references in the question and the schema columns or tables, EMSL plays an important role in alignment. Inspired by EMSL, we propose instead Exact Match Grammar Linking (GL), which improves the model’s ability to generate correct SQL keywords.
In short, this work investigates the role of the schema linking module for text-to-SQL models, regarding its effect on model performance and ro-
bustness. We summarize our key findings below: • EMSL is not a necessary building block of text-
to-SQL models. A reasonable text-to-SQL encoder design can replace EMSL and make the model more robust. • Compared to the cross relation score between the question and schema items together, a direct relation score between the question and schema items is more interpretable and can improve the performance of a state-of-the-art model. • When it is unclear how to design the optimal encoder, EMSL improves the model performance significantly. Inspired by EMSL, we introduce Exact Match Grammar Linking (GL) to generate better SQL keywords.

heading: 2 Proposal I: Construct the Schema Linking without the EMSL
text: Our first proposal is that the schema linking should not include EMSL. Schema linking itself is essential for text-to-SQL models, but EMSL can be replaced with a better mechanism.

heading: 2.1 Schema Linking Feature
text: Figure 1 presents an example of schema linking features. The word ‘singers’ in the question exactly matches (modulo stemming) the schema table name ‘singer’, giving feature value 1. It does not match the table ‘concert’, giving value 0; and matches one of the three words in ‘singer in concert’, giving value 0.33. Such type of schema linking mechanism based on exact lexical matching (EMSL) is the most common used in existing text-to-SQL models (Guo et al., 2019; Bogin et al., 2019; Wang et al., 2020; Cai et al., 2021; Xu et al., 2021; Lei et al., 2020; Yu et al., 2021; Shi et al.,
2021). Schema linking has been shown to be essential for achieving good performance (Guo et al., 2019; Wang et al., 2020). For example, Wang et al. (2020) consider that the representations produced by vanilla self-attention were insensitive to textual matches even though their initial representations were identical, i.e., EMSL is needed for textual matches. Some works add ConceptNet (Speer and Havasi, 2012) to get more linking features (Guo et al., 2019; Tan et al., 2021) and thus improve the model performance. However, we argue that a well-designed encoder can solve this problem, and note that the feature values in Figure 1 are equal to the average dot product results when using lemma one-hot embeddings, suggesting that a proper embedding can replace EMSL. Appendix E provides previous ablation study on EMSL. We test our new proposal experimentally in Section 5.3.

heading: 2.2 Text-to-SQL Encoder Design
text: For a text-to-SQL encoder, we expect that the correct schema item vectors obtained from the encoder are as close to the question vector as possible. The SQL cares about which schema item to use instead of the words in the schema item. Therefore, unlike keeping every question word vector, only one vector is used to present the schema item even if it contains multiple words. Since both the encoder mechanics and content style are different between question and schema, many models (Yu et al., 2018a; Guo et al., 2019; Wang et al., 2020) uses different encoders to encode the question and schema separately. The upper part of Figure 2 shows a design case, which is the structure of the RATSQL model (Wang et al., 2020) containing three encoders with similar structure and size.
We believe that the shortcoming of the original RATSQL design is the use of three encoders. For example, in the initial state, the parameters of the three encoders are different. Therefore, even though the word ‘singers’ appears in the question, the vector v6 initially generated by the table encoder is probably irrelevant to all vectors output by the sentence encoder. If using EMSL, this does not matter: in both training and evaluation we can link v6 to v2 through EMSL. However, without EMSL, we need to relate v6 from the table encoder to the vectors from the question encoder, which is more challenging to train than using only one encoder, as shown in the lower part of Figure 2. Since the
output of our modification is the same as the original, it can be easily replaced and connected to the subsequent modules.
In the lower part of Figure 2, our modification is inspired by several text-to-SQL models with BERT, including RATSQL+BERT (Wang et al., 2020; Guo et al., 2019; Zhang et al., 2019). In our modification, RATSQL uses only one encoder instead of three. We believe using three encoders is one of the main reasons why the base RATSQL performance significantly drops when removing EMSL. For the convenience of discussion, we named our modified RATSQL as RATSQLO, where O means one encoder.
RATSQLO uses only one encoder whose structure and size are the same as the original question encoder. For the schema item representation, RATSQL takes the hidden state after all the words of the entire schema item are encoded, while RATSQLO takes the average of all word encodings. The advantage of our RATSQLO is that v6, v8, and v2 initially have a certain similarity, which benefits the schema linking in both single and multi words. RATSQLO also deals with words outside the embedding vocabulary better than RATSQL. Suppose the word concert and name are outside the vocabulary: v7 and v9 from the RATSQL table encoder will be the same since their inputs are the same UNK vector. However, the RATSQLO encoder will output different vectors for v7 and v9, as the contexts before and after the word concert and name are different. In this way, even if there are multiple UNK words, the RATSQLO encoding vector will be different.

heading: 3 Proposal II: Schema Linking Module Design Choice
text: We believe that a text-to-SQL model with good performance can ignore the schema linking feature, but it must include a schema linking module. While implementation details of such models differ,
the common factor is the calculation of a similarity score between each question word and schema item: correct schema items should obtain higher similarity scores.
Schema linking modules output attention scores from computing the schema linking feature and word embeddings. There are currently two attention mechanism designs. The first calculates a score that relates the question on one side, to the schema items on the other. The second approach also considers the attention scores between different schema items (Guo et al., 2019), thus it takes the question and schema items together as input to produce the attention scores. In Figure 3, we elucidate the inputs and outputs associated with the two types of attention computations.
Both designs have their own rationale, but the
first design is more interpretable: it requires the model to infer the correct schema item from the question, so you can see which part of the question is related to each selected schema item. With the second design, it is sometimes difficult to explain why the specific schema item is chosen due to the presence of some other schema items. For ease of understanding, we show and discuss the attention and interpretability of an example under different design choices in Appendix B; we test the effects on performance in Section 5.4.

heading: 4 Proposal III: Exact Match Grammar Linking (GL)
text: The mechanisms of existing models limit their performance in some specific situations discussed in Section 4.2 and Appendix C. Our third proposal is GL which can provide ideas for addressing these limitations.

heading: 4.1 Overview
text: We propose GL, inspired by the EMSL and the NL question label generation method (Ma et al.,
2020). EMSL attempts to match the words in the question with the schema item words, with the matching result then used as a feature to help the model generate the correct schema items. Similarly, GL matches the words in the question with those related to the SQL keywords. Considering that the number of SQL keywords is limited, GL can be implemented by matching words to specific categories/labels.
In general, GL has two main steps in the text-toSQL process: (1) identify the label of each word, as shown in Figure 4; (2) input both label and word embeddings into the encoder. Formally, given a natural language question Q = q1..qQ, we generate a label li to each qi, as shown in Figure 4. There are different embeddings for li and qi, where the li embeddings Embl is randomly initialized while the qi embeddings Embq can obtain from GLOVE (Pennington et al., 2014) or PLMs. The encoder input X is the sum of embeddings of qi and li:
X = (Embl(l0)+Embq(q0), ..., Embl(lQ)+Embq(qQ))
Specifically, Embl(li) does not enforce binding to specific SQL keywords. We expect the model to learn the meaning of GL label embeddings.
Before we can use GL for text-to-SQL, we need to define the labels and match rules. We use the method of information extraction combined with manual annotation to collect the words related to SQL keywords. We remove the words that match the schema items and then use the tf-idf (term frequency-inverse document frequency) score to extract the words that are highly relevant to the target SQL keyword. After manual correction and adding words of the same type, such as synonyms, we obtain the word categories/labels corresponding to the SQL keywords. Table 1 presents all labels with their matching rules used by this work.
In Table 1, uppercase labels match a class of words, while lowercase labels only match words with the same name as the label, such as ‘at’ and ‘each’ labels. The order of the labels in Table 1 is sorted by their priority. We start matching from the highest priority label and end once the match is successful. Taking Figure 4 as an example, the label for the word ‘average’ in example 1 is ‘AGG’, but that in example 2 is ‘COL’. Because the two examples belong to different domains where the database of the second example contains a column named ‘average’ that can be built an EMSL with the question word ‘average’. Therefore, although the
‘average’ in example 2 satisfies both the matching rules of the ‘AGG’ and ‘COL’ labels, its generated label is ‘COL’ instead of ‘AGG’ since the priority in the ‘COL’ label is higher.

heading: 4.2 Benefits from GL
text: Existing models (Bogin et al., 2019; Guo et al., 2019; Wang et al., 2020; Cao et al., 2021) tend to output the incorrect SQL clause ‘ORDER BY avg(average)’ instead of ‘ORDER BY average’ for the second example in Figure 4, even though their generation process is based on the same decoder. This error states that the ‘average’ word is used twice in generating the ‘average’ column and the ‘avg’ function. Utilizing GL can help existing models solve this problem. When training with GL, the model will learn that most of the examples generating ‘avg’ function require the ‘AGG’ label of GL. Therefore, since there is no ‘AGG’ label in example 2, the model will not tend to generate the ‘avg’ function. For other benefits from GL, please refer to Appendix C.

heading: 5 Experiments

heading: 5.1 Generating Typos
text: To evaluate robustness against typos, we randomly insert a letter into the correct schema annotation word. (This is enough to break EMSL, so we do not also modify the question words). We generated three typo development sets, named Spider-T1 to Spider-T3. The typos in Spider-T1 are generated by randomly inserting a letter at any position except the end. In contrast, Spider-T2 appends a random letter at the end of the schema annotation words. We examine these separately: the BERT tokenizer may be able to split Spider-T2 typos into a correct word and a suffix, but is less likely to split the Spider-T1 typos well. We convert every schema annotation word in Spider-T1 and T2 to typos when word length is greater than five letters; typos are generally more likely to occur in longer words, and words with more than five letters account for about 40% of the dataset. Spider-T3 is then the same as Spider-T1, but only converts the most frequent schema item words to typos. While Spider-T1 and T2 simulate the impact of large numbers of typos in extreme cases, Spider-T3 evaluates the impact of a more realistic, smaller number of typos. Other typos are possible, e.g. by deleting and swapping letters; we discuss these in Appendix D.

heading: 5.2 Experimental Setup
text: We evaluate the previous state-of-the-art models on Spider (Yu et al., 2018b), Spider-T, and Spider-Syn (Gan et al., 2021a) datasets. All results report their maximum value. All experiments were performed on a machine with an Intel i5 9600 3.1GHz processor and a 24GB RTX3090 GPU. Since the Spider test set is not publicly accessible and Spider-Syn and Spider-T do not contain test sets, our evaluation is based on the development sets. The SpiderSyn benchmark contains three development sets: Spider-Syn, ADVBERT, and ADVGLOVE, for evaluating model robustness against synonym substitution. Therefore, we have the following evaluation sets: • Spider: The original Spider development set
with 1,034 examples. • Spider-T1, T2 and T3: Three development sets
that replace the correct word with typos, introduced in Section 5.1. • Spider-Syn: The human-curated development set built upon Spider, for evaluating synonym substitution in real-world question paraphrases. • ADVBERT: The set of adversarial examples generated by BERT-Attack (Li et al., 2020). • ADVGLOVE: The set of adversarial examples generated using the nearest GLOVE word vector (Pennington et al., 2014; Mrkšić et al., 2016). Our evaluation is based on the exact match metric defined in the original Spider benchmark. This metric measures whether the syntax tree of the predicted query without condition values is the same as that of the gold query. Our experiment setting is consistent with the ablation study in Appendix E. Following the encoder design in Section 2.2, we evaluate different variants of the RATSQL model: • RATSQL: The base RATSQL+GLOVE model
trained on Spider using EMSL in training and evaluation (Wang et al., 2020). • RATSQLO: Our modified RATSQL+GLOVE model trained on Spider using EMSL in training and evaluation, discussed in Section 2.2. • RATSQLB: The RATSQL+BERT model trained on Spider using EMSL in training and evaluation. (Note that RATSQLO+BERT is just RATSQL+BERT: using BERT means that the BERT encoder will replace all encoders in Figure 2). • RATSQLBS: RATSQL+BERT trained on Spider-Syn using EMSL (Gan et al., 2021a). • RATSQLG: RATSQL+GAP trained on Spider using EMSL (Shi et al., 2021).
• w/o EMSL: Models without EMSL in training and evaluation, consistent with Tables 9 and 10. • ManualMAS (Gan et al., 2021a): Schema annotations include synonyms used in Spider-Syn. • AutoMAS (Gan et al., 2021a): Schema annotations include synonyms generated according to the nearest GLOVE word vector.

heading: 5.3 Experiment on EMSL (Proposal I)

heading: 5.3.1 Evaluation on Spider
text: Table 2 presents the exact matching accuracy of models trained on the Spider training set. Without EMSL, our RATSQLO model significantly improves over RATSQL. Models with PLMs also obtained similar results with RATSQLO, which also supported our proposal to remove the EMSL. Detailed experimental results and discussion are shown in Appendix E. Furthermore, we conduct an error analysis in Appendix F.

heading: 5.3.2 Robustness Evaluation
text: Typo Results Table 3 presents the robustness evaluation results on several datasets. For typos, GLOVE will treat them as UNK words, so the RATSQL and RATSQLO cannot obtain good performance on Spider-T1 and T2 due to too many UNK words. The RATSQLO without EMSL significantly outperforms the RATSQL without EMSL in Spider-T3, which is another evidence that the RATSQLO is better in handling UNK words. After using PLMs, the performance on typos has been significantly improved, especially on Spider-T2. Spider-T3 contains only a few typos, i.e., it is close to the Spider to some extent. Thus, the T3 result characteristics are close to Spider, i.e., their performance gap between with and without EMSL is close. With the increase of typos, the performance gap will be expanded, where the model+PLM without EMSL will be better.
Synonym Substitution Results Gan et al. (2021a) propose three development sets for evaluating the robustness of text-to-SQL models
against synonym substitution, including: SpiderSyn, ADVBERT, and ADVGLOVE. Table 3 shows that models without EMSL consistently outperform those with EMSL when evaluated against Spider-Syn, ADVGLOVE and ADVBERT. When using PLMs, RATSQLB and RATSQLG without EMSL show a huge performance improvement on these three development sets with only a tiny performance loss on Spider. RATSQLO without EMSL consistently outperforms RATSQL without EMSL, which means a reasonable design can reduce reliance on EMSL. Unlike other models, the RATSQLBS without EMSL outperforms that with EMSL in all evaluation sets. We discuss this in Appendix A. Although the S2SQL (Hui et al., 2022) model achieves pretty good performance in the Spider, its EMSL module causes its performance on Spider-Syn to be much worse than that of the RATSQLG without EMSL.
MAS Results Gan et al. (2021a) also propose a MAS method to improve the robustness of text-toSQL models. MAS provides multiple annotations to repair the breaking of EMSL due to synonym substitutions. Although we advocate not relying on EMSL, MAS can still improve the performance of models without EMSL, as shown in Table 4. Comparing the data in Table 3 and Table 4, ManualMAS improves the performance of RATSQLB and RATSQLBS with and without EMSL on Spider-
Syn development set since the ManualMAS provide synonym annotations appearing in the SpiderSyn. In the same way, AutoMAS has also improved their performance on ADVGLOVE. Experimental results show that although MAS is designed to repair EMSL, it is still effective for models without EMSL. Besides, based on MAS, the overall performance of the model without EMSL is still better than that with EMSL. In general, even though EMSL is not used, a reasonable annotation is still essential to the text-to-SQL problem.

heading: 5.3.3 Discussion
text: We introduce RATSQLO, an enhanced text-to-SQL encoder design, challenging the need for Exact Match Schema Linking (EMSL) assumed by previous research. RATSQLO offers an alternative perspective, arguing that the benefit of consolidating the encoder actually promotes schema linking. As a result, relying solely on PLM can only address certain issues. For instance, PLMs have input limitations that may suffice for current text-to-SQL benchmarks with small-scale schemas. However, for large-scale schemas, modifications to the PLM
encoding method are necessary. If the PLM were to encode the question and schema separately, the EMSL would still be required.

heading: 5.4 Experiment on Schema Linking Module Design Choices (Proposal II)
text: As discussed in Section 3, there are two design choices for the schema linking module. The first calculates a score that relates the question on one side, to the schema items on the other. The second approach assumes the attention between one schema item and others is needed, and it therefore takes question and schema items together as input to produce the score. The original RATSQL chose the 2nd design, named it RATSQL2G here. We modify the RATSQL according to the 1st design, named it RATSQL1G, and observe that its performance drops slightly, as shown in Table 5. Error analysis shows that RATSQL1G tends to use the schema items mentioned in the question and is not so good at dealing with implicit schema items.
Although the performance of the RATSQL1G is slightly worse, we found that its schema linking performance is not inferior. The accuracy of the SELECT clause is the best way to measure the schema linking performance because every SQL contains at least one SELECT clause that only contains schema items. The SELECT accuracy of the RATSQL1G is slightly (0.4%) better than the RATSQL2G, which inspired us that the RATSQL1G is likely to perform well if removing the implicit schema items. Fortunately, we found NatSQL, an SQL intermediate representation that removes many implicit schema items from the SQL (Gan et al., 2021b). Experiments show that the performance of RATSQL1G+NatSQL is bet-
ter than the RATSQL2G+NatSQL. Table 5 gives a detailed performance comparison, from which it can be found that by replacing the design, the RATSQLG+NatSQL is improved to the second place, evaluated on the development set, which is close to the current best published model. It should be noted that RATSQLG+NatSQL does not use the complex graph neural network as S2SQL and LGESQL, nor does it use the ELECTRA, which is shown to be better than GAP (Clark et al., 2020; Cao et al., 2021; Hui et al., 2022).

heading: 5.5 Experiment on GL (Proposal III)
text: We assemble GL onto RATSQLG+NatSQL, obtaining 0.9% absolute improvement, rising from 75.5% to 76.4%, which improves the performance of the RATSQL model to be consistent with the best-published model, as shown in Table 5. Although the performance improvement of GL is less apparent than that of the EMSL in previous ablation studies, GL is an indispensable module for solving specific problems. We compared the RATSQLG+NatSQL+GL with S²SQL and LGESQL, where S²SQL and LGESQL are the current top 2 opensource models in the exact match metric of the Spider leaderboard. Table 6 presents the output of these models for two examples in the Spider development set. No models without GL, including RATSQLG+NatSQL, generate the correct SQL. We discuss how GL can help the model generate correct SQL in Section 4.2 and Appendix C, respectively.
During our experiment, we faced a number of challenges with using the GL. Unlike schema linking, which creates a direct link from question tokens to a schema item, GL doesn’t establish a sim-
ilar connection. In GL, it just identifies question tokens related to SQL keywords based on rules defined in Table 1. GL requires the neural model to find out which SQL keywords it’s connected to by training. However, we found this difficult to accomplish when in scenarios involving complex SQL queries, the model struggled to accurately connect GL with the appropriate SQL keywords. To address this, we opted for using NatSQL from Spider-SS (Gan et al., 2022) in our (RATSQL1G with NatSQL and GL) model, rather than the original version. The benefit of Spider-SS is that it breaks down SQL/NatSQL according to question clauses. This means the model rarely comes across complex SQL/NatSQL during training, simplifying the task of associating GL with the proper SQL keywords. To further increase the chances of success with GL, we also made slight adjustments to NatSQL, which you can find in Appendix C. We ran ablation studies to check if NatSQL modification or the use of Spider-SS data impacted the results. Experiments showed that without GL, the outcomes using Spider-SS data remained consistent with those using Spider. Similarly, the performance of the adjusted NatSQL, when used without GL, is also consistent with the original NatSQL.

heading: 6 Related Work
text: Schema Linking Schema linking has been an important design choice for existing text-to-SQL models (Guo et al., 2019; Bogin et al., 2019; Wang et al., 2020; Chen et al., 2020; Cao et al., 2021). Besides designing new models, some works focus on investigating the effect of schema linking. Lei et al. (2020) demonstrate that more accurate schema linking conclusively leads to better text-to-SQL parsing performance. To support further schema linking studies, Lei et al. (2020) and Taniguchi et al. (2021) invest human resources to annotate schema linking corpus, respectively. Guo et al. (2019) and Wang et al. (2020) conducted an ablation study on EMSL, respectively, and the results show that removing the EMSL would lead to the greatest decrease in model performance. These studies have influenced many follow-up works to use EMSL (Cai et al., 2021; Xu et al., 2021; Lei et al., 2020; Yu et al., 2021; Shi et al., 2021). Moreover, in the ablation study of BRIDGE (Lin et al., 2020), its performance declines notably when the PLM is removed, as it does not utilize EMSL. Our work found that existing text-to-SQL models with EMSL tend to overly rely
on this schema linking module, which degrade their robustness. Meanwhile, more advanced pretrained language models can replace EMSL and thus improve the model robustness, without notably degrading the performance.
Robustness of Text-to-SQL Existing works on improving the robustness of the text-to-SQL model are mainly through adversarial training, data augmentation, and repairing EMSL. Xiong and Sun (2019) and Radhakrishnan et al. (2020) propose data augmentation techniques for improving the generalization in cross-domain text-to-SQL and in search-style questions resepctivly. However, these approaches only support SQL queries executed on a single table, e.g., WikiSQL. Zeng et al. (2020) introduce a SpiderUTran dataset that includes original Spider (Yu et al., 2018b) examples and some untranslatable questions examples. SpiderUTran can be used to evaluate whether the text-to-SQL model can distinguish the untranslatable NL question. Gan et al. (2021a) investigate the robustness against synonym substitution for cross-domain textto-SQL translation and found that synonym substitution would break EMSL, giving a significant drop in performance; to solve this problem, they proposed the MAS method to repair the broken EMSL. Following (Gan et al., 2021a), our work found that the EMSL can be replaced by better encoding, and models without EMSL has better generalization ability.

heading: 7 Conclusion
text: In this work, we investigate the role of schema linking for text-to-SQL models regarding model performance and robustness. In particular, we demonstrate that by leveraging pretrained language models, EMSL is no longer a necessary building block to ensure a high performance on text-to-SQL benchmarks. We observe that when EMSL is used, models become overly reliant on it, making them vulnerable to attacks that break the exact-match assumptions of EMSL. Beyond this, by studying different schema linking module designs we find that a direct relation between the question and schema items is more interpretable and works well with intermediate representation SQL. Finally, inspired by EMSL, we introduce Exact Match Grammar Linking for dealing with some examples where existing models can easily make mistakes.

heading: 8 Limitation
text: We notice that there are some works based on pretrained large language models has obtained good performance (Scholak et al., 2021; Li et al., 2023). Due to limited computing resources, our evaluation mainly focuses on model architectures specially designed for text-to-SQL problems, and we did not conduct experiments on recent pretrained large language models, such as T5 (Raffel et al., 2020) and GPT3 (Brown et al., 2020), due to limited computing resources. However, note that most models with top text-to-SQL performance still employ specialized architecture design, e.g., with EMSL. We consider extending our study to recent pretrained large language models as future work.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful comments. Matthew Purver acknowledges financial support from the UK EPSRC under grant EP/S033564/1, and from the Slovenian Research Agency for research core funding (No. P2-0103 and No. P5-0161). Yujian Gan is supported by the ARCIDUCA project, funded by EPSRC (EP/W001632/1).

heading: A Further Discussion on EMSL
text: The text-to-SQL model can quickly locate the correct schema items through EMSL, but this advantage will cause the models to not work properly when EMSL fails. To better understand the impact of EMSL on text-to-SQL models, we present the question-table attention 3 extracted from RATSQLB with and without EMSL in Figure 5. In the first example, we can see that the alignment score between table singer and question word singer is the biggest, while we can not observe a clear connection between other tables and question word singer. However, when removing the EMSL in the second example, the alignment score between table singer and question word singer drop clearly, and the connection between other tables and question word singer becomes clear. It can be seen that under other conditions unchanged, only removing EMSL has a considerable impact on the model trained with EMSL.
The third example is extracted from RATSQLB without EMSL. Different from the RATSQLB with EMSL, the singer table has a high alignment score not only with the word singer but also with the whole sentence. Since the loss function only calculates whether the output schema items are correct, the model does not care which question word
3It is named m2t_align_mat in the code: https: //github.com/microsoft/rat-sql/blob/master/ ratsql/models/spider/spider_enc_modules.py
the correct schema item is linked to. Therefore, the attention of the RATSQLB without EMSL is quite different from that with EMSL. The significant difference of the trained models may be one of the reasons why the overall performance of RATSQLBS without EMSL is better than that with EMSL. Because the training data in RATSQLBS contain many synonym substitution examples, and these examples do not have EMSL features, it requires the model to find a balance between states shown in examples 1 and 3 of Figure 5, which increases the difficulty of training.

heading: B Attention Visualization of Different Schema linking Module Design Choices
text: Figure 6 presents the attention weight of schema tables and illustrates why the first design choice is more interpretable. The SQL for the question in Figure 6 is ‘SELECT T1.City FROM Airports AS T1 JOIN Flights AS T2 ON T1.AirportCode = T2.DestAirport GROUP BY T1.City ORDER BY count(*) DESC LIMIT 1’. So, the table ‘airports’ and ‘flights’ are needed. Although models under both design choices predict this example correctly, their attention scores are quite different. We ob-
serve that the attention under the first design can locate the proper question words. However, the attention of the table ‘flights’ can not locate any question words when using the second design choice, which is difficult to explain why the ‘flights’ table was selected instead of the ‘airlines’ table with similar attention. It should be noted that the ‘flights’ table is mentioned implicitly, but it does not prevent the first design choice from giving it the proper attention.

heading: C More Benefits from GL
text: It is difficult for existing models (Bogin et al., 2019; Guo et al., 2019; Wang et al., 2020; Cao et al., 2021) to generate the ‘WHERE isOfficial = ‘T’ ’
clause in Figure 7 since there are no similar examples in the training data. This example requires the model to generate the WHERE condition based on a single NL word ‘official’. The word ‘official’ partly exactly matches the column ‘isOfficial’ and implicitly mentions the WHERE condition value. The implicit WHERE condition value challenges existing models since most WHERE condition values are explicitly mentioned in the training data. It is difficult for models to understand this implicit expression. Hence, models tend to regard the word ‘official’ as an occasionally mentioned column, and the WHERE condition will not be generated. GL can solve this special problem by giving a new label BCOL to the word ‘official’, different from the common exact match schema linking labels: COL and TABLE. Figure 8 presents two more examples with the BCOL label and newly designed NatSQL.
We found that the type of column in these special WHERE conditions is Boolean-like. For example, the ‘isOfficial’ column only contain the ‘T’ and ‘F’ values, and some others Boolean-like column only contain the ‘0’ and ‘1’ values. Therefore, we first analyze the database data and mark the columns with Boolean-like type. When generating the GL labels, if the word matches a schema column marked as Boolean-like type, give it a ‘BCOL’ label instead of ‘COL’. However, the addition of ‘BCOL’ only cannot solve this problem, i.e., the model still does not generate the ‘isOfficial’ condition. We found that training data make models tend to give fewer WHERE conditions to simpler questions. To avoid conflicts with other simple questions, we move the WHERE condition of Boolean-like type to a BCOL clause, as shown in the new NatSQL of Figure 7 The BCOL clause in this newly designed NatSQL is finally converted to a WHERE condition. At this point, with the BCOL label and clause, our method can generate correct SQL for questions similar to Figure 7.

heading: D More Typos
text: Besides generating typos by inserting a letter, we also generate typos by deleting a letter and swapping the letter position, named the generated development set Spider-T4 and Spider-T5, respectively. Like Spider-T1 and T2, here we only convert the words whose length is greater than five letters to typos. Table 7 presents the exact match accuracy on Spider-T4 and Spider-T5 development sets. Since PLM handles typos in Spider-T4 and T5 similar to
Spider-T1, their evaluation results are also similar. Besides, we observe that the results of models using GLOVE in Spider-T4 are the best, followed by in T5, then in T2, and finally in T1. To understand this phenomenon, we found that although the number of generated typos is the same among these datasets, Spider-T1 has the most GLOVE UNK words, followed by T2, then T5, and T4 contains the least UNK words. It can be seen that in the case of fewer UNK words, the model+GLOVE can generate better encoding so that the model+GLOVE without EMSL surpasses that with EMSL in SpiderT4 and T5.

heading: E Ablation Study on EMSL
text: Table 9 presents the ablation study results of three base models. The results of RATSQL here are different from that of (Wang et al., 2020) because Wang et al. (2020) remove the cell value linking first and then EMSL. According to the magnitude of the decline, our results are similar to theirs. According to (Wang et al., 2020; Guo et al., 2019),
they observe the biggest performance degradation by removing EMSL. Since then, EMSL has become a necessary module for most researchers to build text-to-SQL models.
We want to challenge this view and carry out the comparative experiment in Table 10. Comparing Table 9 and Table 10, it can be found that PLMs compensate for the function of EMSL, i.e., the performance in Table 10 is less degraded than that in Table 9 after removing EMSL.
From another perspective, BERT and its subsequent pretrained language model significantly improve the performance of models that do not use EMSL, which explains why some models can achieve higher performance improvements through BERT. For example, EditSQL (Zhang et al., 2019) does not use EMSL, while it obtains the highest performance improvement by extending BERT, as shown on the Spider leaderboard 4.

heading: F Further Discussion on Section 5.3.1
text: F.1 BERT vs GLOVE The base RATSQL uses GLOVE (Pennington et al., 2014) for word embedding. There are two main reasons why BERT (Devlin et al., 2019) is better
4https://yale-lily.github.io/spider
than GLOVE at schema linking. The first reason is that BERT can better deal with out-of-vocabulary words. BERT converts these words into subwords, so BERT makes sure different word is represented by a unique vector. However, GLOVE cannot handle out of vocabulary words. Researchers generally replace them with a custom unknown (UNK) word vector. Suppose there are multiple words outside the GLOVE vocabulary in one schema. In that case, it is equivalent to multiple schema items being annotated as UNK, which will cause the model without EMSL to be unable to distinguish different schema items due to the same word vector.
The second reason is that GLOVE is not as good as BERT in the face of schema items containing multi-words. As opposed to static embeddings provided by GLOVE, BERT provides dynamic lexical representations generated by analyzing the context. Take the bandmate id column in the Spider dataset as an example. The cosine of the vectors for the two words bandmate and id in GLOVE is negative, which means if we sum these two vectors together to represent the bandmate id column, the sum vector will inevitably lose some information. The word vector output by BERT is calculated based on the context, so although adjacent words may be unrelated in word meaning, their word vectors will still be highly correlated. Figure 9, generated by the bertviz (Vig, 2019), presents the BERT head view
of attention patterns in the one transformer layer where the word bandmate clearly links to the word id.
F.2 Error Analysis Table 8 presents the error type statistics in the error column prediction. We count the prediction errors of single words, multiple words, and words outside the GLOVE vocabulary (UNK word) when the predicted SQL structure is correct. As BERT does not share GLOVE’s vocabulary limitations, the UNK entry for RATSQLB is empty. Random initialization means that model results after each training may vary slightly, so we only focus on the more salient features.
Although the results of RATSQL and RATSQLO are similar, RATSQLO consistently outperforms RATSQL in three error types when EMSL is removed; this supports the view we discuss in Section 2.2. More importantly, the single-word performance of RATSQLO without EMSL is close to that of RATSQL and RATSQLO. As discussed in Appendix F.1, the representation ability on multiword of GLOVE is worse than that of BERT. The results support this view where the performance of RATSQLO and RATSQL on multi-word is worse than that on single-word. When replacing the GLOVE with BERT, due to the improvement of its multi-word representation ability, the performance of RATSQLB with and without EMSL are close in single and multiple words. From the right side of Table 8, it can also be found that the BERT brings around 5% absolute improvement on multi-word, while that on single-word is only 2%.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5
3 B1. Did you cite the creators of artifacts you used? Section 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We consider extending our study to recent pretrained large language models as future work