GR: A Graph-Guided Generate-and-Rerank Framework for Complex and Cross-domain Text-to-SQL Generation
Year: 2023

Abstract:
We present a framework called GR for complex and cross-domain Text-to-SQL generation. GR aims to address two limitations of current approaches: (1) The structure of the abstract syntax tree (AST) is not fully explored during the decoding process which is crucial for complex SQL generation; (2) Domain knowledge is not incorporated to enhance their ability to generalise to unseen domains. GR consists of a graph-guided SQL generator and a knowledge-enhanced re-ranking mechanism. Firstly, during the decoding process, an ASTGrammar bipartite graph is constructed for joint modelling the AST and corresponding grammar rules of the generated partial SQL query. The graph-guided SQL generator captures its structural information and fuses heterogeneous information to predict the action sequence, which can uniquely construct the AST for the corresponding SQL query. Then, in the inference stage, a knowledge-enhanced re-ranking mechanism is proposed to introduce domain knowledge to re-rank candidate SQL queries from the beam output and choose the final answer. The SQL re-ranker is based on a pre-trained language model (PLM) and contrastive learning with hybrid prompt tuning is incorporated to stimulate the knowledge of the PLM and make it more discriminative. The proposed approach achieves state-of-theart results on the Spider and Spider-DK benchmarks, which are challenging complex and cross-domain benchmarks for Text-to-SQL semantic analysis.

heading: 1 Introduction
text: Complex and cross-domain Text-to-SQL generation aims to translate natural language utterances into structurally complex SQL queries, where no database overlaps in training and testing. The challenges mainly lie in two aspects: (1) Zero-shot cross-domain setting, databases are not overlapped
∗Corresponding author.
between training and test sets and belong to different domains. A domain represents a certain type of application scenario, e.g. medical and geographical. It is difficult for approaches to understand domain knowledge properly (Gan et al., 2021a) and generalise well to unseen databases. (2) Complexity, SQL queries involving multiple tables, containing nested queries and clauses such as GROUPBY and HAVING, are more complex in their structure.
Recently, various works have been proposed for complex and cross-domain Text-to-SQL generation based on the encoder-decoder paradigm. In order to improve generalizability to unseen database schemas, many approaches focus on the encoder part. Graph-based approaches (Cao et al., 2021; Wang et al., 2020a) have been proposed for jointly encoding the natural language utterance and relational structure of the database schema, thereby capturing the semantic relationships between them. For generating complex SQL queries, most decoding approaches (Cao et al., 2021; Guo et al., 2019; Wang et al., 2020a) pre-define abstract syntax description language (ASDL) rules and autoregressively generate the SQL query as an AST. Specifically, an AST is produced progressively in a pre-order depth-first traversal order based on a sequence of actions. In this way, the skeleton of complex SQL queries can be generated flexibly.
Despite much progress on complex and crossdomain Text-to-SQL generation, there are still two limitations. Firstly, The structure of AST is not fully explored during the decoding process which
338
is crucial for generating complex SQL queries correctly. They predict the next action only based on the previous action sequence, without considering the structural information of the AST and how it changes dynamically. In addition, domain knowledge required by different domains is not incorporated, which results in the model not being well generalised to new domains. As shown in Table 1, for the first sample, “target airport” does not appear in the utterance and the model needs to understand that “Arriving at Luton Airport” means “Luton Airport” is the target airport for generating the correct SQL query. For the second sample, players should be listed in ascending order of birth, as the younger the birth date, the older the age. If the requirement is changed to “the ages of all players are listed”, then it should be in descending order. (Gan et al., 2021b) has pointed out that understanding domain knowledge is crucial for cross-domain Text-to-SQL generalization.
To address the aforementioned limitations, this paper proposes a Graph-Guided Generate-andRerank Framework (G3R) for complex and crossdomain Text-to-SQL generation. It consists of a graph-guided SQL generator and a knowledgeenhanced re-ranking mechanism. To make better use of the structural information of AST, we construct an AST-Grammar bipartite graph for both the AST and grammar rules of the generated partial SQL query. A graph-guided SQL generator is proposed to capture structural changes of the bipartite graph dynamically and fuse the heterogeneous information from the encoder to predict the action sequence.
Furthermore, we try to improve generalizability to unseen domains from a different perspective, by introducing a knowledge-enhanced re-ranking mechanism to choose the best SQL query from the beam output in the inference stage. PLM is adopted as an SQL re-ranker and hybrid prompt tuning is adopted to make the re-ranking task similar to language modelling, stimulating the knowledge in PLMs to bridge the gap between different domains. In addition, sometimes the differences between candidate SQL queries are subtle and contrastive learning is adopted to push away the distance of candidate queries’ representations and thus make them more distinguishable.
In summary, the main contributions of this paper are listed as follows:
• A Graph-Guided Generate-and-Rerank
Framework (G3R) is proposed for complex and cross-domain Text-to-SQL generation. An AST-Grammar bipartite graph is constructed to model the AST and grammar rules of the generated partial SQL query jointly and a novel graph-guided SQL generator is proposed to capture the structural information and fuse heterogeneous information to generate the SQL query.
• A knowledge-enhanced re-ranking mechanism is proposed to introduce domain knowledge to choose the best SQL query from the beam output. PLM is adopted as an SQL reranker and contrastive learning with hybrid prompt tuning is incorporated to stimulate the knowledge of PLMs and make it more discriminative. As far as we know, we are the first to leverage the abundant knowledge of PLMs to re-rank SQL queries to mitigate performance degradation in unseen domains.
• Comprehensive experiments were conducted on Spider and Spider-DK. The results show that the proposed approach achieves superiority over some state-of-art approaches.

heading: 2 Related work
text: Our work is related to two lines of research, complex and cross-domain Text-to-SQL generation and prompt tuning.

heading: 2.1 Complex and cross-domain Text-to-SQL generation
text: In order to improve cross-domain generalizability, many approaches focus on the encoder part. (Bogin et al., 2019a) adopts a graph neural network (GNN) to deal with the graph structure of database schema. (Cao et al., 2021; Wang et al., 2020a) construct a heterogeneous graph for jointly encoding natural language utterances and relational structure in the database schema so that the network can generalise to unseen database schemas. In contrast, our proposed approach tackles this problem from a different perspective, by exploring the knowledge of PLMs and re-ranking candidate SQL queries.
Most decoding approaches for complex and cross-domain Text-to-SQL generation (Brunner and Stockinger, 2021; Wang et al., 2020a; Guo et al., 2019) consider the ASDL rules as the prior knowledge and adopt a syntax-based SQL generator (Yin and Neubig, 2017) to output a sequence
of actions based on a pre-defined fixed set of grammar rules. Then an abstract syntactic tree (AST) corresponding to the target SQL query is uniquely constructed in pre-order depth-first traversal order. However, they cast SQL generation into sequenceto-sequence translation and ignore the structural information of AST and how it dynamically changes during the decoding process.

heading: 2.2 Prompt Tuning
text: Fine-tuning PLMs with task-specific heads has been widely applied to natural language processing and achieved great success in many downstream tasks (Radford et al., 2019; Zhou et al., 2021). However, the abundant knowledge in PLMs is not fully exploited because there is a big gap between the fine-tuning objectives and pre-training objectives. Subsequently, GPT-3 (Brown et al., 2020) proposed prompt tuning for downstream tasks. It is a new paradigm that adopts natural language prompts to make downstream tasks similar to language modelling and does not require the addition of model parameters.
A number of works defined hard templates manually where each token is meaningful and understandable (Schick and Schütze, 2021; Gu et al., 2022) and some approaches generated hard templates automatically (Gao et al., 2021; Shin et al., 2020). However, there is no need to limit templates to be human-interpretable because the aim is to find a way to enable PLMs to perform downstream tasks effectively. Soft prompts (Wu and Shi, 2022; Lester et al., 2021) have been proposed and the tokens (i.e. virtual tokens) in the template are continuous vectors which can be learnt during the tuning stage. In addition, (Han et al., 2022) proposed to insert some tunable embeddings into a hard prompt template which is called “Hybrid Prompt Tuning”.

heading: 3 Preliminaries
text: Autoregressive top-down SQL generation In this paper, we tackle the SQL generation problem by generating the AST through syntax-based autoregressive top-down decoding (Yin and Neubig, 2017; Krishnamurthy et al., 2017; Rabinovich et al., 2017; Yin and Neubig, 2018), which guarantees to decode of syntactically valid SQL queries. The generation process is considered to be the sequential application of actions. We define three actions including ApplyRule, SelectTable and SelectColumn. The ApplyRule action applies a grammar rule to
expand a non-terminal node in the AST, while the SelectTable and SelectColumn actions populate a terminal node by appending a table or a column name from the database schema.
In AST, non-terminal nodes sketch the general structure of the target SQL query, while terminal nodes correspond to operations, constants and variables. The decoding process does not end until all leaf nodes are terminal nodes and the SQL generation is regarded as a sequence generation problem.
Problem Setting Given a natural language question Q = {w1, w2, ..., wn} and a corresponding database schema S = 〈C, T 〉, the goal is to generate an action sequence â, which can construct an AST and deterministically be converted to a SQL query y. Here the database schema S consists of tables T = {t1, t2, ..., t|T |}, columns C = {c1, c2, ..., c|C|} and a set of foreign key, primary key column pairs describing relations between columns.
AST-Grammar Bipartite Graph A bipartite graph is a simple graph where the graph vertices are decomposed into two disjoint sets such that no two graph vertices within the same set are adjacent. In order to represent the structure of the partial AST and grammar rules that have been generated, we design an AST-Grammar Bipartite graph G = 〈 Vast, E ,Vg 〉 . Vg is the set of grammar rule nodes and Vast is the set of AST nodes, which refers to the head and child nodes of all grammar rule nodes. E is the set of edges. A grammar rule node is considered an intermediate node between its head and child AST nodes, so a grammar rule node is only adjacent to AST nodes and vice versa. As shown in Figure 1, the grammar
rule “2 : sql_unit → (from, select)” is composed of one head AST node “sql_unit” and two child AST nodes “from” and “select”. When a grammar rule is generated, the graph will be expanded. The dashed box in Figure 1 represents the nodes and edges generated after the ‘sql_unit’ node is expanded using the grammar rule 2. In the beginning, the graph has only one root AST node “sql”. As the actions are generated, this graph gradually expands until the SQL is completely generated.
The AST-grammar bipartite graph jointly models the generated partial AST and the corresponding grammar rules. After graph representation learning, the representation of a grammar rule node contains not only the structural information of the AST, but also more fine-grained information from its surrounding AST nodes. Each AST node represents a construct occurring in the SQL query and can guide future SQL generation. For example, the AST node “from” in Figure 1 means that the subtree with it as the root node is the from clause of the SQL query. Therefore, the next grammar rules generated should be used to create the from clause. For existing grammar rule embedding approaches in code generation, Grape (Zhu et al., 2022) creates a grammar relation graph that only includes grammar rule nodes, TreeGen (Sun et al., 2020a) enhances grammar rules by adding position and depth embedding and uses Tree Convolution to incorporate information from their ancestors, GrammarCNN (Sun et al., 2019) adopts three CNNs to capture the local fea-
ture of an AST node, features of AST sequences and tree paths. However, They considered either the structure of AST or the relationship between grammar rules rather than modelling them jointly.

heading: 4 Method

heading: 4.1 Graph-Guided SQL Generator
text: As mentioned above, existing SQL generators lack the ability to capture the structure of AST. We propose a graph-guided SQL generator based on the Encoder-Decoder framework to solve this problem. The architecture of the proposed graph-guided SQL generator is shown in the left part of Figure 2.
We leverage LGESQL (Cao et al., 2021) (Please refer to the Appendix A.1 for further details) encoder to embed the question and database schema items into joint representations W = [w1, ...,wn], T = [t1, ..., t|T |], C = [c1, ..., c|C|] for each question token wi ∈ Q, table ti ∈ T , and column ci ∈ C respectively. Considering a generated partial action sequence â′ = {a1, .., at−1} and the corresponding AST-Grammar Bipartite graph Gbt−1, the graph-guided SQL generator aims to learn the structural information of Gbt−1 and fuse multiple heterogeneous inputs to predict the next action at. It consists of two parts: (1) Bipartite Graph Reader, graph attention network (GAT) (Veličković et al., 2017) is adopted to learn the node representation of Gbt−1; (2) SQL Decoder, a neural network which is similar to the decoder of Transformer is proposed
to fuse heterogeneous input and predict the next action.
Bipartite Graph Reader In the decoding process, the AST-Grammar Bipartite graph will be expanded gradually. As the graph structure will change dynamically, we apply GAT which employs multi-head attention for graph representation learning.
For the graph Gbt−1 that containsm grammar rule nodes and h AST nodes, it has a node list N = {ng1, ..., ngm, nast1 , ..., nasth }. Grammar rules and AST elements are represented as trainable embedding matrices Xg ∈ R|V g |×k, Xast ∈ R|V ast|×k where |V g|, |V ast| are the sizes of grammar rule dict and AST element dict respectively. k is the dimension of the embedding vector. We initialize the representation of each node by looking up the table. For the special AST node ‘tab_id’ and ‘col_id’, which can generate tables and columns, we initialize it using the corresponding representation from the encoder. In this way, we have node initial embeddings X = [xg1, ...,x g m,x ast 1 , ...,x ast h ] ∈ R(m+h)×k. Then we use the GAT network to obtain the final hidden vectors of grammar rule nodes Hg = [hg1, ...,h g m] ∈ Rm×k that incorporate the structural information of AST.
SQL Decoder The SQL decoder considers heterogeneous information, including the question Q, database schema S, generated actions â′ and the structure of Gbt−1 to predict the next action at. The neural structure of the SQL decoder is similar to the decoder of Transformer (Vaswani et al., 2017) because it can help alleviate long-dependency problem (Sun et al., 2020b). It is composed of a stack of blocks (N blocks in total), and each block contains three sub-layers (namely, the self-attention sub-layer, the encoder attention sub-layer and the feed-forward sub-layer). The residual connection and layer normalization are incorporated into each sub-layer. The input of the SQL decoder is Hg.
Encoder Attention Sub-layer The main difference with the decoder of Transformer is in the encoder attention sub-layer. The decoder should be aware of the question and the database schema when predicting an action. The encoder attention sub-layer adopts multi-head attention to incorporate the information from the encoding stage in a way similar to the decoder’s attention to the encoder in Transformer (Vaswani et al., 2017). Concretely speaking, we concatenate the representa-
tions of the questions with the representations of the database schema items E = [W ,T ,C] which are considered as key, value of the multi-head attention layer.
Action Prediction The final representation H ∈ Rm×k is obtained through the stack of the blocks. We apply self attention to construct a representation Hr ∈ Rk for ApplyRule action:
Ar = softmax(tanh(WrH)) (1)
Hr = ArH (2)
where Wr is a parameter. Ht ∈ Rk and Hc ∈ Rk are constructed for SelectTable and SelectColumn action in the same way.
For ApplyRule Action, the probability distribution Pr over the grammar rule space is calculated as follows:
Pr = softmax(tanh(WsHr)) (3)
Where Ws is a weight parameter of the fully connected layer mapping Hr to the grammar rule space.
For the SelectTable action (SelectColumn action), we apply multi-head attention to calculate the similarity matrix At ∈ R|T | where the query is Ht, key and value are T . Then softmax activation function is applied to calculate the probability for all tables Pt ∈ R|T |:
Pt = softmax(At) (4)
The decoding process of generating a SQL query is shown in Algorithm 1. The model is optimized by minimizing the negative log-likelihood Lg of the ground truth action sequence:
Lg = − N∑
i=0
log
Ni∏
j=1
p(aj |a<j , Q, S) (5)
Where N denotes the number of samples in the training set and Ni represents the length of the action sequence for the i-th sample.

heading: 4.2 Knowledge-Enhanced Re-ranking Mechanism with contrastive soft prompt tuning
text: In the inference stage, we applied the beam search decoding method which keeps M locally best candidates from the decoder output and generates the
M most likely candidate action sequences. However, top-1 prediction does not necessarily correspond to the best SQL query and (Kelkar et al., 2020) first introduced a re-ranking algorithm on complex Text-to-SQL generation and the re-ranker is based on BERT (Devlin et al., 2019) with a linear classification layer. The improvement of their proposed re-ranking algorithm is unstable and highly dependent on a threshold. And at certain threshold settings, it even has a negative effect. The reason is they don’t explore the knowledge of PLM sufficiently. (Hui et al., 2021) proposed a featureenhanced re-ranker which is also based on PLM with a linear layer. In order to introduce domain knowledge, we propose a knowledge-enhanced reranking mechanism. The SQL re-ranker is based on a PLM and hybrid prompt tuning is adopted to explore the knowledge in PLMs to bridge the gap between different domains without introducing extra parameters. Contrastive learning is applied to push away distances of candidate queries’ representations so that they are more distinguishable.
Hybrid prompt tuning PLM is adopted as the SQL re-ranker and the template T is designed by inserting some tunable embeddings into a hard prompt:
{E([BES]), V, E(“it”), E(“is”), E([mask]), E(Q), E(y), E([SEP])} (6)
Where V = [v1, ..., vx] is a learnable virtual template word and x is the length. Q and y represent the question and query respectively. The scoring task can be formalized as a binary classification task and transformed into a cloze-style objective form, by predicting the masked position and mapping the predicted words to the corresponding class labels. The label word set is defined as {“bad”, “great”} and we can determine whether the candidate query is the selected answer or not based on the PLM predicting “great” or “bad” at the mask position. The probability distribution P ∈ R2 over label space is calculated as follows:
Hv = MLMhead(T,Q, y) (7)
P = softmax(Hv(“bad”), Hv(“great”)) (8)
Contrastive Learning The differences between the candidate queries sometimes are subtle, resulting in undifferentiated representations of the mask
position and similar scores for different queries. We introduce contrastive learning to push away the representation of each candidate query, thus making them more distinguishable.
Given a question Q and all its candidate queries Y = {y1, ..., yM}, we construct inputs I = {iy11 , iy12 , ..., iyM1 , iyM2 } for SQL re-ranker by using two different prompt template T1 and T2 where V1 and V2 are distinct and randomly initialised. For a query yi, the inputs i yi 1 and i yi 2 constructed from two templates form a positive pair while negative pairs are formed with other 2(M−1) inputs. All inputs I for Q are in a mini-batch and the contrastive learning loss (Oord et al., 2018; Chen et al., 2020) Lcl is calculated as follows :
Lcl = −log 2M∑
i=1
exp(sim(zi, z+)/τ)
exp( ∑2M
j=1 1[j 6=i]sim(zi, zj)/τ) (9)
where zi, z+ represent the hidden states of the PLM output at the mask position of the i-th sample and its corresponding positive sample respectively. 1[j 6=i] ∈ {0, 1} is a indicator function evaluating to 1 iff i 6= j. sim(·) is the cosine similarity function and τ is a temperature hyper-parameter that controls the sensitivity of the product. Further more, we introduce a margin loss Ld to constrain the two templates to be different:
Ld = max(0, cos(V1, V2)− d) (10)
where cos(·) is the cosine similarity and d is a hyperparameter. Cross-entropy loss Lce is adopted as the loss function for SQL re-ranker following the same way as (Nam et al., 2014):
Lce =− NQ×2M∑
i=1
2∑
j=1
(yij log(P j i ))
+ (1− yij)(1− log(P ji ))
(11)
where NQ represents the number of questions. The total loss function Lr for SQL re-ranker is the sum of the three loss functions:
Lr = Lce + λ1Lcl + λ2Ld (12)
where λ1 and λ2 are hyperparameters. In order to fine-tune the SQL re-ranker, we construct a training (dev) set containing abundant positive and negative samples. For the training (dev) set, positive samples are the ground truth, while negative samples are the wrong samples generated by the SQL generator on the training (dev) set.
Knowledge-Enhanced Re-Ranking Mechanism For each query yi ∈ Y , the log probability of a positive label output by the SQL re-ranker is considered to be the semantic score. Two scores sT1i , sT2i can be obtained by using two templates. We take into account the log-likelihood sgi of its corresponding action sequence produced by the SQL generator and get the final score si:
si = s g i +
sT1i + s T2 i
2 (13)
The candidate query with the highest score is considered the final result. The proposed SQL reranker and knowledge-enhanced re-ranking mechanism are general and model agnostic which can be applied to any Text-to-SQL approach.

heading: 5 Experiments

heading: 5.1 Dataset
text: In order to evaluate the proposed Graph-Guided Generate-and-Rerank framework, we conduct experiments on Spider (Yu et al., 2018) and SpiderDK (Gan et al., 2021b).
Spider It is a large-scale complex and crossdomain Text-to-SQL generation dataset1. There are 8659 samples in the training set across 146 databases and 1034 samples in the development set across 20 databases distinct from those in the training set. The test set is not public and can only be accessed through an evaluation server, we evaluate the proposed approach mainly on the development set.
Spider-DK It is a human-curated dataset based on the dev set of Spider 2. The questions of the dataset are selected from Spider with some domain knowledge added. It focuses on evaluating the ability to understand domain knowledge.

heading: 5.2 Implementations
text: For the Graph-Guided SQL generator, we use GLOVE (Pennington et al., 2014) word embeddings of size 300 and ELECTRA (Clark et al., 2020) to encode the input question and the database schema, followed by 8 LGESQL encoder layers. The number of GAT layers is 2 and the number of parameters is about 385M (With ELECTRA). The whole model is trained by the Adam optimizer
1Spider: https://yale-lily.github.io//spider. 2Spider-DK: https://github.com/ygan/Spider-DK
(Kingma and Ba, 2015) with a learning rate of 5e-5 and 2e-4 for with and without a pre-trained language model (PLM) respectively. The mini-batch size of the input is set to 10. In the inference stage, the beam size is set to 5.
As for the SQL re-ranker, Grappa (Yu et al., 2021) is adopted for complex and cross-domain text-to-SQL generation because it is pre-trained for table semantic parsing and contains rich knowledge to bridge the gap between different domains. The number of parameters is about 355M. The length x of the learnable virtual template word for the hybrid prompt is set to 10, and λ1 and λ2 are set to 0.1 and 1 respectively. It is optimized by Adam with a learning rate of 1e-5.

heading: 5.3 Evaluation Metrics
text: Following the previous work (Cao et al., 2021), we adopt exact match accuracy as the evaluation metric. It is calculated by decomposing SQL into several clauses and conducting a set comparison in each SQL clause.
Model Easy Medium Hard Extra ALL
Without PLM
IRNet 70.1 49.2 39.5 19.1 53.2 RAT-SQL 80.4 63.9 55.7 40.6 62.7 LGESQL 86.3 69.5 61.5 41.0 67.6
G3 87.1 71.3 64.9 39.8 69.0 G3R 89.1 74.1 67.8 41.8 71.4
With ELECTRA

heading: 5.4 Main Results
text: Table 2 shows the experimental results of several Text-to-SQL methods without and with PLM. Regarding baseline models, LGESQL achieves the best performance and our proposed G3R clearly outperforms it by a substantial margin which shows the superiority of our approach. With word vectors GLOVE, G3R achieves an absolute improvement of 3.8%, 1.7% over LGESQL on exact match accuracy on the dev set and test set respectively. With PLM ELECTRA, G3R surpasses all the SOTA baselines and achieves an absolute improvement of 3% and 0.9% over LGESQL on the dev set and test set respectively. It achieves an accuracy of 78.1%, and 72.9% on the dev and test set which is competitive compared to some newly released T5-3B based approaches (Li et al., 2023; Zhao et al., 2022). However, the size of our model is only one-fifth of theirs.
In addition, more fine-grained performances of some approaches ordered by the level of difficulty are shown in Table 3. We can observe that: (1) As the difficulty increases, the syntactic structure of SQL queries becomes more complex and the accuracy of all approaches decreases. (2) With word vectors GLOVE, compared with LGESQL, G3R achieves improvements of 2.8%, 4.6%, 6.3% and 0.8% at the easy, medium, hard and extra hard levels respectively. With PLM ELECTRA, G3R also achieves improvements of 4.4%, 8.1% and 2.5% over LGESQL at medium, hard, and extra hard levels respectively. It demonstrates the effectiveness of our proposed framework. (3) When the re-ranking mechanism is removed, the accuracy of G3 (with ELECTRA) is decreased by 1.5%, but it
still achieves an improvement of 1.5% compared with LGESQL. Notably, G3 (with ELECTRA) has a significant improvement in complex SQL query generation, achieving improvements of 4.6% and 0.6% at the hard and extra hard levels. It demonstrates that our proposed graph-guided SQL generator can exploit the structural information of the AST and generate complex structured SQL queries more accurately.

heading: 5.5 Re-Ranking Results
text: We construct the training set and dev set of the SQL re-ranker in the manner mentioned in Section 4.2. Bertrand-DR (Kelkar et al., 2020) is adopted as a baseline for comparison with our knowledge-enhanced re-ranking mechanism. To be fair, it is also based on Grappa, with a linear layer
for binary classification. As shown in Table 4, the beam accuracy is on average about 6% higher than the top-1 accuracy. Our proposed KE-R can effectively filter out the correct answer from the beam. When it is applied, the exact match accuracy of G3, G3+ELECTRA and are improved by 2.4% and 1.5% respectively. Compared with Bertrand-DR, KE-R in combination with G3 and G3+ELECTRA achieves improvements of 0.7% and 0.9% which shows its superiority.
To further evaluate the ability of our proposed SQL re-ranker to introduce domain knowledge, we conduct experiments on Spider-DK and the results are shown in Table 5. It is obvious that: (1) The dataset is more challenging and the accuracy of all approaches has decreased. However, our proposed G3 still outperforms the other two baselines and achieves an improvement of 2.4% compared with LGESQL+ELECTRA. (2) With KE-R, the accuracy of G3+ELECTRA is improved by 1.9% compared to 0.8% with Bertrand-DR, indicating that our proposed SQL re-ranker can effectively incorporate domain knowledge. (3) When both contrastive learning and margin loss are removed, the accuracy drops to 50.6% and it still gains an improvement of 0.9% compared with G3+ELECTRA which demonstrates the effectiveness of the hybrid prompt tuning. (4) When contrastive learning and margin loss are removed, the accuracy decreases by 0.8% and 0.6% respectively, suggesting that they both contribute to the significant improvement.

heading: 6 Conclusion
text: In this paper, a novel Graph-Guided Generated-andRerank framework is proposed for complex and cross-domain Text-to-SQL generation. In specific, we design the AST-Grammar Bipartite graph and propose a Graph-Guided SQL generator to capture the structural information of the generated complex SQL query. A knowledge-enhanced re-ranking mechanism is proposed to introduce domain knowledge to bridge the gap between different domains and re-rank candidate SQL queries generated from top-ranked action sequences. Experimental results on the Spider and Spider-DK benchmark datasets show that the proposed method outperforms other competitive Test-to-SQL baselines.

heading: 7 Limitation
text: Our proposed graph-guided SQL generators are superior in generating complex SQL queries. How-
ever, the model has a large number of parameters and requires more computational resources, which is a common problem with current methods of generating complex SQL queries.
In addition, the proposed knowledge-enhanced re-ranking mechanism is proposed to leverage the knowledge in PLM to choose the best SQL query from the beam output. However, it does not take into account the database schema which can be the source of domain knowledge.
In the future, we will design lighter models for complex and cross-domain text-to-SQL generation and explore some other re-ranking mechanisms to incorporate the prior knowledge of database schema.

heading: 8 Acknowledgement
text: We would like to thank anonymous reviewers for their valuable comments and helpful suggestions. We thank Tencent Cloud Xiaowei and the Big Data Computing Center of Southeast University for supporting this project. We thank Tao Yu, Yusen Zhang and Hongjin Su for their careful assistance with the evaluation. This work was funded by the National Natural Science Foundation of China (62176053).

heading: A Appendix
text: A.1 LGESQL Encoder
Unlike other code generation tasks, Text-to-SQL generation needs to consider not only the natural language question but also the database schema. It is a daunting problem how to jointly encode the
Algorithm 1 The process of generating a complex SQL query.
Input: Question q, Database schema S, Maximum decoding steps Md. Output: The action sequence â corresponding to the SQL query.
1: â← (“[BES]” ), Gb0 ← “sql” 2: W ,T ,C ← Encoder(Q,S) 3: for i← 1; i ≤Md; i++ do 4: Bipartite Graph Reader captures the struc-
ture information of Gbi−1 and Hg is obtained (4.1).
5: ai ← Decoder(Hg,W ,T ,C) 6: if ai = “[EOS]” then 7: â← â+ ai 8: Break 9: else
10: Gbi ← Update the graph structure of Gbi−1 by applying the predicted action ai. 11: â← â+ ai 12: end if 13: end for 14: return â
question and database schema, as there exist various relations between these heterogeneous inputs. In order to address the problem, LGESQL constructs a node-centric heterogeneous graph Gn. It consists of three kinds of nodes: tables, columns and question tokens. The relations between different types of nodes are shown in Table 6.
Besides, there exist many meta-paths between the nodes. Meta-path is defined as a composite relation linking two nodes, that can capture multi-hop semantics. For example, a meta-path <Question, “Question-ExactMatch-Column”, Column, “Column-BelongsTo-Table”, Table> means the question mentions a column in the table. As the length of the path increases, the number of possible meta-paths increases exponentially. In order to tackle this problem, LGESQL proposes to utilize a line graph to capture the topological structure of edges explicitly. An edge-centric graph Ge is constructed from the original node-centric heterogeneous graph Gn and a dual relational attention network (Dual RGAT) (Wang et al., 2020b) is adopted to capture the structure of the original graph and the line graph iteratively. Finally, the representations of the nodes are obtained through the LGESQL encoder.
A.2 Decoding Process The decoding process is shown in Algorithm 1.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1, Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1, Section 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 1, Section 5

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The dataset is open source and uses the setup from previous work.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will design lighter models for complex and cross-domain text-to-SQL generation and explore some other re-ranking mechanisms to incorporate the prior knowledge of database schema