ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation
Authors: Kun Zhang
XieXiong Lin
Yuanzhuo Wang
Xin Zhang
Fei Sun
Cen Jianhe
Hexiang Tan
Xuhui Jiang
Huawei Shen
Venue: EMNLP 2023 Findings (2023)
Keywords: Text-to-SQL
Retrieval-Augmentation

Abstract: Text-to-SQL is the task that aims at translating natural language questions into SQL queries.
Existing methods directly align the natural language with SQL Language and train one encoder-decoder-based model to fit all questions. 
However, they underestimate the inherent structural characteristics of SQL, as well as the gap between specific structure knowledge and general knowledge. This leads to structure errors in the generated SQL.
To address the above challenges, we propose a retrieval-argument framework, namely ReFSQL.
It contains two parts, structure-enhanced retriever and the generator.
Structure-enhanced retriever is designed to identify samples with comparable specific knowledge in an unsupervised way. 
Subsequently, we incorporate the retrieved samples’ SQL into the input, enabling the model to acquire prior knowledge of similar SQL grammar. 
To further bridge the gap between specific and general knowledge, we present a mahalanobis contrastive learning method, which facilitates the transfer of the sample toward the specific knowledge distribution constructed by the retrieved samples.
Experimental results on five datasets verify the effectiveness of our approach in improving the accuracy and robustness of Text-to-SQL generation.
Our framework has achieved improved performance when combined with many other backbone models (including the 11B flan-T5) and also achieved state-of-the-art performance when compared to existing methods that employ the fine-tuning approach.

ReFSQL: A Retrieval-Augmentttion Framework for Text-to-SQL
Generation
Kun Zhangl? 2,2 Xiexiong Lin", Yuanzhuo Wangk24 Xin Zhang', Fei Sun Jianhe Cen', Xuhui Jiang 12 Hexiang Tan Huawei Shen 1.2
DDaaa Intelligence System Research Center, Institute of Computing Technology, Chinese Academy of Sciences; "School of Computer Science and Technology, University of Chinese Academy of Sciences; *Ant Group. "Big Data Academy, Zhongke; (zhangkun 8z, wangyuanzhuo, sunfei, jiangxuhui 19g, tanhexiang21s)@ice ac. cn (shenhuawei jeiteeaawwwwwweee..e...eeee ac.cn {xiexiong. 1e.xxxxxxgxoooo com com {evanzhangxin)ggaii... com
Abstract
Esample
Exsemple
Q1: tunes daiia
Qt: What fao diffrveet eoch staino
Text-to-SQL is the task that aims at translating natural language questions into SQL queries. Existing methods directly align the natural language with SQL Language and train one encoder-decoder--asse model to fit all ques- tions. However, they underestimate the inher- ent structural characteristics of SQL,, as well as the gap between specific structure knowledge and general knowledge. This leads to struc- ture errors in the generated SQL. To address the above challenges, we propose a retrieval- argument framework, namely ReFSQL. con- tains two parts, -enaancce-e retriever and the generator. Structure-enhanced retriever is designed to identify samples with compara- ble specific knowledge in an unsupervised way. Subsequently, we incorporate the retrieved sam- ples' SQL into the input, enabling the model to acquire prior knowledge of similar SQL gram- mar. To further bridge the gap between specific and general knowledge, lanobis contrastive learning method, which fa- cilitates the transfer of the sample toward the specific knowledge distribution constructed by the retrieved samples. Experimental results on five datasets verify the effectiveness of our ap- proach in improving the accuracy and robust- ness of Text-to-SQL generation. Our frame- work has achieved improved performance when combined with many other backbone models (including the 11B flan-T5) and also achieved sate-of-the-ar art performance when compared to existing methods that employ the fine-tuning approach.
SE:
atation
Station id
stator
S2:
14
stattse
Scation
SQLISELECT DISTINCT mane EROM
SELECT DISTICTT FROM
ON utebon
Exaeple3
Esample
Q2: Lis compores of to seugs
Qasiiee: For auch iip ondi what monnge poss inppe ty done that the with 87
NL:
Compassc
SQL. SELECT
SILUCT Thanno Compasse JOIN vong A8 ONON T1 TZarepor 38
36aRiE INEER 30 SURON watter
WTEER to
GROUP BY op code
Figure I: Example I, 2, and 3 are three Text-to-Sql in- stances. Example shows the difference in the grammar between natural language and SQL.
translating their information needs into the struc- tured query language (SQL), which serves the standard database language. To facilitate the query- ing process for professsosi users, researchers have introduced the Text-to-SQL task, which seeks to automate the translation of users' natural lan- guage questions into SQL queries.
present maha-
Recently, with the rapid development of lan- guage models, many work (Qi et al., 2022; Li al., 2023) have leveraged one mncoder-deccoderbbseee model to fit the entire training set, directly trans- formed SQL queries into a scrialization structure and aligned them with natural languages. However, such methods can easily lead to structural errors in the generated SQL, c.g., missing "JOIN IN" opera- tions, conducting incorrect comparison operations, etc. The observed phenomenon can be attributed to the underestimation of the inherent structural characteristics as well as the gap between specific knowledge and general knowledge.
1 Introduction
Relational databases, which serve ubiquitous components within data management systems, of- fer means of storing heterogeneous data types, encompassing text, integer, float, and various other formats. However, the proficient utilization of man- aging databases by ordinary users remains a chal- lenge, primarily due to their limited proficiency in "Corresponding author.
Natural languages exhibit inherent characteris- tics where the appearance of each word is highly influenced by the preceding or following sequence. In contrast, as shown in Example 4 in Figure 1, SQL language possesses unique inherent structural characteristics, i.e., heavily relies on the database schema and SQL grammar but lacks robust natural language attributes. Previous methods that directly

---

align the distributions of natural language and SQL query may fail capture these inherent structural characteristics in SQL.
Mahalanobis distance measure of the distance between the sample and the specific distribution. By utilizing the Mahalanobis distance, we employ a contrastive loss function to facilitate the trans- fer of the sample toward the specific knowledge distribution.
In the real world, SQL structures exhibit signif- icant variations due to diverse structural charac- teristics, such as different database schemas and complex SQL grammars. As illustrated in Figure 1, Example and Example undergo drastic changes in their SQL representations when the underlying schema differs. The conventional approach in Text- to-SQL is to train one-size- fits--lttllll model to fit the entire training set, acquiring the general knowledge, i.e., the distribution the global space. However, such models struggle to acquire specific knowledge, i.e. the distribution of the local space, and adapt to the varied samples. The gap between specific and general knowledge significantly contributes to structural errors in the generated SQL. This dis- crepancy is particularly pronounced in cases with complex structures, where the general distribution learned by the model fails to adequately fit these cases unless substantial amount of data, which can be prohibitively expensive in practice, is avail- able.
In general, our main contributions are listed follows:
We propose a retrieval-angmentation frame- work for Text-to-SQL generation, which can adapt to samples with various inherent SQL characteristics and bridge the gap between specific knowledge and general knowledge.
To further bridge the gap between specific and general knowledge, we present a mahalanobis contrastive learning method, which facilitates the transfer the sample toward the specific knowledge distribution.
We verify the effectiveness of the proposed framework on five widely-used datasets and achieve otatc-of-the-ar performance in meth- ods that employ the fine-tuning approach.
To overcome these challenges, we draw inspira- tion from the process of human cognition. As de- picted in Figure 1, humans can compose appropri- ate SQL queries by referring to similar samples that share common specific knowledge, including ques- tion semantics, schema structures, SQL queries. In this paper, we propose eerreval-augmenteed framework for Text-to-SQL. generation. To iden- tify samples with comparable specific knowledge, we design a diructure-enhaneee retriever that takes into account question semantics and schema struc- ture. Subsequently, we incorporate the retrieved samples' SQL into the input, enabling our model acquire prior knowledge of similar SQL gram- mar. To further enhance the model's ability ac- quire specific knowledge, we employ contrastive learning approach. As highlighted in (Kao et al., 2021), the utilization of contrastive learning of- fers a viable solution to mitigate the gaps arising from disparate distributions. The retrieved sam- ples serve as building blocks for constructing the specific distribution that corresponds to the current sample. Through contrastive learning, we guide the samples toward the specific knowledge distribution. However, quantifying the distance from a sample to the distribution poses a challenge within con- trastive learning. To address this, we introduce the
Preliminaries
Given natural language query Q and a database schema S = (T.C), the objective is to generate the corresponding SQL query V The natural language query 2 represented as a sequence of tokens i=1
Q = [ [0] while the schema S consists of
collection of tables T { along with their |C| )- The content of
[7]
associated columns C = [C.) the database S is denoted as V. Each table con- tains set of columns represented as C, {cy))=r- Similarly, table names and column names are to- kenized, such that a table name tę consists of [ty] tokens and the same applics to column names. this study, the predicted SQL query is presented as sequence of tokens, y [1x)
Methodology
In this section, we will describe the proposed frame- work ReFSQL, which comprises two main com- ponents: the structure-enhanced retriever and the generator. The structure-enhanced retriever finds similar samples based on questions and schemas and constructing prompts using their correspond- ing SQL queries. On the other hand, the generator aims to bridge the gap between specific and general

---

Structure-chhaneed Retriever
Generator
Queeiio::
Quesiion Represcutation Learning
Rank Module
Decoder
tlight titghte segarting
0000
Kank List
Negative
Sample
Qsollios
Question 1
* Queiiies
Question:
Contrasiive Leurnings
Question 3
Matt the
Negative
0000
heado of department ordezed
Semple
Question N
puch
puli
Datalias schuma (inclading lables calumns)
Interaction Graph Learaing
Rank
Pusitive
Figure 2: The overall architecture of our framework ReFSQL.
knowledge. achieves this by employing the Ma- halanobis distance to guide the sample toward the distribution of similar samples. Figure 2 provides an overview of the ReFSQL framework.
between two tree formats of SQL queries, denoted as ir1 and (r2. This similarity can calculated using the following equation:
6, TED(tri,
(1)
3.1 Stracture-enhanced Retriever
Here, de represents the similarity between tri and fr2. To construct the positive set, we randomly select ls samples with the highest similarity scores, while for the negative set, we also choose k samples with the lowest similarity scores.
this section, we present the design of a structure- enhanced retriever aimed at identifying samples that exhibit a similar knowledge to the current sample. In Text-to-SQL tasks, the question and schema play crucial roles in generating accurate To enhance the representation of the question and schema, leverage the question- SQL pairs and schema linking techniques. Thus, the structure-enhanced retriever consists of two schema
We employ the pre-trained Bert model (Devlin et al., 2018) 8) with fixed parameters to encode the question and obtain its representation. The encod- ing process is performed as follows:
modules: the retriever (LSES retriever) and the SQL-structure- enhanced question retriever (SQLSE retriever). We provide detailed description of these modules below.
hg: Bert(q:):
(2)
Here, hg. denotes the representation of the question Ji-
We adopt the contrastive learning framework proposed in (Chen et al., 2020) and employ a cross-entropy objective with negative samples as described in (Chen et al., 2017). For each sample di. we denote the positive sample as q,t and the neg- ative sample 0, The training objective for the pair (has h1zt)) with i pairs is defined as follows:
3.1.1 SOL-Structure--ehanned Question Retriever
In this section, we present unsupervised method enhance question representation via SQL struc- ture information.
In our dataset, each sample comprises a SQL query and corresponding question. In order to acquire effective question representations, we gen- erate contrastive samples based on the similarity between SQL queries. Inspired by the methodology introduced in (Yu et al., 2018a), leverage the tree structure of SQL queries quantify their sim- ilarity. Specifically, we employ a straightforward yet powerful method called Pree-Editt- Distance- based (TED) similarity to measure the similarity
exp(Daas cas (heg l//)
Lsc= log
(3)
Enjeq cQ exp DDos (Tigs hag) /T)
Here, of E Q represents the sample constructed from positive and negative samples. 7 is temper- ature hyperparameter, and Doos (h,, h2) denotes the cosine similarity
h h2
[ ha

---

After obtaining the learned question represen- tations, we employ the cosine similarity function to rank the samples. Finally, we select the top m similar samples for each sample.
explicit names assigned to schema compo- nents. Consequenty, we employ distinction between exact matches and partial matches to mitigate the deleterious impact of impre- cise correspondences, thereby minimizing the noise engendered by imperfect matches.
3.1.2
Linking-Stuuctur--basel Schema Retriever
After we obtain samples by the previous Section 3.1.1, we utilize neural-based models to perform a reranking task with focus the schema structure. In this section, introduce the linking- structure-based schema retriever. The construction of the interaction graph, which eaptures the diverse relations between questions and databases, clab- orated. Following that, we adopt an unsupervised learning approach to derive the representation of the interaction graph. By leveraging this graph rep- resentation, we retrieve the relevant samples from the dataset.
Interaction Graph Learning The interaction graph learning module consists of two trainable components: the graph encoder and the edge de- coder. The graph encoder encodes the interaction graph and the edge predictor is used as down- stream task. As mentioned above, the interaction graph is heterogeneous graph. To model the con- tent and structure of the interaction graph, we lever- age R-GCN to encode the interaction graph. We denote the interaction graph of ith sample as gi. then the graph embedding hg. is calculated as.
Interaction Graph Construction We extract a diverse range of relations as triplets, establish- ing connections between tokens derived from both the question and the schema. These triplets are then seamlessly integrated to form a comprehen- sive graph representation. Our primary focus cen- ters around two distinct types of relations: schema encoding and schema linking.
hg R-GGC((g))
(4)
Then we graph embedding learned above to retrieve the most similar samples through the cosine similarity function.
We design a simple yet effective SQL prompt to make the language model learn the SQL char- acter during encoding. The input of ith sample mainly contains three parts, the question Q, the database schema S, involving S, =< T,C >, and the SQL prompt S.P. We follow (Qi al., 2022) to serialize the inputs. Formally,
Schema Encoding. Schema encoding rela- tions pertain to the interconnections among schema items, explicitly delineating the structural information encapsulated within database schema. Notable instances of such relations encompass BELONGS-TE which denotes the association between column and its corresponding table, and FOREIGN-KEY. which establishes a linkage between the for- cign key in one table and the primary key in another table. By comprehensively captur- ing these schema encoding relations, a more holistic understanding of the schema's inter- nal structure can be attained.
SP. the similar SQL was:sqli, sqhi,
sqlip
X, [.[S,tt -11,--- ,1//71 22 621
SPP
(5)
where sqli, is denoted as the jth SQL of similar samples for the i sample. t; is the table name, Cúj is the j-th column name of the i-th table. We adopt the "I" symbol delineate the boundaries between Q, S, different tables, and SQL prompts. We utilize the ":" symbol to differentiate between the table name and its corresponding columns.
Schema linking. Schema linking relations en- compass the associations between schema and question items. In line with RAT-SQL (Wang et al., 2020), we leverage n-gram matches to signify instances wherein the question refer- ences specific schema items. However, the detection of these relations has proven to be formidable task in prior investigations, pri- marily owing to the prevalent discrepancy be- tween natural language expressions and the
3.2 Generator
In this section, we will describe the generator, which consists of two components: the encoder and the decoder.
3.2.1 Encoder
In our framework, encoders can be replaced, so we take the pre-trained model T5 as an example. For the ith sample,

---

between the representation of the current sam- ple zj and the distribution of retrieved samples So (h2,) ~ o2,, This Mahalanobis distance can be calculated as (hx, - -u))0² (hz, - 1i), where denotes the identity matrix. The hyperparameter controls the temperature of the contrastive loss.
hx, TS-Encoder((X)
(6)
where hx, is denoted as the encoded state of X₃
3.2.2 Decoder
As described in Section 1, there exists a gap between the specific knowledge and the general knowledge which disturbs the learning of samples. To further bridge this gap, introduce contrastive learning to guide the sample representation toward the distribution of similar samples and farther away from the distribution of dissimilar samples.
3.2.3 Training Details
The loss in the decoding stage mainly consists of two parts, the MLE loss, and the contrastive loss. Specifically, the MLE loss, denoted by LCE, is computed as follows:
Contrastive Sample Construction To optimize the efficacy of the contrastive learning approach, we propose more refined strategy for construct- ing contrastive samples. Following the method- ology outlined in Section 3.1.1, identify the top m samples, denoted as M. From this set, we select the top & samples after applying the linking-trructur-eenaare
N LCE N
yo, log Dt.j
(8)
Here, N is the number of samples in the training data, while V represents the vocabulary size. Bi denotes the true label of the j-th word in the i- th sample, and Ut, represents the corresponding predicted probability generated by the model.
retriever for construct- ing positive samples. Conversely, we construct a negative sample set from the remaining lower- ranked samples in M. This process allows us to derive a distribution of similar semantic represen- tations based on the positive samples, represented fa N( o²t). Simultaneously, obtain a distribution of dissimilar semantic repre- sentations based on the negative samples, denoted as I (h,-)) N(p oo? ).
Thus, the overall loss is computed as follows:
L = LCE + LMA.
(9)
Experiment
In this section, we present a comprehensive evalu- ation of our proposed framework, ReFSQL, using five extensively utilized benchmark datasets. The obtained results serve as strong evidence to validate the effectiveness of our method.
Mahalanobis Contrastive Learning
To transfer the sample representation toward a distribution of similar semantic samples and distance it from distribution of dissimilar sam- ples, we propose the employment of the Maha- lanobis contrastive mechanism (Li al., 2022b). This mechanism aims to minimize the mar- gin between the sample representation h,, and the similar semantic distribution fo, N(u (o while simultaneously maximizing the margin between the sample representation by: and the dissimilar semantic distribution fo (h,.))
4.1 Datasets and Preprocessing
We conduct experiments on several widely-used Text-to-SQL datasets. The details are shown below. Spider Spider (Yu et al., 2018b) is a challeng- ing benchmark that poses significant demands the cross-domain and multi-table Text-to-SQL. task. The dataset consists of a training set with 7,000 samples, a dev set with 1,034 samples, and a con- cealed test set with 2,147 samples.
N (O"T(
The contrastive loss, denoted Liaas can be
defined as follows:
Spider-DK, Spider-Syn, and Spider-Realistic Realistic To evaluate the robustness of our model, we em- ploy a training strategy using the Spider train- ing set while assessing its performance threc distinct evaluation sets: Spider-DK (Gan et al., 2021b), Spider-Syn (Gan al., 2021a), and Spider- Realistic (Deng et al., 2020). The Spider-DK eval- uation set consists of 535 samples and focuses on the integration of domain knowledge to rephrase
exp (me (Falboz)) ). L.M A-MA=-M0 A= Exsess exp (Dma (Jo(bee)) liss))
(7)
Here, 2² E M represents the samples retrieved by the current sample F;, and Dmo refers to the Ma- halanobis distance (De Maesschalck et al., 2000)

---

Table 1: EM and EX results on Spider's development set, and EM and FI results on WikiSQL's test set (%).
Spider
WikiSQL
Approach
DevEM.
DevEX.
TestEM.
TestFl.
RAI-SOL+GAP+NNSSQL (Gan et al., 2021c)
73.7
75.0
BRIDGE (Linn al., 2020)
71.1
70.3
85.7
91.1
LGESQL + ELECTRA (Xu et al., 2020)
75.1
T5-3B(Shaw et al., 2020)
71.5
74.4
UnifiedSKG(TS-Baan (Xie al., 2022)
71.7
86.0
UNISAR (Dou ct al., 2022)
70.0
86.7
91.7
Uni Parser(TS-Base) (I.iu et al., 2022)
61.2
91.3
Uni-Parser[TS-Laun Large) (Liu al., 2022)
86.9
92.1
ChalGPT (Liu et al., 202.3)
70.1
RASAT + PICARD (Qi et al., 2022)
75.3
80.5
RESDSOLL/TS-BBBNNATTSO (Li al., 2023)
80.5
84.2
RESISQL((S-BBBAATTSS--RREESS
83.1
86.2
RESDSSOLLFFF
84.8
87.0
86.6
88.1
91.7
93.4
the questions. In contrast, Spider-Syn comprises 1034 samples where synonyms are used to replace schema-related words in the questions. Lastly, Spider-Realistie comprising 508 samples, involves the removal of explicitly mentioned column names in the questions to simulate real-world scenarios with more ambiguous queries.
task adapt the seq-to-seq model to the struc- tured input.
LGESQL (Cao et al., 2021) integrates rela- tional features using line graph, and intro- duces an auxiliary task, called graph pruning. enhance the encoder's capability.
WikiSQL (Zhong et al., 2017) is typical table-based question answering dataset, which has 80654 hand-annotated examples of questions, SQL queries and the corresponding answers from execu- tion.
UNIFTEDSKG (Xie et al., 2022) unifies 21 structured knowledge grounding tasks into one single text-to-text model.
Uni-Parser (Liu al., 2022) proposes uni- fied semantic parser method for question an- swering (QA) on both KB and DB.
4.2 Baseline Models and Evaluation Metrics
We compare the proposed model with several base- line methods, including the current state-of-the-art model over the two benchmark datasets.
UNISAR (Dou al., 2022) proposes uni- fied structure-aware model solve text-to- SQL across various settings.
T5: (Shaw al., 2020) applies the pre-trained T5 text-to-SQL. task.
ChatGPT (Liu et al., 2023) explores using chatgpt to solve text-to-SQL. tasks.
RATSQL: (Qi et al., 2022) improves the encoder by adding the relation-aware self- attention module.
To assess the performance of the Text-to-SQL parser, we employ three evaluation metrics: Exact- set-Match accuracy (EM), Execution accuracy (EX), and answer accuracy (F1) (Liu et al., 2022; Zhong et al., 2017). The EM metric determines whether the predicted SQL query can precisely match the gold SQL. query by converting them into specialized data structure. On the other hand, the EX metric compares the execution results of the predicted SQL query with the gold SQL query. In practice, we combine the EM and EX scores evaluate the model and select the model with the best overall performance.
(Li al., 2023) the schema is ranked and relevant schema items are injected into the encoder, while the decoder generates the skeleton first and then the SQL query.
RESDSQL:
(Scholak et al., 2021) improves the decoder by constraining beam search to generate grammatically correct SQL queries.
PICARD:
RAT-SQL- + GRAPPA (Yu et al., 2020) de- signs a schema item classification pre-training

---

4.3 Results on Spider and WikiSQL
deliver impressive performance even on large lan- guage models. Additionally, we investigated the impact of model size. As illustrated in Table 2, our framework consistently showcases larger perfor- mance gaps when compared to its T5-3B counter- part. This observation aligns with previous findings in other fine-tuning tasks, highlighting the ability of larger pre-trained models to capture more knowl- edge effectively.
We implement comprehensive experiments to ver- ify the effectiveness of our framework. The frame- work is integrated with the RESDSQL backend model, which utilizes fine-tuned techniques. The experimental results are presented in Table 1. Our ReFSQL framework, when applied to the RES- DSQL backbone model, outperforms all baseline models on the two benchmark datasets. The ap- proach achieves state-of-the-ar performance in methods that employ the fine-tuning approach. Specifically, after adapting to our framework, the RESDSQL-based model improves the EM by 1.8 on Spider and 1.5 WiKiSQL compared with the original model. In addition, our framework also has an improvement effect on other metries.
Table 3: Evaluation results of removing di different mod- ules of our framework (%).
Spider
Approach
DevEM.
DevEX.
ROS.O//EEEEEaSSSAAASSSS--RRROO RESDSQI (el.n- RRPSQI (w/O the SR)
86.6
88,1
85.3
87.4
(BSDSg..HHa////3MMMSS))----- (w/ the CLM)
87.8
Table 2: Evaluation results of our framework adapted with different modcls(%).
4.5
Ablation Study
We conducted comprehensive ablation study on the development set of the Spider dataset to thor-
Spider
Approach
DevEM.
DevEX.
oughly examine the individual impact of each mod- ule within our framework.
TS-small
47.6
47.8
TS-small-ReFSQL.
54.3
53,9
Ta-B
TS--BSs+ReeSOLL
61.8
61.7
Structure-enhanced retriever To assess the efficacy of the structure-enhanced retriever, we conducted an experiment. Instead, we lied solely Bert for calculating the seman- tic similarity between questions. We selected samples with higher ranks as positive exam- ples and randomly sampled other samples as negative examples within the batch. The re- sults, as presented in Table 3, demonstrate a decrease in EM and EX scores when this mod- ule is removed. This outcome suggests that the retriever plays a crucial role in obtaining similar samples and facilitating the model's 's acquisition of specific knowledge during the text-to-SQL. process.
Ta-Laage
65.3
67.2
ReLLLL
67.8
69.8
T5-3B
71.5
74.4
TS-3B-ReFSQL
73.6
76.5
RASAT-PICARD
75.3
80.5
RASAT-PICARD+REESQQ.
77.2
82.1
RESDIQQ.(15::BB+NAAASSC
80.5
84.2
RESDSQQLTSSBBD-NIISOL-RRROO
83.1
86.0
RESDSQLI/FIn-TTSNNAATOOOL
84.8
87.0
86.6
88.1
4.4 Analyses on Different Models
As mentioned above, our framework can be used for different fine-tuning approaches. To further validate the performance of our framework, we have integrated our framework with some different models, the results are shown in Table 2.
Mahalanobis Contrastive Learning mecha- nism We further investigate the efficacy of the mahalanobis contrastive learning mechanism in our framework. The results, presented in Ta- ble 3, indicate that removing this contrastive learning mechanism results in decrease in the EM and EX metries. This finding sug- gests that the contrastive learning mechanism plays a crucial role in guiding the represen- tation of samples toward a distribution that encompasses similar samples.
Our framework demonstrates flexibility and ef- fectiveness, allowing for adaptation with numer- ous backbone models. Remarkably, our frame- work yields notable improvements when applied to small-size models like T5-small. This suggests bridge the performance
that our framework gap between small and large models, achieving comparable effects. Furthermore, we conducted experiments using larger scale model, Flan-T5, and observed a substantial improvement of nearly 2%. These results indicate that our framework can

---

Table Evaluation results on Spider-DK, Spider-Syn. and Spider-Realiste (%).
Spider-DK
Spider-Syn
Spider-Realistic
Approach
EM. EX.
EM.
EX.
EM.
EX.
LGESOL+ ELECTRA (Xu et al., 2020)
48.4
64.6
69.2
15-3B+PICARID (Shaw et al., 2020)
59.4
65.3
63.2
65.0
RASAT PICARD (Qi et al., 2022)
69.7
71.9
RESDSSQUI13-B))NNASSQQ. (Li et al., 2023)
53.3
66.0
69.1
76.9
77.4
81.9
55.6
67.6
71.3
79.2
79.6
84.2
56.9
69.2
72.8
80.6
82.4
86.5
4.6 Robustness
2022) fuse the base image encoder with relevant images retrieved from the training data to address the challenge of long-tail visual recognition. (Xiao et al., 2021) incorporate relevant sentences in the target style to improve the unsupervised style trans- fer model
To evaluate the robustness of our framework, we train our model on the training set of the Spi- three
der dataset and assess its performance challenging Spider variants: Spider-DK, Spider- Syn, and SpiderRealistic. The results, presented Table 4, reveal a surprising and signifi- cant performance advantage of RESDSQL(Flan T5)+NatSQ overall strong competitors across all three datasets. This finding suggests that our framework can also enhance the robustness of Text-to-SQL parsers.
Conclusion
This paper presents research focus the Text- to-SQL task, aiming to improve SQL generation through a retrieval-augmented framework called Ref-SQL. The main objective is address the gap between specific knowledge and general knowl- edge. To obtain specific knowledge, a structure- enhanced retriever is devised to identify similar samples based on question semantics and schema structure. Additionally, contrastive learning ap- proach is employed to facilitate the transfer of sam- ples towards similar semantic distribution, further mitigating the aforementioned gap. The effective- ness of RefSQL evaluated on five widely-used benchmark datasets, where surpasses all baseline models, verifying its superior performance.
5 Related Work
5.1 Text-to-SQL
In recent studies, significant advancements have been made in enhancing the performance of seq-to- seq models for text-to-SQL tasks, focusing on both the encoding and decoding stages. (Qi al., 2022) introduce relation-aware self-attention into the encoder, enabling the capture of important struc- tural information such as schema linking. (Scholak et al., 2021) proposes beam search constraint dur- ing inference to ensure the generation of grammati- cally correct decoding results. (Li et al., 2023)en- hance the encoder by incorporating the most rele- vant schema items into the input sequence, leading improved performance.
Limitations
To train the retrieval model effectively, it is nec- essary to allocate sufficient computing resources. Additionally, our research focus is limited to the English language due to its availability of wide range of analytical tools and resources, which sur- pass those available for other languages.
5.2 Retrieval-augmentee generation
Reureeee generation, which incorpo- rates large language models with external retrieval modules, has achieved impressive performance in various tasks in recent years (Li et al., 2022a). One line of research focuses enhancing lan- guage models with retrieval modules provide additional knowledge (Si et al., 2022; Borgeaud al., 2022). Another approach involves leveraging retrieval techniques to extract useful information from the training data. For example, (Long et al.,
8 Acknowledgement
Thanks to reviewers for their helpful comments on this paper. This paper is funded by the the National Natural Science Foundation of China (No.62172393, U1836206 and U21B2046), Zhongyuanyingcai program--undee to central plains science and technology innovation leading

---

talent program (N0.204200510002), AntGroup Re- search Intern Program and Major Public Welfare Project of Henan Province 100.203000112000
Yujian Gan, Xinyun Chen. Jinxia Xie, Matthew Purver, John R Woodward, John Drake, and Qiaofu Zhang. 2021c. Natural sql: Making sql easier to infer from natural language specifications, arXiv preprint arXiv:2109.05553
References
Chia- Haia-Hsiang Kao, Wei-Chen Chiu, and Pin- Yu Chen, 2021. Maml is a noisy contrastive learner in classifi- cation. arXiv preprint arXiv:2106.
Sehastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptist
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial In- telligence (AAAI).
Lespiau, Bogdan Damoe, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference chine learning, pages 2206-2240. PMLR.
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. Lgesql: line graph en- hanced text-to-sql model with mixed local and non- local relations. arXiv preprint arXiv:21000 01090.
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022a. A survey eetrieval-augmentt text generation. arXiv preprint arXiv:2202.00 110.
Mingzhe Li. Xiexiong Lin, Xiuying Chen. Jinxiong Chang, Qishen Zhang, Feng Wang, Taifeng Wang. Zhongyi Liu, Wei Chu, Dongyan Zhao, et al. 2022b. Keywords and instances: A hierarchical contrastive learning framework unifying hybrid granularities for text generation. arXiv preprint Ar0iv:2205. 223222355.336.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- termational conference on machine learning, pages 1597-1607, PMLR.
Xi Victoria Lin, Richard Socher, and Caiming Xiong- 2020. Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. arXiv preprint arXiv:2012. 12627.
Ting Chen, Yizhou Sun, Yuc Shi, and Liangjie Hong. 2017. On sampling strategies for neural network- based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 767- 776.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. 2023. A comprehensive evaluation of chat- gpt's shot text-to-sql sqll capability, arXiv preprint arXiv:2303. 13547.
Roy De Maesschalck, Delphine Jouan- RimannRi and Désiré L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems, 50(1):1-18.
Ye Liu, Semih Yavuz, Rui Meng. Dragomir Radev, Caiming Xiong, and Yingbo Zhou. 2022. Uni- parser: Unified semantic parser for question answer- ing on knowledge base and database. arXiv preprint 2xXiv:221.00566:
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2020. Structure- grounded pretraining for text-to-sql. arXiv preprint arXiv:2010. 127.33
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. 2022. Retrieval augmented classification for long-tail tail vi- sual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 6959-6969.
Jacob Devlin, Ming- Meng-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprini 1rXiv:1810. 1810.04055.
Longxu Dou, Yan Gao, Mingyang Pan, Dingzirui Wang, Wanxiang Che, Dechen Zhan, and Tian-Guand Lou. 2022. Unisar: A unified structure-awan aware autoregres- sive language model for text-to-sql. arXiv preprint arXivv2203.07781.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Chenghu Zhou, Xinbing Wang. Quanshi Zhang, and Zhouhan Lin. 2022. Rasat: Integrating relational structures into pretrained seq2seq model for text-to- sql. arXiv preprint arXiv:2205.06900 Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John Woodward, Jinxia Xie, and Peng- sheng Huang. 2021a. Towards robustness of text- to-sql models against synonym substitution. arXiv preprint 10Xiv:2106.0100
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. arXiv preprint arXiv:2109.0550
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2020. Compositional general- ization and natural language variation: Can seman- tic parsing approach handle both? arXiv preprint arXiv:2010.12725.
Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. Exploring underexplored limitations of cross- domain text-to-sql generalization. arXiv preprint arXiv:2109.0115..

---

Chenglei Si. Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Li- juan Wang. 2022. Prompting gpt-3 to be reliable. arXiv preprint 100.09000
Kai Wang. Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020. Relational graph attention network for aspect-based sentiment analysis. arXiv preprint Xix::00204.22600
Fei Xiao, Liang Pang, Yanyan Lan, Yan Wang, Huawei Shen, and Xueqi Cheng. 2021. Transductive learning for unsupervised text style transfer. arXiv preprint arXiv:2109.07812.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text lan- guage models. arXiv preprint arXiv:2201. 00596..
Peng Xu, Dhruv Kumar, Wei Yang. Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Kit Cheung, Si- mon JD Prince, and Yanshuai Cao. 2020. Optimizing deeper transformers small datasets. arXiv preprint a5X1v:2012.15355.
Tao Yu, Chien-Sheng Wu, Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-traiming for table semantic parsing. arXiv preprint arXiv:2009.13844
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang. Zifan Li, and Dragomir Radev. 2018a. Syntaxsqinet: Syntax tree networks for complex and cross-domaintext-tor task. arXiv preprint arXiv:1810.05237.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, et al. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint 180.008877..8
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint (ale::779000000
A Example Appendix
This is section the appendix.


Reviews:
[{"review_id": "9RdWQHdP0d", "reviewer": "Reviewer_fMpS", "summary": "", "questions": "1. Why do authors fix the parameters of BERT (line 213)? If they are not updated, then what’s the purpose of contrastive learning in that section?\n2. What’s the model performance of keeping linking-structure-based schema retriever and Mahalanobis contrastive learning while removing the SQL prompting? Maybe this is helpful to rationalize the input design.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Why do authors fix the parameters of BERT (line 213)? If they are not updated, then what’s the purpose of contrastive learning in that section?\n\nA1: Thank you for taking the time to review our paper and provide feedback. We appreciate your comments on the presentation of our paper.\nWe appreciate your attention to this detail. To address your concern, we want to clarify that during the process of contrastive learning training, we indeed update the parameters of the Bert model. We recognize that the explanation may not have been adequately conveyed in the current version of our paper, and we are committed to addressing this by providing a comprehensive and clear clarification in the revised version. Your insights are invaluable, and we are dedicated to ensuring that our paper reflects the accurate implementation of our approach.\n\nQ2: What’s the model performance of keeping linking-structure-based schema retriever and Mahalanobis contrastive learning while removing the SQL prompting? Maybe this is helpful to rationalize the input design.\n\nA2: We appreciate your insightful comment regarding the model performance in the absence of SQL prompting while retaining the linking-structure-based schema retriever and Mahalanobis contrastive learning. To comprehensively evaluate the contribution of SQL prompting, we performed additional ablation experiments on the Spider dataset. The obtained results indicate that while the impact of SQL prompting is discernible, it remains relatively modest in comparison to the overall performance achieved by our approach. This observation emphasizes the valuable insights gained from your suggestion and contributes to the ongoing exploration of input design optimization.\n\n|                       Model                       | DevEM. | DevEX. |\n|:-------------------------------------------------:|:------:|:------:|\n|           RESDSQL(Flan-T5)+NATSQL+ReFSQL          |  86.6  |  88.1  |\n| RESDSQL(Flan-T5)+NATSQL+ReFSQL(w/o SQL Prompting) |  86.3  |  87.8  |\n|             RESDSQL(3B)+NATSQL+ReFSQL             |  83.1  |  86.0  |\n|    RESDSQL(3B)+NATSQL+ReFSQL(w/o SQL prompting)   |  82.9  |  85.6  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Structure-enhanced retriever is designed to improve similar sample retrieval.\n2. The methods generalize well on different sizes of models such as T5-small and Flan-T5.\n3. Extensive experiments on different variants of Spider datasets to test the robustness of the model.\n", "reasons_to_reject": "Besides applying existing techniques (Li et al. 2022) to the application of Text-to-SQL, there are no significant weaknesses in this work if the authors can answer the questions properly (see Questions). ", "questions_to_authors": "1. Why do authors fix the parameters of BERT (line 213)? If they are not updated, then what’s the purpose of contrastive learning in that section?\n2. What’s the model performance of keeping linking-structure-based schema retriever and Mahalanobis contrastive learning while removing the SQL prompting? Maybe this is helpful to rationalize the input design.\n", "paper_topic_and_main_contributions": "This work proposed a framework called ReFSQL for the task of Text-to-SQL semantic parsing. This framework contains two parts, structure-enhanced retriever and the generator. More specifically, a structure-enhanced retriever that incorporates question semantics and schema structure is proposed. This retriever is used to obtain samples with similar SQL grammar. Two-stage retrieval is used: use question semantics to retrieve a rank list and then use schema structure for reranking. Furthermore, contrastive learning with Mahalanbis distance is used to improve the decoding process, facilitating the transfer of the sample toward the specific knowledge distribution. Experiments on Spider dataset and its variants show the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Missing space in line 571 before enhance.\n"}, "extra_text": {}}, {"review_id": "3qYhAb8JA2", "reviewer": "Reviewer_sEnG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: This paper is challenging to follow, and the proposed method is highly complex, making it difficult to reproduce.\n\nA1:  We apologize for any confusion caused by the paper writing, in the future, we will improve our writing and increase the readability of our papers.\nWe would like to offer additional context to enhance the comprehension of our approach. Specifically, we emphasize that our model, designed as a plug-and-play module, can be seamlessly integrated into different backbone models. Our proposed method RefSQL only contains a simple retrieval and a contrastive learning module. It is important to note that published models can be our backbone models, such as RESDSQL(3B)+NATSQL, RatSQL, Flan-T5, and so on.\nWe will also publish our code soon to ensure the reproducibility of the work.\n\nQ2: The proposed method comprises several complicated modules and has more parameters than the baselines. It remains unclear whether the main performance gain originates from a particular module or if the improvement is merely due to having more parameters.\n\nA2: To address this, we have conducted a rigorous statistical analysis of the parameters associated with various modules within our method. The findings, as documented in the provided results and corroborated by Table 2 in the paper, reveal that performance enhancement is attainable through modules with a comparatively smaller parameter count.\n\n|                   Model                   | Parameters |\n|:-----------------------------------------:|:----------:|\n| SQL-Structure-enhanced Question Retriever |    110M    |\n|  Linking-Structure-based Schema Retriever |    1.4M    |\n| Contrastive Learning                      |    0.2M    |\n| Overall                                   |   111.6M   |\n\n|      Model      | Parameters | Dev_EM |\n|:---------------:|:----------:|--------|\n|      T5-3B      |     3B     |  71.5  |\n| T5-3B+our model |  3B+111.6M |  73.6  |\n\nFrom above we can see that on the T5-3B model, after adding parameters with a proportion of parameters, the model performance improved by 2.1 on Dev_EM.\nThis analysis emphasizes the nuanced interplay between module complexity, parameter allocation, and performance improvement. We believe that this clarification further elucidates the factors influencing the observed gains and will contribute to a more comprehensive understanding of the distinct contributions of individual modules to the overall efficacy of our approach.\n\nQ3: The authors claim that one of their main contributions is the use of a Mahalanobis contrastive learning method to narrow the distribution gap between retrieved examples and current samples. However, there are no experiments to verify whether Mahalanobis yields better results than standard contrastive learning.\n\nA3: We would like to elucidate the rationale behind our approach. Our focus lies in guiding samples towards a distribution comprised of similar instances, rather than disparate, isolated ones. This necessitates the consideration of both mean and covariance aspects within the distribution, making Mahalanobis contrastive learning particularly well-suited to our specific context. This approach inherently aligns with our objective of enhancing the relevance of retrieved examples.\nIn response to your concern, we have performed an ablation study to directly compare the performance of Mahalanobis contrastive learning with standard contrastive learning. The results, which we present below, shed light on the efficacy of our method:\n\n|                      Model                     | Dev_EM | Dev_EX |\n|:----------------------------------------------:|:------:|--------|\n|         RESDSQL(Flan-T5)+NATSQL+ReFSQL         |  86.6  |  88.1  |\n| RESDSQL(Flan-T5)+NATSQL+ReFSQL(w/ standard CL) |  85.7  |  87.5  |\n|            RESDSQL(3B)+NATSQL+ReFSQL           |  83.1  |  86.0  |\n|   RESDSQL(3B)+NATSQL+ReFSQL(w/ standard CL)    |  81.8  |  84.9  |\n\nThese findings emphasize the inherent advantages of our chosen approach and substantiate our claim that Mahalanobis contrastive learning contributes to the amelioration of the distribution gap, leading to improved performance. \n\nQ4: The proposed method involves multiple modules, which could impact training and inference speed. There should be experiments conducted to study and analyze these effects.\n\nA4:\nIn the inference stage: \nThis approach ensures that the inference speed during the operational phase remains unaffected and optimal. Since the retriever is trained well.\nIn the training stage:\nWe conduct experiments and compare our model with some baseline models.\n\n|       Model       | Training Time Cost | Dev_EX |\n|:-----------------:|:------------------:|--------|\n|       T5-3B       |      60.2hours     |  71.5  |\n|  T5-3B+our model  | 54.3hours |  73.6  |\n|      Flan-T5      |      85.6hours     |  81.2  |\n|  Flan-T5+ourmodel | 79.6hours |  82.9  |\n\nIt is worth noting that the convergence process of our two-stage model is notably efficient, with convergence achieved within a modest 50-epoch span. Furthermore, the total training duration for our model does not exceed 2 hours on our equipment.\nBy comparison, training the baseline model, RESDSQL(3B)+NATSQL, to attain optimal outcomes demands a significantly more extended training duration, involving more than 200 epochs and spanning over 50 hours on the same equipment.\nIn addition, after concatenating the retrieved information in the input, the model converges faster", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The idea of using retrieval methods to enhance the process is reasonable.\n\n2.This paper demonstrates significant improvements over existing methods.", "reasons_to_reject": "1.This paper is challenging to follow, and the proposed method is highly complex, making it difficult to reproduce.\n\n2.The proposed method comprises several complicated modules and has more parameters than the baselines. It remains unclear whether the main performance gain originates from a particular module or if the improvement is merely due to having more parameters. The current version of the ablation study does not provide definitive answers to these questions.\n\n3.The authors claim that one of their main contributions is the use of a Mahalanobis contrastive learning method to narrow the distribution gap between retrieved examples and current samples. However, there are no experiments to verify whether Mahalanobis yields better results than standard contrastive learning.\n\n4.The proposed method involves multiple modules, which could impact training and inference speed. There should be experiments conducted to study and analyze these effects.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the task of text-to-SQL generation and introduces a retrieval-augmented model to enhance SQL generation. The proposed method utilizes a structure-enhanced retriever to retrieve examples, which are then employed to improve the SQL generation process. To further enhance the model's performance, the author also incorporates a Mahalanobis contrastive learning method to maximize the representation of both retrieved and current examples.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typo:\nLine 365, the term L_{ma} should be consist in the whole paper."}, "extra_text": {}}, {"review_id": "STv0V5gHPA", "reviewer": "Reviewer_wb9q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Poor writing, bad typesetting, and the figures are not vector diagrams.\n\nA1: Thank you for taking the time to review our paper and provide feedback. We will check our paper and update it in the revision.\n\nQ2: How can the motivation \"To further bridge the gap between specific and general knowledge\" be implemented according to the proposed contrastive learning whose optimization is only minimizing the margin of the representation between similar samples?\n\nA2: To provide clarity, we wish to emphasize that our focus lies in narrowing the gap by directing the representation of each sample towards a more specific distribution, thereby accentuating specific knowledge rather than homogenizing it towards a general knowledge distribution. In contrast to prior models that employ a one-size-fits-all approach for all samples, our methodology is designed to enhance attention to the nuanced specifics of individual samples.\nOur approach utilizes contrastive learning as a mechanism to minimize the representation margin between the current sample and positive samples (those with similar characteristics), while concurrently maximizing the margin between the current sample and negative samples (those with dissimilar characteristics). This optimization framework, which operates in a comparative manner, enables us to pull the representation of each sample closer to similar samples with shared specific knowledge.\n\nQ3: Most of the recent baselines in Spider build interaction graphs to joint the representation of the questions and schemas (RATSQL, LGESQL, etc.). What are the advantages of the margin methods in the retriever part that split Query+SQL and Schema in two stages, and the self-designed \"Interaction Graph Construction\" methods? There is no analysis in the Methodology and no comparison in the Experiment.\n\nA3: In response, we wish to elucidate that our retriever component operates in a two-stage fashion to effectively address the intricacies of identifying samples with aligned specific knowledge. This method carefully considers both the semantics of the questions and the structural layout of the schemas, ensuring a holistic perspective. To substantiate these design choices, we have augmented our methodology with additional experiments, the results of which are illustrated in the table provided. These results, indeed, reinforce the efficacy of our two-stage approach, showcasing its aptitude in producing favorable outcomes.\nContrasting our method with baselines such as RatSQL and lgeSql, we emphasize the advantages that our margin-based method offers in terms of computational efficiency, model weight, and self-supervised learning. RatSQL and lgeSql introduce additional information for SQL generation and graph-question interaction, which leads to a higher parameter count and more resource-intensive models. By contrast, our approach harnesses a lightweight, self-supervised two-stage model that can be seamlessly integrated with various backbone models.\n|                    Model                   | Dev_EM | Dev_EX |\n|:------------------------------------------:|:------:|--------|\n|       RESDSQL(Flan-T5)+NATSQL+ReFSQL       |  86.6  |  88.1  |\n| RESDSQL(Flan-T5)+NATSQL+ReFSQL(w/ o SQLSE) |  85.7  |  87.4  |\n| RESDSQL(Flan-T5)+NATSQL+ReFSQL(w/ o LSES)  |  85.8  |  87.6  |\n|          RESDSQL(3B)+NATSQL+ReFSQL         |  83.1  |  86.0  |\n|    RESDSQL(3B)+NATSQL+ReFSQL(w/o SQLSE)    |  81.7  |  85.0  |\n| RESDSQL(Flan-T5)+NATSQL+ReFSQL(w/ o LSES)  |  82.0  |  85.3  |\n\nFurthermore, our self-designed \"Interaction Graph Construction\" methodology, although distinct from approaches like ratsql and lgesql, is meticulously crafted to capture pertinent alignment information and table structure characteristics without requiring the addition of syntactic parsing details. This results in a more concise representation that minimizes computational costs, rendering our method a more efficient alternative.\n\n|                                      Model                                     | Dev_EM | Dev_EX |\n|:------------------------------------------------------------------------------:|:------:|--------|\n|                         RESDSQL(Flan-T5)+NATSQL+ReFSQL                         |  86.6  |  88.1  |\n| RESDSQL(Flan-T5)+NATSQL+ReFSQL(w/ w/ interaction graph construction in RatSQL) |  86.6  |  87.9  |\n|                            RESDSQL(3B)+NATSQL+ReFSQL                           |  83.1  |  86.0  |\n|    RESDSQL(3B)+NATSQL+ReFSQL(w/ w/ interaction graph construction in RatSQL)   |  82.9  |  86.0  |\n\nWhile we acknowledge the need for further comprehensive analysis and direct comparisons, our initial findings, as presented, lay the groundwork for a deeper exploration of the distinct strengths our method brings to the table in the context of Spider.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a retriever-generator framework to improve the representation via contrastive learning.\n2. Good experimental results. The baselines equipped with the proposed ReFSQL achieve obvious promotions, especially in Spider (which seems achieving SOTA compared with the leaderboard).", "reasons_to_reject": "1. Poor writing, bad typesetting, and the figures are not vector diagrams.\n2. How can the motivation \"To further bridge the gap between specific and general knowledge\" be implemented according to the proposed contrastive learning whose optimization is only minimizing the margin of the representation between similar samples?\n3. Most of the recent baselines in Spider build interaction graphs to joint the representation of the questions and schemas (RATSQL, LGESQL, etc.). What are the advantages of the margin methods in the retriever part that split Query+SQL and Schema in two stages, and the self-designed \"Interaction Graph Construction\" methods? There is no analysis in the Methodology and no comparison in the Experiment.\n4. Fine-grained ablation tests need to be provided.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a framework that consists of a retriever and a generator. Within, the retriever aims to obtain similar samples according to the similarity score calculated by questions, SQLs and graphs built from schemas. The constructed similar samples as positive samples combined with the negative samples are employed for representation learning for the generator via contrastive learning. The experiments indicate that equipping the proposed ReFSQL can bring an obvious improvement to the backbone methods", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes an approach for text-to-SQL generation that leverages a structure-enhanced retriever to get similar examples which can guide the generation process. Mahalanobis contrastive learning is also employed to enhance the relevance of the retrieved examples. 

Extensive experiments demonstrate the effectiveness of the proposed solution. The paper can improve in terms of writing and clarity: the proposed architecture is overly complex and hard to follow; furthermore, each the authors should better explain the intuition behind each component and better justify its usage. Also an ablation study (which the authors included only in the rebuttal) should be added to the paper.
Decision: Accept-Findings